diff options
| author | Thomas Gleixner <tglx@linutronix.de> | 2023-03-23 21:55:32 +0100 |
|---|---|---|
| committer | Jakub Kicinski <kuba@kernel.org> | 2023-03-28 18:52:28 -0700 |
| commit | bc9d3a9f2afca189a6ae40225b6985e3c775375e (patch) | |
| tree | 4e23464338077861625422ed7db14381cc888997 /include/net/sock.h | |
| parent | d288a162dd1c73507da582966f17dd226e34a0c0 (diff) | |
| download | linux-bc9d3a9f2afca189a6ae40225b6985e3c775375e.tar.gz | |
net: dst: Switch to rcuref_t reference counting
Under high contention dst_entry::__refcnt becomes a significant bottleneck.
atomic_inc_not_zero() is implemented with a cmpxchg() loop, which goes into
high retry rates on contention.
Switch the reference count to rcuref_t which results in a significant
performance gain. Rename the reference count member to __rcuref to reflect
the change.
The gain depends on the micro-architecture and the number of concurrent
operations and has been measured in the range of +25% to +130% with a
localhost memtier/memcached benchmark which amplifies the problem
massively.
Running the memtier/memcached benchmark over a real (1Gb) network
connection the conversion on top of the false sharing fix for struct
dst_entry::__refcnt results in a total gain in the 2%-5% range over the
upstream baseline.
Reported-by: Wangyang Guo <wangyang.guo@intel.com>
Reported-by: Arjan Van De Ven <arjan.van.de.ven@intel.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/r/20230307125538.989175656@linutronix.de
Link: https://lore.kernel.org/r/20230323102800.215027837@linutronix.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Diffstat (limited to 'include/net/sock.h')
| -rw-r--r-- | include/net/sock.h | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/include/net/sock.h b/include/net/sock.h index 573f2bf7e0de72..5edf0038867ce9 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -2131,7 +2131,7 @@ sk_dst_get(struct sock *sk) rcu_read_lock(); dst = rcu_dereference(sk->sk_dst_cache); - if (dst && !atomic_inc_not_zero(&dst->__refcnt)) + if (dst && !rcuref_get(&dst->__rcuref)) dst = NULL; rcu_read_unlock(); return dst; |