net: network drivers no longer need to implement ndo_busy_poll()

Instead of having to implement complex ndo_busy_poll() method,
drivers can simply rely on NAPI poll logic.

Busy polling gains are mainly coming from polling itself,
not on exact details on how we poll the device.

ndo_busy_poll() if implemented can avoid touching
napi state, but it adds extra synchronization between
normal napi->poll() and busy poll handler, slowing down
the common path (non busy polling) with extra atomic operations.
In practice few drivers ever got busy poll because of the complexity.

We could go one step further, and make busy polling
available for all NAPI drivers, but this would require
that all netif_napi_del() calls are done in process context
so that we can call synchronize_rcu().
Full audit would be required.

Before this is done, a driver still needs to call :

- skb_mark_napi_id() for each skb provided to the stack.
- napi_hash_add() and napi_hash_del() to allocate a napi_id per napi struct.
- Make sure RCU grace period is respected after napi_hash_del() before
  memory containing napi structure is freed.

Followup patch implements busy poll for mlx5 driver as an example.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

diff --git a/net/core/dev.c b/net/core/dev.c
index 2002eec2617d626c4b6e331055efcc9c2d29ebea..93009610aee84915a536c7111deed0433710bbb9 100644 (file)
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -4677,10 +4677,11 @@ static struct napi_struct *napi_by_id(unsigned int napi_id)
 }
 
 #if defined(CONFIG_NET_RX_BUSY_POLL)
+#define BUSY_POLL_BUDGET 8
 bool sk_busy_loop(struct sock *sk, int nonblock)
 {
        unsigned long end_time = !nonblock ? sk_busy_loop_end_time(sk) : 0;
-       const struct net_device_ops *ops;
+       int (*busy_poll)(struct napi_struct *dev);
        struct napi_struct *napi;
        int rc = false;
 
@@ -4690,13 +4691,27 @@ bool sk_busy_loop(struct sock *sk, int nonblock)
        if (!napi)
                goto out;
 
-       ops = napi->dev->netdev_ops;
-       if (!ops->ndo_busy_poll)
-               goto out;
+       /* Note: ndo_busy_poll method is optional in linux-4.5 */
+       busy_poll = napi->dev->netdev_ops->ndo_busy_poll;
 
        do {
+               rc = 0;
                local_bh_disable();
-               rc = ops->ndo_busy_poll(napi);
+               if (busy_poll) {
+                       rc = busy_poll(napi);
+               } else if (napi_schedule_prep(napi)) {
+                       void *have = netpoll_poll_lock(napi);
+
+                       if (test_bit(NAPI_STATE_SCHED, &napi->state)) {
+                               rc = napi->poll(napi, BUSY_POLL_BUDGET);
+                               trace_napi_poll(napi);
+                               if (rc == BUSY_POLL_BUDGET) {
+                                       napi_complete_done(napi, rc);
+                                       napi_schedule(napi);
+                               }
+                       }
+                       netpoll_poll_unlock(have);
+               }
                if (rc > 0)
                        NET_ADD_STATS_BH(sock_net(sk),
                                         LINUX_MIB_BUSYPOLLRXPACKETS, rc);