srcu: Prevent sdp->srcu_gp_seq_needed counter wrap

If a given CPU never happens to ever start an SRCU grace period, the
grace-period sequence counter might wrap.  If this CPU were to decide to
finally start a grace period, the state of its sdp->srcu_gp_seq_needed
might make it appear that it has already requested this grace period,
which would prevent starting the grace period.  If no other CPU ever started
a grace period again, this would look like a grace-period hang.  Even
if some other CPU took pity and started the needed grace period, the
leaf rcu_node structure's ->srcu_data_have_cbs field won't have record
of the fact that this CPU has a callback pending, which would look like
a very localized grace-period hang.

This might seem very unlikely, but SRCU grace periods can take less than
a microsecond on small systems, which means that overflow can happen
in much less than an hour on a 32-bit embedded system.  And embedded
systems are especially likely to have long-term idle CPUs.  Therefore,
it makes sense to prevent this scenario from happening.

This commit therefore scans each srcu_data structure occasionally,
with frequency controlled by the srcutree.counter_wrap_check kernel
boot parameter.  This parameter can be set to something like 255
in order to exercise the counter-wrap-prevention code.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 01b5ab92d2516a4153ba3964f68c458e8b3a41a7..6671f9b60a86a50d8853b337ec942517b2a9c0a2 100644 (file)
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -3810,6 +3810,15 @@
        spia_pedr=
        spia_peddr=
 
+       srcutree.counter_wrap_check [KNL]
+                       Specifies how frequently to check for
+                       grace-period sequence counter wrap for the
+                       srcu_data structure's ->srcu_gp_seq_needed field.
+                       The greater the number of bits set in this kernel
+                       parameter, the less frequently counter wrap will
+                       be checked for.  Note that the bottom two bits
+                       are ignored.
+
        srcutree.exp_holdoff [KNL]
                        Specifies how many nanoseconds must elapse
                        since the end of the last SRCU grace period for

diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index c6e2a4a1628bf565561f71508fc50bde2014e91a..cc06dbfc96928d8001ceab464da6d877d274e0db 100644 (file)
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -45,6 +45,10 @@
 static ulong exp_holdoff = DEFAULT_SRCU_EXP_HOLDOFF;
 module_param(exp_holdoff, ulong, 0444);
 
+/* Overflow-check frequency.  N bits roughly says every 2**N grace periods. */
+static ulong counter_wrap_check = (ULONG_MAX >> 2);
+module_param(counter_wrap_check, ulong, 0444);
+
 static void srcu_invoke_callbacks(struct work_struct *work);
 static void srcu_reschedule(struct srcu_struct *sp, unsigned long delay);
 
@@ -496,10 +500,13 @@ static void srcu_gp_end(struct srcu_struct *sp)
 {
        unsigned long cbdelay;
        bool cbs;
+       int cpu;
+       unsigned long flags;
        unsigned long gpseq;
        int idx;
        int idxnext;
        unsigned long mask;
+       struct srcu_data *sdp;
        struct srcu_node *snp;
 
        /* Prevent more than one additional grace period. */
@@ -538,6 +545,17 @@ static void srcu_gp_end(struct srcu_struct *sp)
                        smp_mb(); /* GP end before CB invocation. */
                        srcu_schedule_cbs_snp(sp, snp, mask, cbdelay);
                }
+
+               /* Occasionally prevent srcu_data counter wrap. */
+               if (!(gpseq & counter_wrap_check))
+                       for (cpu = snp->grplo; cpu <= snp->grphi; cpu++) {
+                               sdp = per_cpu_ptr(sp->sda, cpu);
+                               spin_lock_irqsave(&sdp->lock, flags);
+                               if (ULONG_CMP_GE(gpseq,
+                                                sdp->srcu_gp_seq_needed + 100))
+                                       sdp->srcu_gp_seq_needed = gpseq;
+                               spin_unlock_irqrestore(&sdp->lock, flags);
+                       }
        }
 
        /* Callback initiation done, allow grace periods after next. */