locking/pvqspinlock: Order pv_unhash() after cmpxchg() on unlock slowpath

When we unlock in __pv_queued_spin_unlock(), a failed cmpxchg() on the lock
value indicates that we need to take the slow-path and unhash the
corresponding node blocked on the lock.

Since a failed cmpxchg() does not provide any memory-ordering guarantees,
it is possible that the node data could be read before the cmpxchg() on
weakly-ordered architectures and therefore return a stale value, leading
to hash corruption and/or a BUG().

This patch adds an smb_rmb() following the failed cmpxchg operation, so
that the unhashing is ordered after the lock has been checked.

Reported-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Will Deacon <will.deacon@arm.com>
[ Added more comments]
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by:  Waiman Long <Waiman.Long@hp.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Paul McKenney <paulmck@linux.vnet.ibm.com>
Cc: Steve Capper <Steve.Capper@arm.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/20150713155830.GL2632@arm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>

diff --git a/kernel/locking/qspinlock_paravirt.h b/kernel/locking/qspinlock_paravirt.h
index 489a878843371837e385e7c1a6e5459e3a448d81..ab8b1bb8caa411d86cecb915329ed7be7ce4de2b 100644 (file)
--- a/kernel/locking/qspinlock_paravirt.h
+++ b/kernel/locking/qspinlock_paravirt.h
@@ -244,13 +244,17 @@ static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node)
                if (!lp) { /* ONCE */
                        lp = pv_hash(lock, pn);
                        /*
-                        * lp must be set before setting _Q_SLOW_VAL
+                        * We must hash before setting _Q_SLOW_VAL, such that
+                        * when we observe _Q_SLOW_VAL in __pv_queued_spin_unlock()
+                        * we'll be sure to be able to observe our hash entry.
                         *
-                        * [S] lp = lock                [RmW] l = l->locked = 0
-                        *     MB                             MB
-                        * [S] l->locked = _Q_SLOW_VAL  [L]   lp
+                        *   [S] pn->state
+                        *   [S] <hash>                 [Rmw] l->locked == _Q_SLOW_VAL
+                        *       MB                           RMB
+                        * [RmW] l->locked = _Q_SLOW_VAL  [L] <unhash>
+                        *                                [L] pn->state
                         *
-                        * Matches the cmpxchg() in __pv_queued_spin_unlock().
+                        * Matches the smp_rmb() in __pv_queued_spin_unlock().
                         */
                        if (!cmpxchg(&l->locked, _Q_LOCKED_VAL, _Q_SLOW_VAL)) {
                                /*
@@ -305,6 +309,15 @@ __visible void __pv_queued_spin_unlock(struct qspinlock *lock)
                return;
        }
 
+       /*
+        * A failed cmpxchg doesn't provide any memory-ordering guarantees,
+        * so we need a barrier to order the read of the node data in
+        * pv_unhash *after* we've read the lock being _Q_SLOW_VAL.
+        *
+        * Matches the cmpxchg() in pv_wait_head() setting _Q_SLOW_VAL.
+        */
+       smp_rmb();
+
        /*
         * Since the above failed to release, this must be the SLOW path.
         * Therefore start by looking up the blocked node and unhashing it.