powerpc/watchdog: Improve watchdog lock primitive

- Hard-disable interrupts before taking the lock, which prevents
  soft-NMI re-entrancy and therefore can prevent deadlocks.
- Use raw_ variants of local_irq_disable to avoid irq debugging.
- When the lock is contended, spin at low SMT priority, using
  loads only, and with interrupts enabled (where possible).

Some stalls have been noticed at high loads that go away with improved
locking. There should not be so much locking contention in the first
place (which is addressed in a subsequent patch), but locking should
still be improved.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>

diff --git a/arch/powerpc/kernel/watchdog.c b/arch/powerpc/kernel/watchdog.c
index b67f8b03a32d0f12ce29eeb4ac3be0a97384fe72..fda4d044d3264508b93e163978e336d1d188eef4 100644 (file)
--- a/arch/powerpc/kernel/watchdog.c
+++ b/arch/powerpc/kernel/watchdog.c
@@ -71,15 +71,20 @@ static inline void wd_smp_lock(unsigned long *flags)
         * This may be called from low level interrupt handlers at some
         * point in future.
         */
-       local_irq_save(*flags);
-       while (unlikely(test_and_set_bit_lock(0, &__wd_smp_lock)))
-               cpu_relax();
+       raw_local_irq_save(*flags);
+       hard_irq_disable(); /* Make it soft-NMI safe */
+       while (unlikely(test_and_set_bit_lock(0, &__wd_smp_lock))) {
+               raw_local_irq_restore(*flags);
+               spin_until_cond(!test_bit(0, &__wd_smp_lock));
+               raw_local_irq_save(*flags);
+               hard_irq_disable();
+       }
 }
 
 static inline void wd_smp_unlock(unsigned long *flags)
 {
        clear_bit_unlock(0, &__wd_smp_lock);
-       local_irq_restore(*flags);
+       raw_local_irq_restore(*flags);
 }
 
 static void wd_lockup_ipi(struct pt_regs *regs)