workqueue: UNBOUND -> REBIND morphing in rebind_workers() should be atomic

The compiler may compile the following code into TWO write/modify
instructions.

	worker->flags &= ~WORKER_UNBOUND;
	worker->flags |= WORKER_REBIND;

so the other CPU may temporarily see worker->flags which doesn't have
either WORKER_UNBOUND or WORKER_REBIND set and perform local wakeup
prematurely.

Fix it by using single explicit assignment via ACCESS_ONCE().

Because idle workers have another WORKER_NOT_RUNNING flag, this bug
doesn't exist for them; however, update it to use the same pattern for
consistency.

tj: Applied the change to idle workers too and updated comments and
    patch description a bit.

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: stable@vger.kernel.org

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 692d97628a106360683dfef46797952cdf1861e1..c462cd60c3743b58cf4fb15920d50d5fd5f2e23c 100644 (file)
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -1396,12 +1396,15 @@ retry:
        /* set REBIND and kick idle ones, we'll wait for these later */
        for_each_worker_pool(pool, gcwq) {
                list_for_each_entry(worker, &pool->idle_list, entry) {
+                       unsigned long worker_flags = worker->flags;
+
                        if (worker->flags & WORKER_REBIND)
                                continue;
 
-                       /* morph UNBOUND to REBIND */
-                       worker->flags &= ~WORKER_UNBOUND;
-                       worker->flags |= WORKER_REBIND;
+                       /* morph UNBOUND to REBIND atomically */
+                       worker_flags &= ~WORKER_UNBOUND;
+                       worker_flags |= WORKER_REBIND;
+                       ACCESS_ONCE(worker->flags) = worker_flags;
 
                        idle_rebind.cnt++;
                        worker->idle_rebind = &idle_rebind;
@@ -1434,10 +1437,12 @@ retry:
        /* rebind busy workers */
        for_each_busy_worker(worker, i, pos, gcwq) {
                struct work_struct *rebind_work = &worker->rebind_work;
+               unsigned long worker_flags = worker->flags;
 
-               /* morph UNBOUND to REBIND */
-               worker->flags &= ~WORKER_UNBOUND;
-               worker->flags |= WORKER_REBIND;
+               /* morph UNBOUND to REBIND atomically */
+               worker_flags &= ~WORKER_UNBOUND;
+               worker_flags |= WORKER_REBIND;
+               ACCESS_ONCE(worker->flags) = worker_flags;
 
                if (test_and_set_bit(WORK_STRUCT_PENDING_BIT,
                                     work_data_bits(rebind_work)))