mm: memcontrol: rearrange charging fast path

The charging path currently starts out with OOM condition checks when
OOM is the rarest possible case.

Rearrange this code to run OOM/task dying checks only after trying the
percpu charge and the res_counter charge and bail out before entering
reclaim.  Attempting a charge does not hurt an (oom-)killed task as much
as every charge attempt having to check OOM conditions.  Also, only
check __GFP_NOFAIL when the charge would actually fail.

Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Michal Hocko <mhocko@suse.cz>
Cc: Hugh Dickins <hughd@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vladimir Davydov <vdavydov@parallels.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index fe3ad310656d9824fdeac07d41f32c74e0eecf75..f7b6bec9f53876c2a8aedcf7b9cb8cb23e54afc0 100644 (file)
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2575,22 +2575,6 @@ static int mem_cgroup_try_charge(struct mem_cgroup *memcg,
 
        if (mem_cgroup_is_root(memcg))
                goto done;
-       /*
-        * Unlike in global OOM situations, memcg is not in a physical
-        * memory shortage.  Allow dying and OOM-killed tasks to
-        * bypass the last charges so that they can exit quickly and
-        * free their memory.
-        */
-       if (unlikely(test_thread_flag(TIF_MEMDIE) ||
-                    fatal_signal_pending(current) ||
-                    current->flags & PF_EXITING))
-               goto bypass;
-
-       if (unlikely(task_in_memcg_oom(current)))
-               goto nomem;
-
-       if (gfp_mask & __GFP_NOFAIL)
-               oom = false;
 retry:
        if (consume_stock(memcg, nr_pages))
                goto done;
@@ -2612,6 +2596,20 @@ retry:
                goto retry;
        }
 
+       /*
+        * Unlike in global OOM situations, memcg is not in a physical
+        * memory shortage.  Allow dying and OOM-killed tasks to
+        * bypass the last charges so that they can exit quickly and
+        * free their memory.
+        */
+       if (unlikely(test_thread_flag(TIF_MEMDIE) ||
+                    fatal_signal_pending(current) ||
+                    current->flags & PF_EXITING))
+               goto bypass;
+
+       if (unlikely(task_in_memcg_oom(current)))
+               goto nomem;
+
        if (!(gfp_mask & __GFP_WAIT))
                goto nomem;
 
@@ -2640,6 +2638,9 @@ retry:
        if (mem_cgroup_wait_acct_move(mem_over_limit))
                goto retry;
 
+       if (gfp_mask & __GFP_NOFAIL)
+               goto bypass;
+
        if (fatal_signal_pending(current))
                goto bypass;