percpu: implement asynchronous chunk population
authorTejun Heo <tj@kernel.org>
Tue, 2 Sep 2014 18:46:05 +0000 (14:46 -0400)
committerTejun Heo <tj@kernel.org>
Tue, 2 Sep 2014 18:46:05 +0000 (14:46 -0400)
commit1a4d76076cda69b0abf15463a8cebc172406da25
treeeb2f8e317795c30942ae298585708e10652e8537
parentfe6bd8c3d28357174587c4fe895d10b00321b692
percpu: implement asynchronous chunk population

The percpu allocator now supports atomic allocations by only
allocating from already populated areas but the mechanism to ensure
that there's adequate amount of populated areas was missing.

This patch expands pcpu_balance_work so that in addition to freeing
excess free chunks it also populates chunks to maintain an adequate
level of populated areas.  pcpu_alloc() schedules pcpu_balance_work if
the amount of free populated areas is too low or after an atomic
allocation failure.

* PERPCU_DYNAMIC_RESERVE is increased by two pages to account for
  PCPU_EMPTY_POP_PAGES_LOW.

* pcpu_async_enabled is added to gate both async jobs -
  chunk->map_extend_work and pcpu_balance_work - so that we don't end
  up scheduling them while the needed subsystems aren't up yet.

Signed-off-by: Tejun Heo <tj@kernel.org>
include/linux/percpu.h
mm/percpu.c