BACKPORT: zsmalloc: introduce zs_huge_class_size()

Patch series "zsmalloc/zram: drop zram's max_zpage_size", v3.

ZRAM's max_zpage_size is a bad thing.  It forces zsmalloc to store
normal objects as huge ones, which results in bigger zsmalloc memory
usage.  Drop it and use actual zsmalloc huge-class value when decide if
the object is huge or not.

This patch (of 2):

Not every object can be share its zspage with other objects, e.g.  when
the object is as big as zspage or nearly as big a zspage.  For such
objects zsmalloc has a so called huge class - every object which belongs
to huge class consumes the entire zspage (which consists of a physical
page).  On x86_64, PAGE_SHIFT 12 box, the first non-huge class size is
3264, so starting down from size 3264, objects can share page(-s) and
thus minimize memory wastage.

ZRAM, however, has its own statically defined watermark for huge
objects, namely "3 * PAGE_SIZE / 4 = 3072", and forcibly stores every
object larger than this watermark (3072) as a PAGE_SIZE object, in other
words, to a huge class, while zsmalloc can keep some of those objects in
non-huge classes.  This results in increased memory consumption.

zsmalloc knows better if the object is huge or not.  Introduce
zs_huge_class_size() function which tells if the given object can be
stored in one of non-huge classes or not.  This will let us to drop
ZRAM's huge object watermark and fully rely on zsmalloc when we decide
if the object is huge.

[sergey.senozhatsky.work@gmail.com: add pool param to zs_huge_class_size()]
Link: http://lkml.kernel.org/r/20180314081833.1096-2-sergey.senozhatsky@gmail.com
Link: http://lkml.kernel.org/r/20180306070639.7389-2-sergey.senozhatsky@gmail.com
Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Acked-by: Minchan Kim <minchan@kernel.org>
Cc: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit 010b495e2fa32353d0ef6aa70a8169e5ef617a15)
Signed-off-by: Peter Kalauskas <peskal@google.com>
Bug: 113183619
Change-Id: Ic35f8c1ec75f0b78bf2d83729b6aedd2999f25c8

diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h
index 527a56cad37a2c0d3798c82a50f68ff2a94b60b0..70a3f75ee7b3ca779a80698e8de5a4009b916a51 100644 (file)
--- a/include/linux/zsmalloc.h
+++ b/include/linux/zsmalloc.h
@@ -50,6 +50,8 @@ void zs_destroy_pool(struct zs_pool *pool);
 unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t flags);
 void zs_free(struct zs_pool *pool, unsigned long obj);
 
+size_t zs_huge_class_size(struct zs_pool *pool);
+
 void *zs_map_object(struct zs_pool *pool, unsigned long handle,
                        enum zs_mapmode mm);
 void zs_unmap_object(struct zs_pool *pool, unsigned long handle);

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 5111022570362a306440707fb62ebc092b05a1b5..a9d802b3b28937b281a9440abc4932a990bae92c 100644 (file)
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -196,6 +196,7 @@ static int zs_size_classes;
  * (see: fix_fullness_group())
  */
 static const int fullness_threshold_frac = 4;
+static size_t huge_class_size;
 
 struct size_class {
        spinlock_t lock;
@@ -1510,6 +1511,25 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle)
 }
 EXPORT_SYMBOL_GPL(zs_unmap_object);
 
+/**
+ * zs_huge_class_size() - Returns the size (in bytes) of the first huge
+ *                        zsmalloc &size_class.
+ * @pool: zsmalloc pool to use
+ *
+ * The function returns the size of the first huge class - any object of equal
+ * or bigger size will be stored in zspage consisting of a single physical
+ * page.
+ *
+ * Context: Any context.
+ *
+ * Return: the size (in bytes) of the first huge zsmalloc &size_class.
+ */
+size_t zs_huge_class_size(struct zs_pool *pool)
+{
+       return huge_class_size;
+}
+EXPORT_SYMBOL_GPL(zs_huge_class_size);
+
 static unsigned long obj_malloc(struct size_class *class,
                                struct zspage *zspage, unsigned long handle)
 {
@@ -2469,6 +2489,7 @@ struct zs_pool *zs_create_pool(const char *name)
        for (i = zs_size_classes - 1; i >= 0; i--) {
                int size;
                int pages_per_zspage;
+               int objs_per_zspage;
                struct size_class *class;
                int fullness = 0;
 
@@ -2476,6 +2497,28 @@ struct zs_pool *zs_create_pool(const char *name)
                if (size > ZS_MAX_ALLOC_SIZE)
                        size = ZS_MAX_ALLOC_SIZE;
                pages_per_zspage = get_pages_per_zspage(size);
+               objs_per_zspage = pages_per_zspage * PAGE_SIZE / size;
+
+               /*
+                * We iterate from biggest down to smallest classes,
+                * so huge_class_size holds the size of the first huge
+                * class. Any object bigger than or equal to that will
+                * endup in the huge class.
+                */
+               if (pages_per_zspage != 1 && objs_per_zspage != 1 &&
+                               !huge_class_size) {
+                       huge_class_size = size;
+                       /*
+                        * The object uses ZS_HANDLE_SIZE bytes to store the
+                        * handle. We need to subtract it, because zs_malloc()
+                        * unconditionally adds handle size before it performs
+                        * size class search - so object may be smaller than
+                        * huge class size, yet it still can end up in the huge
+                        * class because it grows by ZS_HANDLE_SIZE extra bytes
+                        * right before class lookup.
+                        */
+                       huge_class_size -= (ZS_HANDLE_SIZE - 1);
+               }
 
                /*
                 * size_class is used for normal zsmalloc operation such