vmwgfx: fix incorrect VRAM size check in vmw_kms_fb_create()
authorXi Wang <xi.wang@gmail.com>
Wed, 21 Dec 2011 10:18:33 +0000 (05:18 -0500)
committerDave Airlie <airlied@redhat.com>
Thu, 22 Dec 2011 12:31:02 +0000 (12:31 +0000)
Commit e133e737 didn't correctly fix the integer overflow issue.

- unsigned int required_size;
+ u64 required_size;
...
required_size = mode_cmd->pitch * mode_cmd->height;
- if (unlikely(required_size > dev_priv->vram_size)) {
+ if (unlikely(required_size > (u64) dev_priv->vram_size)) {

Note that both pitch and height are u32.  Their product is still u32 and
would overflow before being assigned to required_size.  A correct way is
to convert pitch and height to u64 before the multiplication.

required_size = (u64)mode_cmd->pitch * (u64)mode_cmd->height;

This patch calls the existing vmw_kms_validate_mode_vram() for
validation.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Reviewed-and-tested-by: Thomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
drivers/gpu/drm/vmwgfx/vmwgfx_kms.c

index 8aa1dbb45c67efad383e6f938383efeb11589210..f94b33ae221546a9d9170f3e0eb6b8292776721b 100644 (file)
@@ -1093,7 +1093,6 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
        struct vmw_surface *surface = NULL;
        struct vmw_dma_buffer *bo = NULL;
        struct ttm_base_object *user_obj;
-       u64 required_size;
        int ret;
 
        /**
@@ -1102,8 +1101,9 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
         * requested framebuffer.
         */
 
-       required_size = mode_cmd->pitch * mode_cmd->height;
-       if (unlikely(required_size > (u64) dev_priv->vram_size)) {
+       if (!vmw_kms_validate_mode_vram(dev_priv,
+                                       mode_cmd->pitch,
+                                       mode_cmd->height)) {
                DRM_ERROR("VRAM size is too small for requested mode.\n");
                return ERR_PTR(-ENOMEM);
        }