IB: Increase DMA max_segment_size on Mellanox hardware

By default, each device is assumed to be able only handle 64 KB chunks
during DMA. By giving the segment size a larger value, the block layer
will coalesce more S/G entries together for SRP, allowing larger
requests with the same sg_tablesize setting.  The block layer is the
only direct user of it, though a few IOMMU drivers reference it as
well for their *_map_sg coalescing code. pci-gart_64 on x86, and a
smattering on on sparc, powerpc, and ia64.

Since other IB protocols could potentially see larger segments with
this, let's check those:

 - iSER is fine, because you limit your maximum request size to 512
   KB, so we'll never overrun the page vector in struct iser_page_vec
   (128 entries currently). It is independent of the DMA segment size,
   and handles multi-page segments already.

 - IPoIB is fine, as it maps each page individually, and doesn't use
   ib_dma_map_sg().

 - RDS appears to do the right thing and has no dependencies on DMA
   segment size, but I don't claim to have done a complete audit.

 - NFSoRDMA and 9p are OK -- they do not use ib_dma_map_sg(), so they
   doesn't care about the coalescing.

 - Lustre's ko2iblnd does not care about coalescing -- it properly
   walks the returned sg list.

This patch ups the value on Mellanox hardware to 1 GB, which matches
reported firmware limits on mlx4.

Signed-off-by: David Dillow <dillowda@ornl.gov>
Signed-off-by: Roland Dreier <roland@purestorage.com>

diff --git a/drivers/infiniband/hw/mthca/mthca_main.c b/drivers/infiniband/hw/mthca/mthca_main.c
index 8a40cd539ab11f02855f8513ec8d98e86d59c199..f24b79b805f2b0463ae3383e1c62f0133d83c8ef 100644 (file)
--- a/drivers/infiniband/hw/mthca/mthca_main.c
+++ b/drivers/infiniband/hw/mthca/mthca_main.c
@@ -1043,6 +1043,9 @@ static int __mthca_init_one(struct pci_dev *pdev, int hca_type)
                }
        }
 
+       /* We can handle large RDMA requests, so allow larger segments. */
+       dma_set_max_seg_size(&pdev->dev, 1024 * 1024 * 1024);
+
        mdev = (struct mthca_dev *) ib_alloc_device(sizeof *mdev);
        if (!mdev) {
                dev_err(&pdev->dev, "Device struct alloc failed, "

diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c
index 2765a3ce9c24db25d1fbc5616cebf3da8f0c3e24..c83501122d77370dd24344fa61e51f8a93b7351f 100644 (file)
--- a/drivers/net/mlx4/main.c
+++ b/drivers/net/mlx4/main.c
@@ -1109,6 +1109,9 @@ static int __mlx4_init_one(struct pci_dev *pdev, const struct pci_device_id *id)
                }
        }
 
+       /* Allow large DMA segments, up to the firmware limit of 1 GB */
+       dma_set_max_seg_size(&pdev->dev, 1024 * 1024 * 1024);
+
        priv = kzalloc(sizeof *priv, GFP_KERNEL);
        if (!priv) {
                dev_err(&pdev->dev, "Device struct alloc failed, "