rxe: Fix dma.length computation in wr_set_sge_list
wr_set_sge_list() summed the SGE lengths with a loop that never
advanced sg_list:
while (num_sge--)
tot_length += sg_list->length;
so tot_length ended up as num_sge * sg_list[0].length instead of the
true sum, and wqe->dma.length / wqe->dma.resid were written with that
wrong value. The per-SGE entries themselves were unaffected because
they are populated by the preceding memcpy().
The kernel rxe driver requires dma.length == sum(sge[i].length) and
enforces it in rxe_mr.c:copy_data(), so a multi-SGE WR posted through
the ibv_qp_ex builder API (ibv_wr_set_sge_list) on rxe completes with
IB_WC_LOC_PROT_ERR once finish_packet()/copy_data() runs off the end
of the SGE list.
The legacy ibv_post_send path (init_send_wqe) is unaffected; it sums
the lengths with an indexed for loop.
Fix by computing the total with an indexed loop, matching the style
already used in rxe_post_one_recv() and init_send_wqe() in this file.
Fixes: 1a894ca10105 ("Providers/rxe: Implement ibv_create_qp_ex verb")
Signed-off-by: Jared Holzman <jholzman@nvidia.com>
This is the userspace components for the Linux Kernel's drivers/infiniband subsystem. Specifically this contains the userspace libraries for the following device nodes:
The userspace component of the libibverbs RDMA kernel drivers are included under the providers/ directory. Support for the following Kernel RDMA drivers is included:
Additional service daemons are provided for:
This project uses a cmake based build system. Quick start:
$ bash build.sh
build/bin will contain the sample programs and build/lib will contain the shared libraries. The build is configured to run all the programs ‘in-place’ and cannot be installed.
$ apt-get install build-essential cmake gcc libudev-dev libnl-3-dev libnl-route-3-dev ninja-build pkg-config valgrind python3-dev cython3 python3-docutils pandoc
Supported releases:
$ dnf builddep redhat/rdma-core.spec
NOTE: Fedora Core uses the name ‘ninja-build’ for the ‘ninja’ command.
$ zypper install cmake gcc libnl3-devel libudev-devel ninja pkg-config valgrind-devel python3-devel python3-Cython python3-docutils pandoc
Install required packages:
$ yum install cmake gcc libnl3-devel libudev-devel make pkgconfig valgrind-devel
Developers on CentOS 7 or Amazon Linux 2 are suggested to install more modern tooling for the best experience.
CentOS 7:
$ yum install epel-release $ yum install cmake3 ninja-build pandoc
Amazon Linux 2:
$ amazon-linux-extras install epel $ yum install cmake3 ninja-build pandoc
NOTE: EPEL uses the name ‘ninja-build’ for the ‘ninja’ command, and ‘cmake3’ for the ‘cmake’ command.
To set up software RDMA on an existing interface with either of the available drivers, use the following commands, substituting <DRIVER> with the name of the driver of your choice (rdma_rxe or siw) and <TYPE> with the type corresponding to the driver (rxe or siw).
# modprobe <DRIVER> # rdma link add <NAME> type <TYPE> netdev <DEVICE>
Please note that you need version of iproute2 recent enough is required for the command above to work.
You can use either ibv_devices or rdma link to verify that the device was successfully added.
Bugs should be reported to the linux-rdma@vger.kernel.org mailing list In your bug report, please include:
Information about your system:
How to reproduce the bug.
If the bug is a crash, the exact output printed out when the crash occurred, including any kernel messages produced.
See Contributing to rdma-core.
Stable versions are released regularly with backported fixes (see Documentation/stable.md) The current minimum version still maintained is ‘v33.X’