| SCSI RDMA Protocol (SRP) Target driver for Linux |
| ================================================= |
| |
| The SRP Target driver is designed to work directly on top of the |
| OpenFabrics OFED-1.x software stack (http://www.openfabrics.org) or |
| the Infiniband drivers in the Linux kernel tree |
| (http://www.kernel.org). The SRP target driver also interfaces with |
| the generic SCSI target mid-level driver called SCST |
| (http://scst.sourceforge.net). |
| |
| How-to run |
| ----------- |
| |
| A. On srp target machine |
| 1. Please refer to SCST's README for loading scst driver and its |
| dev_handlers drivers (scst_disk, scst_vdisk block or file IO mode, nullio, ...) |
| |
| Example 1: working with real back-end scsi disks |
| a. modprobe scst |
| b. modprobe scst_disk |
| c. cat /proc/scsi_tgt/scsi_tgt |
| |
| ibstor00:~ # cat /proc/scsi_tgt/scsi_tgt |
| Device (host:ch:id:lun or name) Device handler |
| 0:0:0:0 dev_disk |
| 4:0:0:0 dev_disk |
| 5:0:0:0 dev_disk |
| 6:0:0:0 dev_disk |
| 7:0:0:0 dev_disk |
| |
| Now you want to exclude the first scsi disk and expose the last 4 scsi disks as |
| IB/SRP luns for I/O |
| echo "add 4:0:0:0 0" >/proc/scsi_tgt/groups/Default/devices |
| echo "add 5:0:0:0 1" >/proc/scsi_tgt/groups/Default/devices |
| echo "add 6:0:0:0 2" >/proc/scsi_tgt/groups/Default/devices |
| echo "add 7:0:0:0 3" >/proc/scsi_tgt/groups/Default/devices |
| |
| Example 2: working with VDISK FILEIO mode (using md0 device and file 10G-file) |
| a. modprobe scst |
| b. modprobe scst_vdisk |
| c. echo "open vdisk0 /dev/md0" > /proc/scsi_tgt/vdisk/vdisk |
| d. echo "open vdisk1 /10G-file" > /proc/scsi_tgt/vdisk/vdisk |
| e. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices |
| f. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices |
| |
| Example 3: working with VDISK BLOCKIO mode (using md0 device, sda, and cciss/c1d0) |
| a. modprobe scst |
| b. modprobe scst_vdisk |
| c. echo "open vdisk0 /dev/md0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk |
| d. echo "open vdisk1 /dev/sda BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk |
| e. echo "open vdisk2 /dev/cciss/c1d0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk |
| f. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices |
| g. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices |
| h. echo "add vdisk2 2" >/proc/scsi_tgt/groups/Default/devices |
| |
| 2. modprobe ib_srpt |
| |
| |
| B. On initiator machines you can manually do the following steps: |
| 1. modprobe ib_srp |
| 2. ibsrpdm -c (to discover new SRP target) |
| 3. echo <new target info> > /sys/class/infiniband_srp/srp-mthca0-1/add_target |
| 4. fdisk -l (will show new discovered scsi disks) |
| |
| Example: |
| Assume that you use port 1 of first HCA in the system ie. mthca0 |
| |
| [root@lab104 ~]# ibsrpdm -c -d /dev/infiniband/umad0 |
| id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4, |
| dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 |
| [root@lab104 ~]# echo id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4, |
| dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 > |
| /sys/class/infiniband_srp/srp-mthca0-1/add_target |
| |
| OR |
| |
| + You can edit /etc/infiniband/openib.conf to load srp driver and srp HA daemon |
| automatically ie. set SRP_LOAD=yes, and SRPHA_ENABLE=yes |
| + To set up and use high availability feature you need dm-multipath driver |
| and multipath tool |
| + Please refer to OFED-1.x SRP's user manual for more in-details instructions |
| on how-to enable/use HA feature |
| |
| To minimize QUEUE_FULL conditions, you can apply scst_increase_max_tgt_cmds |
| patch from SRPT package from http://sourceforge.net/project/showfiles.php?group_id=110471 |
| |
| |
| Performance notes |
| ----------------- |
| |
| In some cases, for instance working with SSD devices, which consume 100% |
| of a single CPU load for data transfers in their internal threads, to |
| maximize IOPS it can be needed to assign for those threads dedicated |
| CPUs using Linux CPU affinity facilities. No IRQ processing should be |
| done on those CPUs. Check that using /proc/interrupts. See taskset |
| command and Documentation/IRQ-affinity.txt in your kernel's source tree |
| for how to assign CPU affinity to tasks and IRQs. |
| |
| The reason for that is that processing of coming commands in SIRQ context |
| can be done on the same CPUs as SSD devices' threads doing data |
| transfers. As the result, those threads won't receive all the CPU power |
| and perform worse. |
| |
| Alternatively to CPU affinity assignment, you can try to enable SRP |
| target's internal thread. It will allows Linux CPU scheduler to better |
| distribute load among available CPUs. To enable SRP target driver's |
| internal thread you should load ib_srpt module with parameter |
| "thread=1". |
| |
| |
| Send questions about this driver to scst-devel@lists.sourceforge.net, CC: |
| Vu Pham <vuhuong@mellanox.com> and Bart Van Assche <bvanassche@acm.org>. |
