| iSCSI extensions for RDMA driver |
| ================================ |
| |
| Installation & Configuration: |
| --------------------------- |
| For installation and configuration, see iscsi README. |
| There are no specific configuration options for iSER. |
| See below for performance optimizations as well as troubleshooting. |
| There is also a HOWTO on http://community.mellanox.com/docs/DOC-1479 |
| |
| Performance considerations: |
| --------------------------- |
| |
| In order to achieve better performance, it is recommended to specify |
| "QueuedCommands 128" parameter per iSER target, since the transport |
| is very fast and you usually want to connect it to fast backstorage. |
| |
| For performance tuning of initiator and target machines, see |
| http://community.mellanox.com/docs/DOC-1483 |
| |
| Note that if you have an SSD controller that is close to a particular |
| NUMA node, you want the HCA to be close to the same node. |
| |
| Limitations: |
| ------------- |
| * Bidirectional commands are not supported |
| * Maximum number of concurent login requests that can be handled is 127 by default. |
| Note that there may be more connections, but only up to 127 login requests |
| can be handled at the same time. If you wish to increase this, load isert_scst with |
| module parameter isert_nr_devs set to the number of login requests you need to handle. |
| |
| |
| Troubleshooting: |
| ----------------- |
| * Initiator fails to connect to target. The following message is seen in dmesg: |
| Failed to accept conn request, err: -22 |
| The cause of this is often compilation issues if you have OFED or MLNX_OFED installed: |
| If you are compiling for OFED/MLNX_OFED, make sure OFED is installed for |
| the kernel you are running. Also, make sure you followed ALL steps described |
| in README.iser_ofed. |
| If you are compiling for non-OFED kernel, make sure you don't have |
| OFED/MLNX_OFED installed. |
| |
| |
| * Discovery of iSER targets takes a long time or login to all discovered targets fails. |
| iSCSI discovery does not have a way to determine between iSCSI and iSER |
| enabled portals. Thus, initiator tries to connect to all interfaces it |
| discovered (by default discovery is done over iSCSI TCP). |
| In order to prevent this behaviour, you should specify |
| "allowed_portal <target interface IP>" parameter for each target you want |
| to export through specific RDMA capable adapters. |
| |
| |
| * Initiator keeps connecting and disconnecting from target in a loop |
| with constant interval after target reboot. |
| The problem may be that connection requests from initiator are received |
| on wrong port/HCA. This can be one due to one (or both) of the following issues: |
| 1) net.ipv4.conf.all.arp_ignore sysclt is not set to 2 |
| rdma-cm relies on ARP responses being received on the same interface |
| that sent the request. Linux default does not do that. |
| In order to make Linux behave good for rdma-cm, you _MUST_ add |
| "net.ipv4.conf.all.arp_ignore = 2" to /etc/sysctl.conf |
| 2) You have more than 1 HCA and PCI mappings to netdev devices is not |
| persistent between reboots. Possible solution is to have udev rules |
| for mapping the ibX devices in persistent way. |
| See below for udev scripts example: |
| |
| /lib/udev/net.sh |
| ------------------- |
| #!/bin/sh |
| |
| . /etc/sysconfig/net.conf |
| |
| type_fd="/sys/${DEVPATH}/type" |
| if [ ! -f $type_fd ]; then |
| exit |
| fi |
| type=`cat /sys/${DEVPATH}/type` |
| |
| if [ "$type" = "32" ]; then # IPoIB interface |
| i=0 |
| CONFDEV="DEV${i}" |
| CONFPCI=${!CONFDEV} |
| PCI=`basename $PHYSDEVPATH` |
| while [ -n "$CONFPCI" ]; do |
| if [ "$CONFPCI" = "$PCI" ]; then |
| devid=$(printf "%d\n" `cat /sys/$DEVPATH/dev_id`) |
| let id=$i*2+$devid |
| DEV="ib$id" |
| echo "$DEV" |
| exit |
| fi |
| let i=i+1 |
| CONFDEV="DEV$i" |
| CONFPCI=${!CONFDEV} |
| done |
| fi |
| |
| /etc/sysconfig/net.conf |
| ----------------------- |
| DEV0="0000:01:00.0" |
| DEV1="0000:02:00.0" |
| |
| /etc/udev/rules.d/90-network.rules |
| ------------------------------------- |
| ACTION=="add", SUBSYSTEM=="net", PROGRAM="/lib/udev/net.sh", RESULT=="?*", NAME="$result" |
| |
| |
| * Login to all targets from initiator sometimes times out. |
| It may be a network problem (try running tools like ibdiagnet |
| and rping between target and initiator hosts). The description of those tools |
| is beyond the scope of this readme. |
| Another issue may be that you failed to set net.ipv4.conf.all.arp_ignore sysctl |
| to the value of 2 (see above problem for more detailed explanation). |
| |
| |
| * When running IO, latency is getting higher and higher all the time. |
| If you have enabled intel_iommu either in kernel command line or in |
| kernel config (it may be enabled by default), you should specify |
| iommu=pt on kernel command line to avoid the latency issue. |
| |
| |