| iSCSI SCST target driver |
| ======================== |
| |
| Version 3.5.0, 21 December 2020 |
| ---------------------------- |
| |
| ISCSI-SCST is a deeply reworked fork of iSCSI Enterprise Target (IET) |
| (http://iscsitarget.sourceforge.net). Reasons of the fork were: |
| |
| - To be able to use full power of SCST core. |
| |
| - To fix all the problems, corner cases issues and iSCSI standard |
| violations which IET has. |
| |
| See for more info http://iscsi-scst.sourceforge.net. |
| |
| This version is compatible with SCST version 2.0.0 and higher. |
| |
| |
| Installation if your Linux kernel already has iSCSI-SCST built-in |
| ----------------------------------------------------------------- |
| |
| Simply run "make all", then "make install". |
| |
| |
| Installation out of Linux kernel tree |
| ------------------------------------- |
| |
| See HOWTOs in the doc/ subdirectory. |
| |
| Only vanilla kernels from kernel.org and RHEL/CentOS 5.2 kernels are |
| supported, but it should work on other (vendors') kernels, if you manage |
| to successfully compile on them. The main problem with vendor's kernels |
| is that they often contain patches, which appear only in the next |
| version of the vanilla kernel, therefore it's quite hard to track such |
| changes. Thus, if during compilation for some vendor's kernel your |
| compiler complains about redefinition of some symbol, you should either |
| switch to vanilla kernel, or add or change as necessary the |
| corresponding to that symbol "#if LINUX_VERSION_CODE" statement. |
| |
| Kernel versions 2.6.26 and higher are supported. |
| |
| If during compilation you see message like "*** No rule to make target |
| `xxx.h', needed by `yyy.o'. Stop.", then your autogenerated |
| dependencies don't match your compiler configuration anymore. You should |
| run "make extraclean" to remove them. On the next compilation they will |
| be regenerated. |
| |
| If you experience problems during kernel module load or running, check |
| your system and/or kernel logs (or run dmesg command for the few most |
| recent kernel messages). |
| |
| To use full power of TCP zero-copy transmit functions, especially |
| dealing with user space supplied via scst_user module memory, iSCSI-SCST |
| needs to be notified when Linux networking finished data transmission. |
| For that you should enable CONFIG_TCP_ZERO_COPY_TRANSFER_COMPLETION_NOTIFICATION |
| kernel config option. This is highly recommended, but not required. |
| Basically, iSCSI-SCST works fine with an unpatched Linux kernel with the |
| same or better speed as other open source iSCSI targets, including IET, |
| but if you want even better performance you have to patch and rebuild |
| the kernel. Without CONFIG_TCP_ZERO_COPY_TRANSFER_COMPLETION_NOTIFICATION |
| enabled you will just revert to the original behavior of other open |
| source iSCSI targets, when for data transmission: |
| |
| - For in-kernel allocated memory (scst_vdisk and pass-through |
| handlers) usage of SGV cache on transmit path (READ-type commands) |
| will be disabled, but data will still be sent in zero-copy manner. |
| |
| - For user space allocated memory (scst_user handler) all transmitted |
| data will be additionally copied into temporary TCP buffers. The |
| performance hit will be quite noticeable. |
| |
| Usage |
| ----- |
| |
| See HOWTOs in the doc/ subdirectory. |
| |
| If you want to use Intel CRC32 offload and have corresponding hardware, |
| you should load crc32c-intel module. Then iSCSI-SCST will do all digest |
| calculations using this facility. |
| |
| In 2.0.0 usage of iscsi-scstd.conf as well as iscsi-scst-adm utility is |
| obsolete. Use the sysfs interface facilities instead. |
| |
| The flow of iSCSI-SCST inialization should be as the following: |
| |
| 1. Load of SCST and iSCSI-SCST kernel modules with necessary module |
| parameters, if needed. |
| |
| 2. Start iSCSI-SCST service. |
| |
| 3. Configure targets, devices, LUNs, etc. either using scstadmin |
| (recommended), or using the sysfs interface directly as described below. |
| |
| It is recommended to use TEST UNIT READY ("tur") command to check if |
| iSCSI-SCST target is alive in MPIO configurations. |
| |
| Also see SCST README file how to tune for the best performance. |
| |
| CAUTION: Working of target and initiator on the same host isn't fully |
| ======= supported. See SCST README file for details. |
| |
| |
| Migration from the obsolete proc interface |
| ------------------------------------------ |
| |
| Sysfs enabled scstadmin supports old procfs config file format, so with |
| it you should do the following steps to migrate your proc-based |
| configuration to the sysfs interface: |
| |
| 1. Load SCST modules |
| |
| 2. Run "scstadmin -config old_config_file" |
| |
| 3. Start iSCSI-SCST |
| |
| 4. Run "scstadmin -write_config new_config_file" |
| |
| 5. Check new_config_file and make sure it has everything written |
| properly. |
| |
| 6. IMPORTANT! Delete /etc/iscsi-scst.conf and forget about it. |
| |
| 7. Start using "scstadmin -config new_config_file" to configure both |
| SCST and iSCSI-SCST. |
| |
| IMPORTANT: With the sysfs interface the flow of iSCSI-SCST initialization |
| ========= has changed (see above). Now scstadmin should be run as the FINAL |
| step of the initialization and only ONCE. |
| |
| Also it is recommended to convert initiators.allow and initiators.deny |
| files to the corresponding sysfs facilities. |
| |
| |
| Migration from IET |
| ------------------ |
| |
| Scalable Informatics Inc. has developed a pretty simple tool, will |
| automatically generate an /etc/scst.conf file from an /etc/ietd.conf |
| file, and then restart daemons. |
| |
| Tool is here: http://download.scalableinformatics.com/gen_scst_conf/ |
| |
| Just run it on an ietd.conf based iSCSI target: |
| |
| $ chmod +x gen_scst_conf.pl |
| $ sudo ./gen_scst_conf.pl |
| |
| and your IET machine should now largely be an iSCSI-SCST machine. With |
| the caveat of a few IET elements not being supported. |
| |
| Current limitations are one LUN per target, and not all LUN options |
| might be mapped well between the systems. This may change if there is |
| demand. For any questions and inquirys feel free to contact Joe Landman |
| <landman@scalableinformatics.com> (and please provide a few sample |
| ietd.conf files for us to play with). |
| |
| |
| Sysfs interface |
| --------------- |
| |
| The sysfs build supports only kernels 2.6.26 and higher, because in |
| 2.6.26 internal kernel's sysfs interface had a major change, which made |
| it heavily incompatible with pre-2.6.26 version. |
| |
| Root of SCST sysfs interface is /sys/kernel/scst_tgt. Root of iSCSI-SCST |
| is /sys/kernel/scst_tgt/targets/iscsi. It has the following entries: |
| |
| - None, one or more subdirectories for targets with name equal to names |
| of the corresponding targets. |
| |
| - IncomingUser[num] - optional one or more attributes containing user |
| name and password for incoming discovery user name. Not exist by |
| default and can be added through "mgmt" entry, see below. |
| |
| - OutgoingUser - optional attribute containing user name and password |
| for outgoing discovery user name. Not exist by default and can be |
| added through "mgmt" entry, see below. |
| |
| - iSNSServer - contains name or IP address of iSNS server with optional |
| "AccessControl" attribute, which allows to enable iSNS access |
| control. Empty by default. |
| |
| - allowed_portal[num] - optional attribute, which specifies, on which |
| portals (target's IP addresses) this target will be available. If not |
| specified (default) the target will be available on all all portals. |
| As soon as at least one allowed_portal specified, the target will be |
| accessible for initiators only on the specified portals. There might |
| be any number of the allowed_portal attributes. The portals |
| specification in the allowed_portal attributes can be a simple |
| DOS-type patterns, containing '*' and '?' symbols. '*' means match |
| all any symbols, '?' means match only any single symbol. For |
| instance, "10.170.77.2" will match "10.170.7?.*". Additionally, you |
| can use negative sign '!' to revert the value of the pattern. For |
| instance, "10.170.67.2" will match "!10.170.7?.*". See examples |
| below. |
| |
| - enabled - using this attribute you can enable or disable iSCSI-SCST |
| accept new connections. It allows to finish configuring global |
| iSCSI-SCST attributes before it starts accepting new connections. 0 |
| by default. |
| |
| - open_state - read-only attribute, which allows to see if the user |
| space part of iSCSI-SCST connected to the kernel part. |
| |
| - per_portal_acl - if set, makes iSCSI-SCST work in the per-portal |
| access control mode. In this mode iSCSI-SCST registers all initiators |
| in SCST core as "initiator_name#portal_IP_address" pattern, like |
| "iqn.2006-10.net.vlnb:ini#10.170.77.2" for initiator |
| iqn.2006-10.net.vlnb connected through portal 10.170.77.2. This mode |
| allows to make particular initiators be able to use only particular |
| portals on the target and don't see/be able to connect through |
| others. See below for more details. |
| |
| - trace_level - allows to enable and disable various tracing |
| facilities. See content of this file for help how to use it. |
| |
| - version - read-only attribute, which allows to see version of |
| iSCSI-SCST and enabled optional features. |
| |
| - mgmt - main management entry, which allows to configure iSCSI-SCST. |
| Namely, add/delete targets as well as add/delete optional global and |
| per-target attributes. See content of this file for help how to use |
| it. |
| |
| Each iSCSI-SCST sysfs file (attribute) can contain in the last line mark |
| "[key]". It is automatically added mark used to allow scstadmin to see |
| which attributes it should save in the config file. You can ignore it. |
| |
| Each target subdirectory contains the following entries: |
| |
| - ini_groups - subdirectory defining initiator groups for this target, |
| used to define per-initiator access control. See SCST core README for |
| more details. |
| |
| - luns - subdirectory defining LUNs of this target. See SCST core |
| README for more details. |
| |
| - sessions - subdirectory containing connected to this target sessions. |
| |
| - IncomingUser[num] - optional one or more attributes containing user |
| name and password for incoming user name. Not exist by default and can |
| be added through the "mgmt" entry, see above. |
| |
| - OutgoingUser - optional attribute containing user name and password |
| for outgoing user name. Not exist by default and can be added through |
| the "mgmt" entry, see above. |
| |
| - Entries defining default iSCSI parameters values used during iSCSI |
| parameters negotiation. Only entries which can be changed or make |
| sense are listed there. |
| |
| - QueuedCommands - defines maximum number of commands queued to any |
| session of this target. Default is 32 commands. |
| |
| - NopInInterval - defines interval between NOP-In requests, which the |
| target will send on idle connections to check if the initiator is |
| still alive. If there is no NOP-Out reply from the initiator in |
| NopInTimeout seconds, the corresponding connection will be closed. Default |
| is 30 seconds. If it's set to 0, then NOP-In requests are disabled. |
| |
| - NopInTimeout - defines the maximum time in seconds a NOP-In request |
| can wait for response from initiator, otherwise the corresponding |
| connection will be closed. Default is 30 seconds. |
| |
| - RspTimeout - defines the maximum time in seconds a command can wait for |
| response from initiator, otherwise the corresponding connection will |
| be closed. Default is 90 seconds. |
| |
| - enabled - using this attribute you can enable or disable iSCSI-SCST |
| accept new connections to this target. It allows to finish |
| configuring it before it starts accepting new connections. 0 by |
| default. |
| |
| - redirect - allows to temporarily or permanently redirect login to the |
| target to another portal. Discovery sessions will not be impacted, |
| but normal sessions will be redirected before security negotiation. |
| The destination should be specified using format "<ip_addr>[:port] temp|perm". |
| IPv6 addresses need to be enclosed in [] brackets. To remove |
| redirection, provide an empty string. For example: |
| echo "10.170.77.2:32600 temp" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/redirect |
| will temporarily redirect login to portal 10.170.77.2 and port 32600. |
| |
| - tid - TID of this target. |
| |
| The "sessions" subdirectory contains the following attribute: |
| |
| - thread_pid - the process identifiers (PIDs) of the iscsird and iscsiwr |
| threads that process SCSI commands associated with this session. |
| |
| Additionally, the "sessions" subdirectory contains one subdirectory for each |
| connected session with name equal to name of the connected initiator. |
| |
| Each session subdirectory contains the following entries: |
| |
| - One subdirectory for each TCP connection in this session. ISCSI-SCST |
| supports 1 connection per session, but the session subdirectory can |
| contain several connections: one active and other being closed. |
| |
| - Entries defining negotiated iSCSI parameters. Only parameters which |
| can be changed or make sense are listed there. |
| |
| - initiator_name - contains initiator name |
| |
| - sid - contains SID of this session |
| |
| - reinstating - contains reinstatement state of this session |
| |
| - force_close - write-only attribute, which allows to force close this |
| session. This is the only writable session attribute. |
| |
| - active_commands - contains number of active, i.e. not yet or being |
| executed, SCSI commands in this session. |
| |
| - commands - contains overall number of SCSI commands in this session. |
| |
| - thread_pid - Process IDs (PIDs) of the iscsi{wr,rd} kernel threads that |
| process the SCSI commands for this session. |
| |
| Each connection subdirectory contains the following entries: |
| |
| - cid - contains CID of this connection. |
| |
| - ip - contains IP address of the connected initiator. |
| |
| - state - contains processing state of this connection. |
| |
| Each initiator group subdirectory contains: |
| |
| - per_sess_dedicated_tgt_threads - if set, each iSCSI session has |
| dedicated, i.e. not shared with other sessions, pool of the |
| iscsi{wr,rd} kernel threads. Useful to control per-session CPU |
| affinity to improve performance. Default: not set. |
| |
| See SCST README for info about other attributes. |
| |
| Below is a sample script, which configures 1 virtual disk "disk1" using |
| /disk1 image and one target iqn.2006-10.net.vlnb:tgt with all default |
| parameters: |
| |
| #!/bin/bash |
| |
| modprobe scst |
| modprobe scst_vdisk |
| |
| echo "add_device disk1 filename=/disk1; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt |
| |
| service iscsi-scst start |
| |
| echo "add_target iqn.2006-10.net.vlnb:tgt" >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| echo "add disk1 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt |
| |
| echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled |
| echo 1 >/sys/kernel/scst_tgt/targets/iscsi/enabled |
| |
| Below is another sample script, which configures 1 real local SCSI disk |
| 0:0:1:0 and one target iqn.2006-10.net.vlnb:tgt with all default parameters: |
| |
| #!/bin/bash |
| |
| modprobe scst |
| modprobe scst_disk |
| |
| echo "add_device 0:0:1:0" >/sys/kernel/scst_tgt/handlers/dev_disk/mgmt |
| |
| service iscsi-scst start |
| |
| echo "add_target iqn.2006-10.net.vlnb:tgt" >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| echo "add 0:0:1:0 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt |
| |
| echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled |
| echo 1 >/sys/kernel/scst_tgt/targets/iscsi/enabled |
| |
| Below is an advanced sample script, which configures more virtual |
| devices of various types, including virtual CDROM and 2 targets, one |
| with all default parameters, another one with some not default |
| parameters, incoming and outgoing user names for CHAP authentication, |
| and special permissions for initiator iqn.2005-03.org.open-iscsi:cacdcd2520, |
| which will see another set of devices. Also this sample configures CHAP |
| authentication for discovery sessions and iSNS server with access |
| control. |
| |
| #!/bin/bash |
| |
| modprobe scst |
| modprobe scst_vdisk |
| |
| echo "add_device disk1 filename=/disk1; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt |
| echo "add_device disk2 filename=/disk2; blocksize=4096; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt |
| echo "add_device blockio filename=/dev/sda5" >/sys/kernel/scst_tgt/handlers/vdisk_blockio/mgmt |
| echo "add_device nullio" >/sys/kernel/scst_tgt/handlers/vdisk_nullio/mgmt |
| echo "add_device cdrom" >/sys/kernel/scst_tgt/handlers/vcdrom/mgmt |
| |
| service iscsi-scst start |
| |
| echo "192.168.1.16 AccessControl" >/sys/kernel/scst_tgt/targets/iscsi/iSNSServer |
| echo "add_attribute IncomingUser joeD 12charsecret" >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| echo "add_attribute OutgoingUser jackD 12charsecret1" >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| |
| echo "add_target iqn.2006-10.net.vlnb:tgt" >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| |
| echo "add disk1 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt |
| echo "add cdrom 1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt |
| |
| echo "add_target iqn.2006-10.net.vlnb:tgt1" >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 IncomingUser1 joe2 12charsecret2" >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 IncomingUser joe 12charsecret" >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 OutgoingUser jim1 12charpasswd" >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| echo "No" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/InitialR2T |
| echo "Yes" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ImmediateData |
| echo "8192" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxRecvDataSegmentLength |
| echo "8192" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxXmitDataSegmentLength |
| echo "131072" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxBurstLength |
| echo "32768" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/FirstBurstLength |
| echo "1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxOutstandingR2T |
| echo "CRC32C,None" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/HeaderDigest |
| echo "CRC32C,None" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/DataDigest |
| echo "32" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/QueuedCommands |
| |
| echo "add disk2 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/mgmt |
| echo "add nullio 26" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/mgmt |
| |
| echo "create special_ini" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/mgmt |
| echo "add blockio 0 read_only=1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/luns/mgmt |
| echo "add iqn.2005-03.org.open-iscsi:cacdcd2520" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/initiators/mgmt |
| |
| echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled |
| echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/enabled |
| |
| echo 1 >/sys/kernel/scst_tgt/targets/iscsi/enabled |
| |
| The resulting overall SCST sysfs hierarchy with an initiator connected to |
| both iSCSI-SCST targets will look like: |
| |
| /sys/kernel/scst_tgt |
| |-- devices |
| | |-- blockio |
| | | |-- blocksize |
| | | |-- exported |
| | | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/luns/0 |
| | | |-- filename |
| | | |-- handler -> ../../handlers/vdisk_blockio |
| | | |-- nv_cache |
| | | |-- read_only |
| | | |-- removable |
| | | |-- resync_size |
| | | |-- size_mb |
| | | |-- t10_dev_id |
| | | |-- threads_num |
| | | |-- threads_pool_type |
| | | |-- type |
| | | `-- usn |
| | |-- cdrom |
| | | |-- exported |
| | | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/1 |
| | | |-- filename |
| | | |-- handler -> ../../handlers/vcdrom |
| | | |-- size_mb |
| | | |-- t10_dev_id |
| | | |-- threads_num |
| | | |-- threads_pool_type |
| | | |-- type |
| | | `-- usn |
| | |-- disk1 |
| | | |-- blocksize |
| | | |-- exported |
| | | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/0 |
| | | |-- filename |
| | | |-- handler -> ../../handlers/vdisk_fileio |
| | | |-- nv_cache |
| | | |-- o_direct |
| | | |-- read_only |
| | | |-- removable |
| | | |-- resync_size |
| | | |-- size_mb |
| | | |-- t10_dev_id |
| | | |-- threads_num |
| | | |-- threads_pool_type |
| | | |-- type |
| | | |-- usn |
| | | `-- write_through |
| | |-- disk2 |
| | | |-- blocksize |
| | | |-- exported |
| | | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/0 |
| | | |-- filename |
| | | |-- handler -> ../../handlers/vdisk_fileio |
| | | |-- nv_cache |
| | | |-- o_direct |
| | | |-- read_only |
| | | |-- removable |
| | | |-- resync_size |
| | | |-- size_mb |
| | | |-- t10_dev_id |
| | | |-- threads_num |
| | | |-- threads_pool_type |
| | | |-- type |
| | | |-- usn |
| | | `-- write_through |
| | `-- nullio |
| | |-- blocksize |
| | |-- exported |
| | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/26 |
| | |-- handler -> ../../handlers/vdisk_nullio |
| | |-- read_only |
| | |-- removable |
| | |-- size_mb |
| | |-- t10_dev_id |
| | |-- threads_num |
| | |-- threads_pool_type |
| | |-- type |
| | `-- usn |
| |-- handlers |
| | |-- vcdrom |
| | | |-- cdrom -> ../../devices/cdrom |
| | | |-- mgmt |
| | | |-- trace_level |
| | | `-- type |
| | |-- vdisk_blockio |
| | | |-- blockio -> ../../devices/blockio |
| | | |-- mgmt |
| | | |-- trace_level |
| | | `-- type |
| | |-- vdisk_fileio |
| | | |-- disk1 -> ../../devices/disk1 |
| | | |-- disk2 -> ../../devices/disk2 |
| | | |-- mgmt |
| | | |-- trace_level |
| | | `-- type |
| | `-- vdisk_nullio |
| | |-- mgmt |
| | |-- nullio -> ../../devices/nullio |
| | |-- trace_level |
| | `-- type |
| |-- sgv |
| | |-- global_stats |
| | |-- sgv |
| | | `-- stats |
| | |-- sgv-clust |
| | | `-- stats |
| | `-- sgv-dma |
| | `-- stats |
| |-- targets |
| | `-- iscsi |
| | |-- IncomingUser |
| | |-- OutgoingUser |
| | |-- enabled |
| | |-- iSNSServer |
| | |-- iqn.2006-10.net.vlnb:tgt |
| | | |-- DataDigest |
| | | |-- FirstBurstLength |
| | | |-- HeaderDigest |
| | | |-- ImmediateData |
| | | |-- InitialR2T |
| | | |-- MaxBurstLength |
| | | |-- MaxOutstandingR2T |
| | | |-- MaxRecvDataSegmentLength |
| | | |-- MaxXmitDataSegmentLength |
| | | |-- NopInInterval |
| | | |-- QueuedCommands |
| | | |-- RspTimeout |
| | | |-- enabled |
| | | |-- ini_groups |
| | | | `-- mgmt |
| | | |-- luns |
| | | | |-- 0 |
| | | | | |-- device -> ../../../../../devices/disk1 |
| | | | | `-- read_only |
| | | | |-- 1 |
| | | | | |-- device -> ../../../../../devices/cdrom |
| | | | | `-- read_only |
| | | | `-- mgmt |
| | | |-- per_portal_acl |
| | | |-- redirect |
| | | |-- rel_tgt_id |
| | | |-- sessions |
| | | | `-- iqn.2005-03.org.open-iscsi:cacdcd2520 |
| | | | |-- 10.170.75.2 |
| | | | | |-- cid |
| | | | | |-- ip |
| | | | | `-- state |
| | | | |-- DataDigest |
| | | | |-- FirstBurstLength |
| | | | |-- HeaderDigest |
| | | | |-- ImmediateData |
| | | | |-- InitialR2T |
| | | | |-- MaxBurstLength |
| | | | |-- MaxOutstandingR2T |
| | | | |-- MaxRecvDataSegmentLength |
| | | | |-- MaxXmitDataSegmentLength |
| | | | |-- active_commands |
| | | | |-- commands |
| | | | |-- force_close |
| | | | |-- initiator_name |
| | | | |-- luns -> ../../luns |
| | | | |-- reinstating |
| | | | `-- sid |
| | | `-- tid |
| | |-- iqn.2006-10.net.vlnb:tgt1 |
| | | |-- DataDigest |
| | | |-- FirstBurstLength |
| | | |-- HeaderDigest |
| | | |-- ImmediateData |
| | | |-- IncomingUser |
| | | |-- IncomingUser1 |
| | | |-- InitialR2T |
| | | |-- MaxBurstLength |
| | | |-- MaxOutstandingR2T |
| | | |-- MaxRecvDataSegmentLength |
| | | |-- MaxXmitDataSegmentLength |
| | | |-- OutgoingUser |
| | | |-- NopInInterval |
| | | |-- QueuedCommands |
| | | |-- RspTimeout |
| | | |-- enabled |
| | | |-- ini_groups |
| | | | |-- mgmt |
| | | | `-- special_ini |
| | | | |-- initiators |
| | | | | |-- iqn.2005-03.org.open-iscsi:cacdcd2520 |
| | | | | `-- mgmt |
| | | | `-- luns |
| | | | |-- 0 |
| | | | | |-- device -> ../../../../../../../devices/blockio |
| | | | | `-- read_only |
| | | | `-- mgmt |
| | | |-- luns |
| | | | |-- 0 |
| | | | | |-- device -> ../../../../../devices/disk2 |
| | | | | `-- read_only |
| | | | |-- 26 |
| | | | | |-- device -> ../../../../../devices/nullio |
| | | | | `-- read_only |
| | | | `-- mgmt |
| | | |-- per_portal_acl |
| | | |-- redirect |
| | | |-- rel_tgt_id |
| | | |-- sessions |
| | | | `-- iqn.2005-03.org.open-iscsi:cacdcd2520 |
| | | | |-- 10.170.75.2 |
| | | | | |-- cid |
| | | | | |-- ip |
| | | | | `-- state |
| | | | |-- DataDigest |
| | | | |-- FirstBurstLength |
| | | | |-- HeaderDigest |
| | | | |-- ImmediateData |
| | | | |-- InitialR2T |
| | | | |-- MaxBurstLength |
| | | | |-- MaxOutstandingR2T |
| | | | |-- MaxRecvDataSegmentLength |
| | | | |-- MaxXmitDataSegmentLength |
| | | | |-- active_commands |
| | | | |-- commands |
| | | | |-- force_close |
| | | | |-- initiator_name |
| | | | |-- luns -> ../../ini_groups/special_ini/luns |
| | | | |-- reinstating |
| | | | `-- sid |
| | | `-- tid |
| | |-- mgmt |
| | |-- open_state |
| | |-- trace_level |
| | `-- version |
| |-- threads |
| |-- trace_level |
| `-- version |
| |
| |
| Configuring iscsi-scst via scstadmin |
| ------------------------------------ |
| |
| All iscsi-scst parameters that were configured in the previous section via the |
| sysfs interface can also be set via scstadmin. Here is an example that shows |
| how to configure the IP address of the iSNSServer and to enable iSNS access |
| control at the same time: |
| |
| # scstadmin -noprompt -set_drv_attr iscsi -attributes iSNSServer="192.168.1.16 AccessControl" |
| |
| Collecting current configuration: done. |
| |
| |
| -> Making requested changes. |
| -> Done, 0 change(s) made. |
| |
| All done. |
| |
| # scstadmin -list_drv_attr iscsi |
| |
| Collecting current configuration: done. |
| |
| Attribute Value Writable KEY |
| ------------------------------------------------------------ |
| iSNSServer 127.0.0.1 AccessControl Yes Yes |
| |
| Dynamic attributes available |
| ---------------------------- |
| OutgoingUser |
| IncomingUser |
| |
| All done. |
| |
| |
| Advanced initiators access control |
| ---------------------------------- |
| |
| ISCSI-SCST allows you to optionally control visibility and accessibility |
| of your target and its portals (IP addresses) to remote initiators. This |
| control includes both the target's portals SendTargets discovery as well |
| as regular LUNs access. |
| |
| This facility supersedes the obsolete initiators.[allow,deny] method, |
| which is going to be removed in one of the future versions. |
| |
| This facility is available only in the sysfs build of iSCSI-SCST. |
| |
| By default, all portals are available for the initiators. |
| |
| 1. If you want to enable/disable one or more target's portals for all |
| initiators, you should define one ore more allowed_portal attributes. |
| For example: |
| |
| echo 'add_target_attribute iqn.2006-10.net.vlnb:tgt allowed_portal 10.170.77.2' >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| |
| will enable only portal 10.170.77.2 and disable all other portals |
| |
| echo 'add_target_attribute iqn.2006-10.net.vlnb:tgt allowed_portal 10.170.77.2' >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| echo 'add_target_attribute iqn.2006-10.net.vlnb:tgt allowed_portal 10.170.75.2' >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| |
| will enable only portals 10.170.77.2 and 10.170.75.2 and disable all |
| other portals. |
| |
| echo 'add_target_attribute iqn.2006-10.net.vlnb:tgt allowed_portal 10.170.7?.2' >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| |
| will enable only portals 10.170.7x.2 and disable all other portals. |
| |
| echo 'add_target_attribute iqn.2006-10.net.vlnb:tgt allowed_portal !*' >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| |
| will disable all portals. |
| |
| 2. If you want to want to allow only only specific set of initiators be |
| able to connect to your target, you should don't add any default LUNs |
| for the target and create for allowed initiators a security group to |
| which they will be assigned. |
| |
| For example, we want initiator iqn.2005-03.org.vlnb:cacdcd2520 and only |
| it be able to access target iqn.2006-10.net.vlnb:tgt: |
| |
| echo 'add_target iqn.2006-10.net.vlnb:tgt' >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| echo 'create allowed_ini' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/mgmt |
| echo 'add dev1 0' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/allowed_ini/luns/mgmt |
| echo 'add iqn.2005-03.org.vlnb:cacdcd2520' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/allowed_ini/initiators/mgmt |
| echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled |
| |
| Since there will be no default LUNs for the target, all initiators other |
| than iqn.2005-03.org.vlnb:cacdcd2520 will be blocked from accessing it. |
| |
| Alternatively, you can create an empty security group and filter out in |
| it all initiators except the allowed one: |
| |
| echo 'add_target iqn.2006-10.net.vlnb:tgt' >/sys/kernel/scst_tgt/targets/iscsi/mgmt |
| echo 'add dev1 0' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt |
| echo 'create denied_inis' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/mgmt |
| echo 'add !iqn.2005-03.org.vlnb:cacdcd2520' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/denied_inis/initiators/mgmt |
| echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled |
| |
| 3. If you want to enable/disable one or more target's portals for |
| particular initiators, you should set per_portal_acl attribute to 1 and |
| specify SCST access control to those initiators. If an SCST security |
| group doesn't have any LUNs, all the initiator, which should be assigned |
| to it, will not see this target and/or its portal. For example: |
| |
| (We assume that an empty group "BLOCKING_GROUP" is already created by for |
| target iqn.2006-10.net.vlnb:tgt by command (see above for more information): |
| "echo 'create BLOCKING_GROUP' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/mgmt) |
| |
| echo 'add iqn.2005-03.org.vlnb:cacdcd2520#10.170.77.2' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/BLOCKING_GROUP/initiators/mgmt |
| |
| will block access of initiator iqn.2005-03.org.vlnb:cacdcd2520 to |
| target iqn.2006-10.net.vlnb:tgt portal 10.170.77.2. |
| |
| Another example: |
| |
| echo 'add iqn.2005-03.org.vlnb:cacdcd2520*' >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/ini_groups/BLOCKING_GROUP/initiators/mgmt |
| |
| will block access of initiator iqn.2005-03.org.vlnb:cacdcd2520 to |
| all target iqn.2006-10.net.vlnb:tgt portals. |
| |
| |
| Troubleshooting |
| --------------- |
| |
| If you have any problems, start troubleshooting from looking at the |
| kernel and system logs. In the kernel log iSCSI-SCST and SCST core send |
| their messages, in the system log iscsi-scstd sends its messages. In |
| most Linux distributions both those logs are put to /var/log/messages |
| file. |
| |
| Then, it might be helpful to increase level of logging. For kernel |
| modules you should make the debug build, by either running "make 2debug" |
| if you work with SCST SVN tree, or by enabling the corresponding |
| debug symbols (see below). |
| |
| If after looking on the logs the reason of your problem is still unclear |
| for you, report to SCST mailing list scst-devel@lists.sourceforge.net. |
| |
| |
| Work if target's backstorage or link is too slow |
| ------------------------------------------------ |
| |
| In some cases you can experience I/O stalls or see in the kernel log |
| abort or reset messages. It can happen under high I/O load, when your |
| target's backstorage gets overloaded, or working over a slow link, when |
| the link can't serve all the queued commands on time, |
| |
| To workaround it you can reduce QueuedCommands parameter for the |
| corresponding target to some lower value, like 8 (default is 32). |
| |
| Also see SCST README file for more details about that issue and ways to |
| prevent it. |
| |
| |
| Performance advices |
| ------------------- |
| |
| 1. If you use Windows XP or Windows 2003+ as initiators, you can |
| consider to decrease TcpAckFrequency parameter to 1. See |
| http://support.microsoft.com/kb/328890/ or google for "TcpAckFrequency" |
| for more details. |
| |
| 2. See how to get the maximum throughput from iSCSI, for instance, at |
| http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html. |
| It's about VMware, but its recommendations apply to other environments |
| as well. |
| |
| 3. ISCSI initiators from pre-CentOS/RHEL 5 reported to have some |
| performance problems. If you use it, it is strongly advised to upgrade. |
| |
| 4. If you are going to use your target in an VM environment, for |
| instance as a shared storage with VMware, make sure all your VMs |
| connected to the target via *separate* sessions, i.e. each VM has own |
| connection to the target, not all VMs connected using a single |
| connection. You can check it using SCST proc or sysfs interface. If you |
| miss it, you can greatly loose performance of parallel access to your |
| target from different VMs. This isn't related to the case if your VMs |
| are using the same shared storage, like with VMFS, for instance. In this |
| case all your VM hosts will be connected to the target via separate |
| sessions, which is enough. |
| |
| 5. Many dual port network adapters are not able to transfer data |
| simultaneously on both ports, i.e. they transfer data via both ports on |
| the same speed as via any single port. Thus, using such adapters in MPIO |
| configuration can't improve performance. To allow MPIO to have double |
| performance you should either use separate network adapters, or find a |
| dual-port adapter capable to to transfer data simultaneously on both |
| ports. You can check it by running 2 iperf's through both ports in |
| parallel. |
| |
| 6. Since network offload works much better in the write direction, than |
| for reading (simplifying, in the read direction often there's additional |
| data copy) in many cases with 10GbE in a single initiator-target pair |
| the initiator's CPU is a bottleneck, so you can see the initiator can |
| read data on much slower rate, than write. You can check it by watching |
| *each particular* CPU load to find out if any of them is close to 100% |
| load, including IRQ processing load. Note, many tools like vmstat give |
| aggregate load on all CPUs, so with 4 cores 25% corresponds to 100% load |
| of any single CPU. |
| |
| 7. For high speed network adapters it can be better if you configure |
| them to serve connections, e.g., from initiator on CPU0 and from |
| initiator Y on CPU1. Then you can bind threads processing them also to |
| CPU0 and CPU1 correspondingly using cpu_mask attribute of their targets |
| or security groups. In NUMA-like configurations it can signficantly |
| boost IOPS performance. |
| |
| 8. See SCST core's README for more advices. Especially pay attention to |
| have io_grouping_type option set correctly. |
| |
| |
| Compilation options |
| ------------------- |
| |
| There are the following compilation options, that could be commented |
| in/out in the kernel's module Makefile: |
| |
| - CONFIG_SCST_DEBUG - turns on some debugging code, including some logging. |
| Makes the driver considerably bigger and slower, producing large amount of |
| log data. |
| |
| - CONFIG_SCST_TRACING - turns on ability to log events. Makes the driver |
| considerably bigger and leads to some performance loss. |
| |
| - CONFIG_SCST_EXTRACHECKS - adds extra validity checks in the various places. |
| |
| - CONFIG_SCST_ISCSI_DEBUG_DIGEST_FAILURES - simulates digest failures in |
| random places. |
| |
| |
| Credits |
| ------- |
| |
| Thanks to: |
| |
| * Ming Zhang <blackmagic02881@gmail.com> for fixes |
| |
| * Krzysztof Blaszkowski <kb@sysmikro.com.pl> for many fixes |
| |
| * Alexey Kuznetsov <kuznet@ms2.inr.ac.ru> for comments and help in |
| debugging |
| |
| * Tomasz Chmielewski <mangoo@wpkg.org> for testing and suggestions |
| |
| * Bart Van Assche <bvanassche@acm.org> for a lot of help |
| |
| Vladislav Bolkhovitin <vst@vlnb.net>, http://scst.sourceforge.net |
| |
