| MaxOutstandingR2T iSCSI parameter and its |
| influence on performance in case of high |
| latency links. |
| |
| |
| Let's consider we have a 1Gbps network between initiator and target with |
| 10ms latency (a good near distance WAN/Internet to another building in |
| the same town). We want to send backup from the initiator to a tape or |
| tape library on the target. We are limited to send only 1 write command |
| at time, because our tape doesn't allow more. We will send 2MB of data |
| in each command. |
| |
| Our initiator and target negotiated typical values InitialR2T No, |
| ImmediateData Yes, FirstBurstLength 65536, MaxBurstLength 262144. Other |
| parameters don't matter for our task, except MaxOutstandingR2T. Let we |
| can run 2 pieces of iSCSI target software on the target: one supporting |
| only MaxOutstandingR2T 1 and another one supporting MaxOutstandingR2T >1. |
| The first target negotiated MaxOutstandingR2T 1, the second one - 16. |
| |
| 10ms means that a 1 byte packet send from the initiator reaches the |
| target in 5ms time. Then 5ms in the opposite direction. 1Gbps bandwidth |
| means that 64K of data transferred from the initiator to the target in |
| about 0.5ms. The maximum throuput we can have with 1Gbps link is about |
| 120MB/s. |
| |
| For sake of simplification we will suppose the targets will process |
| requests and the initiator will process responses quick enough, so we |
| can ignore additional latency the processing on both sides adds. |
| |
| Since each R2T request must contain less than MaxBurstLength data, we |
| need ((2MB - 64K)/256K) = 8 R2T requests to send. The first 64K will be |
| sent as immediate/unsolicited data without explicit R2T request. |
| |
| |
| 1. The first target with MaxOutstandingR2T 1. |
| |
| MaxOutstandingR2T 1 means that only one R2T request can be active on |
| time. I.e., the next request for data transfer can be sent after the |
| previous one completed and all the data received. |
| |
| Thus, on the first target each command will be completed on time: |
| |
| 5 + 0.5 + (5 + 5 + 256K/64K * 0.5) * 7 + (5 + 5 + (256K - 64K)/64K * |
| 0.5) + 5 = 106ms, i.e. 9 IOPS, which is 18MB/s. Remember, we have |
| 120MB/s link. |
| |
| |
| 2. The second target with MaxOutstandingR2T 16. |
| |
| With MaxOutstandingR2T 16 the second target can send all R2T requests at |
| once and the first R2T can be sent before immediate/unsolicited data |
| received. |
| |
| Thus, on the second target each command will be completed on time: |
| |
| 5 + 5 + 5 + 256K/64K * 0.5 * 7 + (256K-64K)/64K * 0.5 + 5 = 35.5ms, |
| i.e. 28 IOPS, which is 56MB/s. |
| |
| |
| Thus, the second target with MaxOutstandingR2T 16 will perform on 56/18 = 311% |
| better that the first target with MaxOutstandingR2T 1. |