		MaxOutstandingR2T iSCSI parameter and its
		influence on performance in case of high
		latency links.


Let's consider we have a 1Gbps network between initiator and target with
10ms latency (a good near distance WAN/Internet to another building in
the same town). We want to send backup from the initiator to a tape or
tape library on the target. We are limited to send only 1 write command
at time, because our tape doesn't allow more. We will send 2MB of data
in each command.

Our initiator and target negotiated typical values InitialR2T No,
ImmediateData Yes, FirstBurstLength 65536, MaxBurstLength 262144. Other
parameters don't matter for our task, except MaxOutstandingR2T. Let we
can run 2 pieces of iSCSI target software on the target: one supporting
only MaxOutstandingR2T 1 and another one supporting MaxOutstandingR2T >1.
The first target negotiated MaxOutstandingR2T 1, the second one - 16.

10ms means that a 1 byte packet send from the initiator reaches the
target in 5ms time. Then 5ms in the opposite direction. 1Gbps bandwidth
means that 64K of data transferred from the initiator to the target in
about 0.5ms. The maximum throuput we can have with 1Gbps link is about
120MB/s.

For sake of simplification we will suppose the targets will process
requests and the initiator will process responses quick enough, so we
can ignore additional latency the processing on both sides adds.

Since each R2T request must contain less than MaxBurstLength data, we
need ((2MB - 64K)/256K) = 8 R2T requests to send. The first 64K will be
sent as immediate/unsolicited data without explicit R2T request.


1. The first target with MaxOutstandingR2T 1.

MaxOutstandingR2T 1 means that only one R2T request can be active on
time. I.e., the next request for data transfer can be sent after the
previous one completed and all the data received.

Thus, on the first target each command will be completed on time:

5 + 0.5 + (5 + 5 + 256K/64K * 0.5) * 7 + (5 + 5 + (256K - 64K)/64K *
0.5) + 5 = 106ms, i.e. 9 IOPS, which is 18MB/s. Remember, we have
120MB/s link.


2. The second target with MaxOutstandingR2T 16.

With MaxOutstandingR2T 16 the second target can send all R2T requests at
once and the first R2T can be sent before immediate/unsolicited data
received.

Thus, on the second target each command will be completed on time:

5 + 5 + 5 + 256K/64K * 0.5 * 7 + (256K-64K)/64K * 0.5 + 5 = 35.5ms,
i.e. 28 IOPS, which is 56MB/s.


Thus, the second target with MaxOutstandingR2T 16 will perform on 56/18 = 311%
better that the first target with MaxOutstandingR2T 1.