librdmacm/docs/rsocket - linux-rdma/rdma-core - Git at Google

 .. Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md
 rsocket Protocol and Design Guide               11/11/2012

 Data Streaming (TCP) Overview
 -----------------------------
 Rsockets is a protocol over RDMA that supports a socket-level API
 for applications.  For details on the current state of the
 implementation, readers should refer to the rsocket man page.  This
 document describes the rsocket protocol, general design, and
 some implementation details.

 Rsockets exchanges data by performing RDMA write operations into
 exposed data buffers.  In addition to RDMA write data, rsockets uses
 small, 32-bit messages for internal communication.  RDMA writes
 are used to transfer application data into remote data buffers
 and to notify the peer when new target data buffers are available.
 The following figure highlights the operation.

    host A                   host B
                           remote SGL
   target SGL  <-------------  [  ]
      [  ] ------
      [  ] --    ------   receive buffer(s)
             --        ----->  +--+
               --              |  |
                 --            |  |
                   --          |  |
                     --        +--+
                       --
                         --->  +--+
                               |  |
                               |  |
                               +--+

 The remote SGL contains the address, size, and rkey of the target SGL.  As
 receive buffers become available on host B, rsockets will issue an RDMA
 write against one of the entries in the target SGL on host A.  The
 updated entry will reference an available receive buffer.  Immediate data
 included with the RDMA write will indicate to host A that a target SGE
 has been updated.

 When host A has data to send, it will check its target SGL.  The current
 target SGE will contain the address, size, and rkey of the next receive
 buffer on host B.  If the data transfer is smaller than the size of the
 remote receive buffer, host A will update its target SGE to reflect the
 remaining size of the receive buffer.  That is, once a receive buffer has
 been published to a remote peer, it will be fully consumed before a second
 buffer is used.

 Rsockets relies on immediate data to notify the remote peer when data has
 been transferred or when a target SGL has been updated.  Because immediate
 data requires that the remote QP have a posted receive, rsockets also uses
 a credit based flow control mechanism.  The number of credits is based on
 the size of the receive queue, with initial credits exchanged during
 connection setup.  In order to transfer data, rsockets requires both
 available receive buffers (published via the target SGL) and data credits.

 Since immediate data is limited to 32-bits, messages may either indicate
 the arrival of application data or may be an internal message, but not both.
 To avoid credit deadlock, rsockets reserves a small number of available
 credits for control messages only, with the protocol relying on RNR NAKs
 and retries to make forward progress.


 Connection Establishment
 ------------------------
 rsockets uses the RDMA CM for connection establishment.  Struct rs_conn_data
 is exchanged during the connection exchange as private data in the request
 and reply messages.

 struct rs_sge {
 	uint64_t addr;
 	uint32_t key;
 	uint32_t length;
 };

 #define RS_CONN_FLAG_NET 1

 struct rs_conn_data {
 	uint8_t		  version;
 	uint8_t		  flags;
 	uint16_t	  credits;
 	uint32_t	  reserved2;
 	struct rs_sge target_sgl;
 	struct rs_sge data_buf;
 };

 Version - current version is 1
 Flags
 RS_CONN_FLAG_NET - Set to 1 if host is big Endian.
                    Determines byte ordering for RDMA write messages
 Credits - number of initial receive credits
 Reserved2 - set to 0
 Target SGL - Address, size (# entries), and rkey of target SGL.
              Remote side will copy this into their remote SGL.
 Data Buffer - Initial receive buffer address, size (in bytes), and rkey.
               Remote side will copy this into their first target SGE.


 Message Format
 --------------
 Rsocket uses RDMA writes with immediate data for all message exchanges.
 RDMA writes of 0 length are used if no additional data beyond the message
 needs to be exchanged.  Immediate data is limited to 32-bits.  Rsockets
 defines the following format for messages.

 The upper 3 bits are used to define the type of message being exchanged,
 with the meaning of the lower 29 bits determined by the upper bits.

 Bits    Message             Meaning of
 31:29    Type               Bits 28:0
 000    Data Transfer     bytes transferred
 001    reserved
 010    reserved - used internally, available for future use
 011    reserved
 100    Credit Update     received credits granted
 101    reserved
 110    Iomap Updated     index of updated entry
 111    Control           control message type

 Data Transfer
 Indicates that application data has been written into the next available
 receive buffer.  The size of the transfer, in bytes, is carried in the lower
 bits of the message.

 Credit Update
 Used to indicate that additional receive buffers and credits are available.
 The number of available credits is carried in the lower bits of the message.
 A credit update message is also used to indicate that a target SGE has been
 updated, in which case the number of additional credits may be 0.  The
 receiver of a credit update message must check for updates to the target SGL
 by inspecting the contents of the SGL.  The rsocket implementation must take
 care not to modify a remote target SGL while it may be in use.  This is done
 by tracking when a receive buffer referenced by a remote target SGL has been
 filled.

 Iomap Updated
 Used to indicate that a remote iomap entry was updated.  The updated entry
 contains the offset value associated with an address, length, and rkey.  Once
 an iomap has been updated, the local application can issue directed IO
 transfers against the corresponding remote buffer.

 Control Message - DISCONNECT
 Indicates that the rsocket connection has been fully disconnected and will no
 longer send or receive data.  Data received before the disconnect message was
 processed may still be available for reading.

 Control Message - SHUTDOWN
 Indicates that the remote rsocket has shutdown the send side of its
 connection.  The recipient of a shutdown message will no longer accept
 incoming data, but may still transfer outbound data.


 Iomapped Buffers
 ----------------
 Rsockets allows for zero-copy transfers using what it refers to as iomapped
 buffers.  Iomapping and direct data placement (zero-copy) transfers are done
 using rsocket specific extensions.  The general operation is similar to
 that used for normal data transfers described above.

    host A                   host B
                           remote iomap
   target iomap  <-----------  [  ]
      [  ] ------
      [  ] --    ------  iomapped buffer(s)
             --        ----->  +--+
               --              |  |
                 --            |  |
                   --          |  |
                     --        +--+
                       --
                         --->  +--+
                               |  |
                               |  |
                               +--+

 The remote iomap contains the address, size, and rkey of the target iomap.  As
 the applicaton maps buffers host B to a given rsocket, rsockets will issue an RDMA
 write against one of the entries in the target iomap on host A.  The
 updated entry will reference an available iomapped buffer.  Immediate data
 included with the RDMA write will indicate to host A that a target iomap
 has been updated.

 When host A wishes to transfer directly into an iomapped buffer, it will check
 its target iomap for an offset corresponding to a remotely mapped buffer.  A
 matching iomap entry will contain the address, size, and rkey of the target
 buffer on host B.  Host A will then issue an RDMA operation against the
 registered remote data buffer.

 From host A's perspective, the transfer appears as a normal send/write
 operation, with the data stream redirected directly into the receiving
 application's buffer.


 Datagram Overview
 -----------------
 The rsocket API supports datagram sockets.  Datagram support is handled through an
 entirely different protocol and internal implementation.  Unlike connected rsockets,
 datagram rsockets are not necessarily bound to a network (IP) address.  A datagram
 socket may use any number of network (IP) addresses, including those which map to
 different RDMA devices.  As a result, a single datagram rsocket must support
 using multiple RDMA devices and ports, and a datagram rsocket references a single
 UDP socket, plus zero or more UD QPs.

 Rsockets uses headers inserted before user data sent over UDP sockets to resolve
 remote UD QP numbers.  When a user first attempts to send a datagram to a remote
 address (IP and UDP port), rsockets will take the following steps:

 1. Store the destination address into a lookup table.
 2. Resolve which local network address should be used when sending
    to the specified destination.
 3. Allocate a UD QP on the RDMA device associated with the local address.
 4. Send the user's datagram to the remote UDP socket.

 A header is inserted before the user's datagram.  The header specifies the
 UD QP number associated with the local network address (IP and UDP port) of
 the send.

 A service thread is used to process messages received on the UDP socket.  This
 thread updates the rsocket lookup tables with the remote QPN and path record
 data.  The service thread forwards data received on the UDP socket to an
 rsocket QP.  After the remote QPN and path records have been resolved, datagram
 communication between two nodes are done over the UD QP.

 UDP Message Format
 ------------------
 Rsockets uses messages exchanged over UDP sockets to resolve remote QP numbers.
 If a user sends a datagram to a remote service and the local rsocket is not
 yet configured to send directly to a remote UD QP, the user data is sent over
 a UDP socket with the following header inserted before the user data.

 struct ds_udp_header {
 	uint32_t	  tag;
 	uint8_t		  version;
 	uint8_t		  op;
 	uint8_t		  length;
 	uint8_t		  reserved;
 	uint32_t	  qpn;  /* lower 8-bits reserved */
 	union {
 		uint32_t  ipv4;
 		uint8_t   ipv6[16];
 	} addr;
 };

 Tag - Marker used to help identify that the UDP header is present.
 #define DS_UDP_TAG 0x55555555

 Version - IP address version, either 4 or 6
 Op - Indicates message type, used to control the receiver's operation.
      Valid operations are RS_OP_DATA and RS_OP_CTRL.  Data messages
      carry user data, while control messages are used to reply with the
      local QP number.
 Length - Size of the UDP header.
 QPN - UD QP number associated with sender's IP address and port.
       The sender's address and port is extracted from the received UDP
       datagram.
 Addr - Target IP address of the sent datagram.

 Once the remote QP information has been resolved, data is sent directly
 between UD QPs.  The following header is inserted before any user data that
 is transferred over a UD QP.

 struct ds_header {
 	uint8_t		     version;
 	uint8_t		     length;
 	uint16_t	     port;
 	union {
 		uint32_t     ipv4;
 		struct {
 			uint32_t flowinfo;
 			uint8_t  addr[16];
 		} ipv6;
 	} addr;
 };

 Verion - IP address version
 Length - Size of the header
 Port - Associated source address UDP port
 Addr - Associated source IP address
	.. Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md
	rsocket Protocol and Design Guide 11/11/2012

	Data Streaming (TCP) Overview
	-----------------------------
	Rsockets is a protocol over RDMA that supports a socket-level API
	for applications. For details on the current state of the
	implementation, readers should refer to the rsocket man page. This
	document describes the rsocket protocol, general design, and
	some implementation details.

	Rsockets exchanges data by performing RDMA write operations into
	exposed data buffers. In addition to RDMA write data, rsockets uses
	small, 32-bit messages for internal communication. RDMA writes
	are used to transfer application data into remote data buffers
	and to notify the peer when new target data buffers are available.
	The following figure highlights the operation.

	host A host B
	remote SGL
	target SGL <------------- [ ]
	[ ] ------
	[ ] -- ------ receive buffer(s)
	-- -----> +--+
	-- \| \|
	-- \| \|
	-- \| \|
	-- +--+
	--
	---> +--+
	\| \|
	\| \|
	+--+

	The remote SGL contains the address, size, and rkey of the target SGL. As
	receive buffers become available on host B, rsockets will issue an RDMA
	write against one of the entries in the target SGL on host A. The
	updated entry will reference an available receive buffer. Immediate data
	included with the RDMA write will indicate to host A that a target SGE
	has been updated.

	When host A has data to send, it will check its target SGL. The current
	target SGE will contain the address, size, and rkey of the next receive
	buffer on host B. If the data transfer is smaller than the size of the
	remote receive buffer, host A will update its target SGE to reflect the
	remaining size of the receive buffer. That is, once a receive buffer has
	been published to a remote peer, it will be fully consumed before a second
	buffer is used.

	Rsockets relies on immediate data to notify the remote peer when data has
	been transferred or when a target SGL has been updated. Because immediate
	data requires that the remote QP have a posted receive, rsockets also uses
	a credit based flow control mechanism. The number of credits is based on
	the size of the receive queue, with initial credits exchanged during
	connection setup. In order to transfer data, rsockets requires both
	available receive buffers (published via the target SGL) and data credits.

	Since immediate data is limited to 32-bits, messages may either indicate
	the arrival of application data or may be an internal message, but not both.
	To avoid credit deadlock, rsockets reserves a small number of available
	credits for control messages only, with the protocol relying on RNR NAKs
	and retries to make forward progress.


	Connection Establishment
	------------------------
	rsockets uses the RDMA CM for connection establishment. Struct rs_conn_data
	is exchanged during the connection exchange as private data in the request
	and reply messages.

	struct rs_sge {
	uint64_t addr;
	uint32_t key;
	uint32_t length;
	};

	#define RS_CONN_FLAG_NET 1

	struct rs_conn_data {
	uint8_t version;
	uint8_t flags;
	uint16_t credits;
	uint32_t reserved2;
	struct rs_sge target_sgl;
	struct rs_sge data_buf;
	};

	Version - current version is 1
	Flags
	RS_CONN_FLAG_NET - Set to 1 if host is big Endian.
	Determines byte ordering for RDMA write messages
	Credits - number of initial receive credits
	Reserved2 - set to 0
	Target SGL - Address, size (# entries), and rkey of target SGL.
	Remote side will copy this into their remote SGL.
	Data Buffer - Initial receive buffer address, size (in bytes), and rkey.
	Remote side will copy this into their first target SGE.


	Message Format
	--------------
	Rsocket uses RDMA writes with immediate data for all message exchanges.
	RDMA writes of 0 length are used if no additional data beyond the message
	needs to be exchanged. Immediate data is limited to 32-bits. Rsockets
	defines the following format for messages.

	The upper 3 bits are used to define the type of message being exchanged,
	with the meaning of the lower 29 bits determined by the upper bits.

	Bits Message Meaning of
	31:29 Type Bits 28:0
	000 Data Transfer bytes transferred
	001 reserved
	010 reserved - used internally, available for future use
	011 reserved
	100 Credit Update received credits granted
	101 reserved
	110 Iomap Updated index of updated entry
	111 Control control message type

	Data Transfer
	Indicates that application data has been written into the next available
	receive buffer. The size of the transfer, in bytes, is carried in the lower
	bits of the message.

	Credit Update
	Used to indicate that additional receive buffers and credits are available.
	The number of available credits is carried in the lower bits of the message.
	A credit update message is also used to indicate that a target SGE has been
	updated, in which case the number of additional credits may be 0. The
	receiver of a credit update message must check for updates to the target SGL
	by inspecting the contents of the SGL. The rsocket implementation must take
	care not to modify a remote target SGL while it may be in use. This is done
	by tracking when a receive buffer referenced by a remote target SGL has been
	filled.

	Iomap Updated
	Used to indicate that a remote iomap entry was updated. The updated entry
	contains the offset value associated with an address, length, and rkey. Once
	an iomap has been updated, the local application can issue directed IO
	transfers against the corresponding remote buffer.

	Control Message - DISCONNECT
	Indicates that the rsocket connection has been fully disconnected and will no
	longer send or receive data. Data received before the disconnect message was
	processed may still be available for reading.

	Control Message - SHUTDOWN
	Indicates that the remote rsocket has shutdown the send side of its
	connection. The recipient of a shutdown message will no longer accept
	incoming data, but may still transfer outbound data.


	Iomapped Buffers
	----------------
	Rsockets allows for zero-copy transfers using what it refers to as iomapped
	buffers. Iomapping and direct data placement (zero-copy) transfers are done
	using rsocket specific extensions. The general operation is similar to
	that used for normal data transfers described above.

	host A host B
	remote iomap
	target iomap <----------- [ ]
	[ ] ------
	[ ] -- ------ iomapped buffer(s)
	-- -----> +--+
	-- \| \|
	-- \| \|
	-- \| \|
	-- +--+
	--
	---> +--+
	\| \|
	\| \|
	+--+

	The remote iomap contains the address, size, and rkey of the target iomap. As
	the applicaton maps buffers host B to a given rsocket, rsockets will issue an RDMA
	write against one of the entries in the target iomap on host A. The
	updated entry will reference an available iomapped buffer. Immediate data
	included with the RDMA write will indicate to host A that a target iomap
	has been updated.

	When host A wishes to transfer directly into an iomapped buffer, it will check
	its target iomap for an offset corresponding to a remotely mapped buffer. A
	matching iomap entry will contain the address, size, and rkey of the target
	buffer on host B. Host A will then issue an RDMA operation against the
	registered remote data buffer.

	From host A's perspective, the transfer appears as a normal send/write
	operation, with the data stream redirected directly into the receiving
	application's buffer.



	Datagram Overview
	-----------------
	The rsocket API supports datagram sockets. Datagram support is handled through an
	entirely different protocol and internal implementation. Unlike connected rsockets,
	datagram rsockets are not necessarily bound to a network (IP) address. A datagram
	socket may use any number of network (IP) addresses, including those which map to
	different RDMA devices. As a result, a single datagram rsocket must support
	using multiple RDMA devices and ports, and a datagram rsocket references a single
	UDP socket, plus zero or more UD QPs.

	Rsockets uses headers inserted before user data sent over UDP sockets to resolve
	remote UD QP numbers. When a user first attempts to send a datagram to a remote
	address (IP and UDP port), rsockets will take the following steps:

	1. Store the destination address into a lookup table.
	2. Resolve which local network address should be used when sending
	to the specified destination.
	3. Allocate a UD QP on the RDMA device associated with the local address.
	4. Send the user's datagram to the remote UDP socket.

	A header is inserted before the user's datagram. The header specifies the
	UD QP number associated with the local network address (IP and UDP port) of
	the send.

	A service thread is used to process messages received on the UDP socket. This
	thread updates the rsocket lookup tables with the remote QPN and path record
	data. The service thread forwards data received on the UDP socket to an
	rsocket QP. After the remote QPN and path records have been resolved, datagram
	communication between two nodes are done over the UD QP.

	UDP Message Format
	------------------
	Rsockets uses messages exchanged over UDP sockets to resolve remote QP numbers.
	If a user sends a datagram to a remote service and the local rsocket is not
	yet configured to send directly to a remote UD QP, the user data is sent over
	a UDP socket with the following header inserted before the user data.

	struct ds_udp_header {
	uint32_t tag;
	uint8_t version;
	uint8_t op;
	uint8_t length;
	uint8_t reserved;
	uint32_t qpn; /* lower 8-bits reserved */
	union {
	uint32_t ipv4;
	uint8_t ipv6[16];
	} addr;
	};

	Tag - Marker used to help identify that the UDP header is present.
	#define DS_UDP_TAG 0x55555555

	Version - IP address version, either 4 or 6
	Op - Indicates message type, used to control the receiver's operation.
	Valid operations are RS_OP_DATA and RS_OP_CTRL. Data messages
	carry user data, while control messages are used to reply with the
	local QP number.
	Length - Size of the UDP header.
	QPN - UD QP number associated with sender's IP address and port.
	The sender's address and port is extracted from the received UDP
	datagram.
	Addr - Target IP address of the sent datagram.

	Once the remote QP information has been resolved, data is sent directly
	between UD QPs. The following header is inserted before any user data that
	is transferred over a UD QP.

	struct ds_header {
	uint8_t version;
	uint8_t length;
	uint16_t port;
	union {
	uint32_t ipv4;
	struct {
	uint32_t flowinfo;
	uint8_t addr[16];
	} ipv6;
	} addr;
	};

	Verion - IP address version
	Length - Size of the header
	Port - Associated source address UDP port
	Addr - Associated source IP address