scst/scst/README.dlm - backupdr - Git at Google

 Synchronization of the Persistent Reservation Information via the DLM
 =====================================================================

 Introduction
 ------------

 In an H.A. setup where multiple servers share data it is required that
 the persistent reservation state is kept consistent across the cluster.
 One possible approach is to use the DLM to keep the PR state synchronized
 across nodes. Since the DLM can associate data with each DLM lock object,
 DLM lock objects can be used to store PR data. The data that is associated
 with a DLM lock object is called the Lock Value Block or LVB. The code in
 scst_dlm.c uses the DLM to keep PR data synchronized across all nodes in
 a cluster.


 Software Components
 -------------------

 The following software components are needed by the code in scst_dlm.c:
 * The DLM kernel driver (dlm.ko). This driver is only built if CONFIG_DLM
   has been set.
 * The DLM control daemon (dlm_controld.pcmk). This daemon passes cluster
   node IDs and IP addresses to the DLM kernel driver via the configfs
   interface of the DLM kernel driver.
 * Corosync to manage cluster membership of the cluster nodes and to assign
   a node ID to each cluster node.
 * A facility to start the DLM control daemon, e.g. Pacemaker.

 On most Linux distributions the software packages that contain this software
 have the names kernel, dlm, corosync and pacemaker.

 NOTE! You might need to apply a DLM bugfix patch, see scst-devel mailing list
 thread https://sourceforge.net/p/scst/mailman/scst-devel/thread/CADHfD59FK6seaammL8b9LM3U3tw5HvYp3kPTk_r1OYkPR7bPhg@mail.gmail.com/#msg34761854
 for more details.


 DLM Configuration
 -----------------

 The DLM kernel module supports the TCP and SCTP communication protocols. An
 advantage of SCTP for H.A. purposes is that it supports multihoming. One of
 these protocols can be selected via the -r <proto> option of dlm_controld.
 That option can be set via the "args" argument of the Pacemaker dlm_controld
 resource. For more information, see also:
 * The dlm_controld(8) man page.
 * In the "Pacemaker 1.1, Clusters from Scratch" guide, the section "Configure
   the Cluster for the DLM".
 * The dlm_controld resource agent: /usr/lib/ocf/resource.d/pacemaker/controld

 Here is an example of how to set up a cluster with two nodes and how to
 configure and start the DLM control daemon:
  1. If a network switch is present between the two nodes, enable IPv4 multicast
     on that switch.
  2. Copy /etc/corosync/corosync.conf.example into /etc/corosync/corosync.conf
     and edit that file.
  3. If a file /etc/default/corosync exists, enable Corosync in that file.
  4. Start Corosync:
       systemctl start corosync || /etc/init.d/corosync start
  5. Check that all configured Corosync rings have two members:
       corosync-cfgtool -s && { corosync-cmapctl | grep members; }
  6. Start pcsd:
       systemctl start pcsd || /etc/init.d/pcsd start
  7. Set up cluster authentication:
       pcs cluster auth centos7-vm centos7b-vm
  8. Start Pacemaker:
       systemctl start pacemaker || /etc/init.d/pacemaker start
  9. If the cluster has only two nodes, disable the Pacemaker quorum policy and
     disable STONITH:
       crm_attribute -t crm_config -n no-quorum-policy -v ignore
       crm_attribute -t crm_config -n stonith-enabled -v false
 10. Check the cluster status:
       pcs status
 11. Create a Pacemaker resource for dlm_controld:
       pcs resource delete dlm
       pcs resource create dlm ocf:pacemaker:controld \
         args="-q0 -f0" allow_stonith_disabled=true \
         op monitor timeout=60 \
 	--clone interleave=true
 12. Check the Pacemaker status:
       pcs status


 Startup and Shutdown
 --------------------

 The startup sequence is as follows:
 * Load the DLM kernel module. If not loaded explicitly, "modprobe scst" will
   load the DLM kernel module implicitly.
 * Load and configure SCST with all target ports disabled.
 * Enable cluster mode for all SCST devices that can be accessed through more
   than one cluster node:
     for x in /sys/kernel/scst_tgt/handlers/*/*/; do
         echo 1 >$x/cluster_mode &
     done
     wait
 * Start Corosync and Pacemaker.
 * Wait until Pacemaker has reached the idle state:
     pacemaker_dc_status() {
 	local dc

 	dc="$(crmadmin -D 2>/dev/null | sed 's/Designated Controller is: //')"
 	[ -n "$dc" ] &&
 	crmadmin -S "$dc" 2>/dev/null |
 	sed 's/^Status of crmd@[^[:blank:]]*:[[:blank:]]\([^[:blank:]]*\).*/\1/'
     }
     timeout=300
     for ((i=0;i<timeout;i++)); do
 	if [ "$(pacemaker_dc_status)" = "S_IDLE" ]; then
 	    echo "Pacemaker reached idle state after $i s"
 	    break
 	fi
 	sleep 1
     done
     if [ "$i" = "$timeout" ]; then
 	echo "Pacemaker did not reach the IDLE state in $i s"
     fi
 * Enable SCST target ports.
 * If no DLM resource has been configured in Pacemaker, start dlm_controld.pcmk
   explicitly.

 The proper shutdown order is as follows:
 * Tell SCST to stop accepting SCSI commands and wait until all initiators have
   logged out:
     for x in $(find /sys/kernel/scst_tgt/targets/ -name enabled); do
         echo 0 > $x &
     done
     wait
     while ls -Ad /sys/kernel/scst_tgt/targets/*/*/sessions/* 2>&1 |
           grep -vE '/sys/kernel/scst_tgt/targets/(copy_manager|scst_local)/'; do
         sleep 1
     done
 * Tell SCST to release the DLM lockspaces:
     while grep -q '^1$' /sys/kernel/scst_tgt/devices/*/cluster_mode 2>/dev/null
     do
         for x in /sys/kernel/scst_tgt/devices/*/cluster_mode; do
             { [ -e "$x" ] && echo 0 > "$x"; } &
         done
         wait
         sleep 1
     done
 * Stop Pacemaker and Corosync
 * Unload the SCST kernel modules
 * Unload the DLM kernel driver


 Lockspace names
 ---------------

 The names of the DLM lockspaces used by SCST follow the following pattern:
 scst-<t10_dev_id> where t10_dev_id is the T10 device ID of the SCST device
 associated with this lockspace.


 Notes
 -----

 Since the lockspace name depends on the t10_dev_id it is not allowed to
 change the t10_dev_id if cluster mode has been enabled.


 Testing
 -------

 Two examples of test suites for the cluster PR support code are:
 * The SCSI conformance tests in the libiscsi project.
 * The Windows Cluster Validation Tests
   (https://technet.microsoft.com/en-us/library/Cc726064.aspx).


 To do
 -----

  * Ensure that PREEMPT AND ABORT affects all cluster nodes instead of
    only the cluster node that received this command.


 See also
 --------

 * Bart Van Assche, Using the DLM as a distributed in-memory database, Linux
   Plumbers North America, Seattle, August 20, 2015
   (https://linuxplumbersconf.org/2015/ocw//system/presentations/2691/original/Using%20the%20DLM%20as%20a%20Distributed%20In-Memory%20Database.pdf).
 * Andrew Beekhof, Pacemaker Configuration Explained, 2015
   (http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/).
 * Andrew Beekhof, Clusters from Scratch, 2015
   (http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Clusters_from_Scratch/index.html).
	Synchronization of the Persistent Reservation Information via the DLM
	=====================================================================

	Introduction
	------------

	In an H.A. setup where multiple servers share data it is required that
	the persistent reservation state is kept consistent across the cluster.
	One possible approach is to use the DLM to keep the PR state synchronized
	across nodes. Since the DLM can associate data with each DLM lock object,
	DLM lock objects can be used to store PR data. The data that is associated
	with a DLM lock object is called the Lock Value Block or LVB. The code in
	scst_dlm.c uses the DLM to keep PR data synchronized across all nodes in
	a cluster.


	Software Components
	-------------------

	The following software components are needed by the code in scst_dlm.c:
	* The DLM kernel driver (dlm.ko). This driver is only built if CONFIG_DLM
	has been set.
	* The DLM control daemon (dlm_controld.pcmk). This daemon passes cluster
	node IDs and IP addresses to the DLM kernel driver via the configfs
	interface of the DLM kernel driver.
	* Corosync to manage cluster membership of the cluster nodes and to assign
	a node ID to each cluster node.
	* A facility to start the DLM control daemon, e.g. Pacemaker.

	On most Linux distributions the software packages that contain this software
	have the names kernel, dlm, corosync and pacemaker.

	NOTE! You might need to apply a DLM bugfix patch, see scst-devel mailing list
	thread https://sourceforge.net/p/scst/mailman/scst-devel/thread/CADHfD59FK6seaammL8b9LM3U3tw5HvYp3kPTk_r1OYkPR7bPhg@mail.gmail.com/#msg34761854
	for more details.


	DLM Configuration
	-----------------

	The DLM kernel module supports the TCP and SCTP communication protocols. An
	advantage of SCTP for H.A. purposes is that it supports multihoming. One of
	these protocols can be selected via the -r <proto> option of dlm_controld.
	That option can be set via the "args" argument of the Pacemaker dlm_controld
	resource. For more information, see also:
	* The dlm_controld(8) man page.
	* In the "Pacemaker 1.1, Clusters from Scratch" guide, the section "Configure
	the Cluster for the DLM".
	* The dlm_controld resource agent: /usr/lib/ocf/resource.d/pacemaker/controld

	Here is an example of how to set up a cluster with two nodes and how to
	configure and start the DLM control daemon:
	1. If a network switch is present between the two nodes, enable IPv4 multicast
	on that switch.
	2. Copy /etc/corosync/corosync.conf.example into /etc/corosync/corosync.conf
	and edit that file.
	3. If a file /etc/default/corosync exists, enable Corosync in that file.
	4. Start Corosync:
	systemctl start corosync \|\| /etc/init.d/corosync start
	5. Check that all configured Corosync rings have two members:
	corosync-cfgtool -s && { corosync-cmapctl \| grep members; }
	6. Start pcsd:
	systemctl start pcsd \|\| /etc/init.d/pcsd start
	7. Set up cluster authentication:
	pcs cluster auth centos7-vm centos7b-vm
	8. Start Pacemaker:
	systemctl start pacemaker \|\| /etc/init.d/pacemaker start
	9. If the cluster has only two nodes, disable the Pacemaker quorum policy and
	disable STONITH:
	crm_attribute -t crm_config -n no-quorum-policy -v ignore
	crm_attribute -t crm_config -n stonith-enabled -v false
	10. Check the cluster status:
	pcs status
	11. Create a Pacemaker resource for dlm_controld:
	pcs resource delete dlm
	pcs resource create dlm ocf:pacemaker:controld \
	args="-q0 -f0" allow_stonith_disabled=true \
	op monitor timeout=60 \
	--clone interleave=true
	12. Check the Pacemaker status:
	pcs status


	Startup and Shutdown
	--------------------

	The startup sequence is as follows:
	* Load the DLM kernel module. If not loaded explicitly, "modprobe scst" will
	load the DLM kernel module implicitly.
	* Load and configure SCST with all target ports disabled.
	* Enable cluster mode for all SCST devices that can be accessed through more
	than one cluster node:
	for x in /sys/kernel/scst_tgt/handlers///; do
	echo 1 >$x/cluster_mode &
	done
	wait
	* Start Corosync and Pacemaker.
	* Wait until Pacemaker has reached the idle state:
	pacemaker_dc_status() {
	local dc

	dc="$(crmadmin -D 2>/dev/null \| sed 's/Designated Controller is: //')"
	[ -n "$dc" ] &&
	crmadmin -S "$dc" 2>/dev/null \|
	sed 's/^Status of crmd@[^[:blank:]]:[[:blank:]]\([^[:blank:]]\).*/\1/'
	}
	timeout=300
	for ((i=0;i<timeout;i++)); do
	if [ "$(pacemaker_dc_status)" = "S_IDLE" ]; then
	echo "Pacemaker reached idle state after $i s"
	break
	fi
	sleep 1
	done
	if [ "$i" = "$timeout" ]; then
	echo "Pacemaker did not reach the IDLE state in $i s"
	fi
	* Enable SCST target ports.
	* If no DLM resource has been configured in Pacemaker, start dlm_controld.pcmk
	explicitly.

	The proper shutdown order is as follows:
	* Tell SCST to stop accepting SCSI commands and wait until all initiators have
	logged out:
	for x in $(find /sys/kernel/scst_tgt/targets/ -name enabled); do
	echo 0 > $x &
	done
	wait
	while ls -Ad /sys/kernel/scst_tgt/targets///sessions/* 2>&1 \|
	grep -vE '/sys/kernel/scst_tgt/targets/(copy_manager\|scst_local)/'; do
	sleep 1
	done
	* Tell SCST to release the DLM lockspaces:
	while grep -q '^1$' /sys/kernel/scst_tgt/devices/*/cluster_mode 2>/dev/null
	do
	for x in /sys/kernel/scst_tgt/devices/*/cluster_mode; do
	{ [ -e "$x" ] && echo 0 > "$x"; } &
	done
	wait
	sleep 1
	done
	* Stop Pacemaker and Corosync
	* Unload the SCST kernel modules
	* Unload the DLM kernel driver


	Lockspace names
	---------------

	The names of the DLM lockspaces used by SCST follow the following pattern:
	scst-<t10_dev_id> where t10_dev_id is the T10 device ID of the SCST device
	associated with this lockspace.


	Notes
	-----

	Since the lockspace name depends on the t10_dev_id it is not allowed to
	change the t10_dev_id if cluster mode has been enabled.


	Testing
	-------

	Two examples of test suites for the cluster PR support code are:
	* The SCSI conformance tests in the libiscsi project.
	* The Windows Cluster Validation Tests
	(https://technet.microsoft.com/en-us/library/Cc726064.aspx).


	To do
	-----

	* Ensure that PREEMPT AND ABORT affects all cluster nodes instead of
	only the cluster node that received this command.


	See also
	--------

	* Bart Van Assche, Using the DLM as a distributed in-memory database, Linux
	Plumbers North America, Seattle, August 20, 2015
	(https://linuxplumbersconf.org/2015/ocw//system/presentations/2691/original/Using%20the%20DLM%20as%20a%20Distributed%20In-Memory%20Database.pdf).
	* Andrew Beekhof, Pacemaker Configuration Explained, 2015
	(http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/).
	* Andrew Beekhof, Clusters from Scratch, 2015
	(http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Clusters_from_Scratch/index.html).