Copyright (c) 2020, Oracle. All rights reserved. Oracle Confidential.

Solaris Cluster How To Remove SCSI Reservations (Doc ID 1011961.1)

APPLIES TO:

Solaris Cluster - Version 3.0 and later

Oracle Solaris on SPARC (64-bit)
Oracle Solaris on SPARC (32-bit)
Oracle Solaris on x86-64 (64-bit)

GOAL

Solaris Cluster uses SCSI reservations for both quorum and fencing, refer to document 1008224.1 for detailed explanation on
SCSI reservation type used.

This document presents the commands available in Solaris Cluster to remove SCSI reservations for SCSI-2 and SCSI-3
compliant devices and Solaris Cluster PGRe (emulated)reservations.

Solaris Cluster uses SCSI (Small Computer System Interface) reservations for both operational (quorum) and data integrity
(fencing) purposes. Removing SCSI reservations from the disk devices of a running cluster can result in a loss of service
and data corruption. Before using any of the commands presented in this document, please consult your Oracle Services
representative.

SOLUTION

There are two types of SCSI reservations used by Solaris Cluster which may requires specific maintenance:

SCSI-2 Reservations
SCSI-3 Persistent Group Reservations, also known as PGR
Solaris Cluster Persistent Group Reservation Emulation also know as PGRe (used also by software quorum), a software
implemantion of SCSI-3 PGR

Solaris Cluster uses only one SCSI reservation type for a given shared storage device. However, both SCSI reservation types
may be used at the same time in the same cluster.
Therefore, before using the commands presented in this document, refer to document 1008156.1 to identify which SCSI
reservation type is being used for each shared storage device in your cluster.

The command used to manage scsi reservation are:

/usr/cluster/lib/sc/reserve: used by the run_reserve program. This command works only when booted in cluster mode.
/usr/cluster/lib/sc/scsi: not used directly by the cluster framework. This command also works when not booted into
cluster mode (boot -x). It will be primarily used in this document for this reason.

It is recommended to disable the failfast mechanism using the 'release' command with the 'disfailfast' option on all nodes
participating in the cluster before removing any SCSI reservations.

1. Removing SCSI-2 Reservations

To remove SCSI-2 Reservations use the 'scsi' command with the 'release' option as shown below.  This command, however,
must be executed from the system which owns the reservation in order to successfully remove the reservation.
Disable the failfast on all node participating in the cluster:

# /usr/cluster/lib/sc/reserve -c disfailfast -z /dev/did/rdsk/d#s2

Release the reservation from the node which is owning it (most likely the only node participating in the cluster):

# /usr/cluster/lib/sc/scsi -c release -d /dev/did/rdsk/d#s2

 The '#' character is a reference to the DID device number of the disk from which you wish to remove a SCSI-2 Reservation.  The
standard Solaris disk device reference, /dev/rdsk/c#t#d#s2, can also be used in place of the DID device reference.

An alternative is to use the 'reserve' command with the 'release' option to remove a SCSI-2 Reservation as shown below.
However this command can only be used when booted in cluster mode.

# /usr/cluster/lib/sc/reserve -c disfailfast -z /dev/did/rdsk/d#s2

# /usr/cluster/lib/sc/reserve -c release -z /dev/did/rdsk/d#s2

SCSI-2 Reservations will be removed automatically if the system that owns the SCSI-2 Reservations is shut down or power
cycled. Likewise, SCSI-2 Reservations will be removed automatically if the storage devices that have SCSI-2 Reservations
on them are shut down or power cycled.

There are other methods that also remove SCSI-2 Reservations without resorting to the commands presented above to do
so. For example, a SCSI bus reset will remove SCSI-2 Reservations from the storage devices affected by the reset.  These
other methods of removing SCSI-2 Reservations are all due to the fact that SCSI-2 Reservations were not defined or
designed to be persistent in nature.

Example #1

If DID device d15 has a SCSI-2 Reservation you wish to remove, use the following commands.

# /usr/cluster/lib/sc/reserve -c disfailfast -z /dev/did/rdsk/d15s2 (on all nodes which have been

booted into cluster mode)
do_enfailfast returned 0
# /usr/cluster/lib/sc/scsi -c release -d /dev/did/rdsk/d15s2
do_scsi2_release returned 0

Alternatively, if DID device d15 corresponds to disk device c3t4d0 on the cluster node where the commands will be executed,
you could use the following commands instead.

# /usr/cluster/lib/sc/reserve -c disfailfast -z /dev/rdsk/c3t4d0s2 (on all nodes which have been

booted into cluster mode)
do_enfailfast returned 0
# /usr/cluster/lib/sc/reserve -c release -z /dev/rdsk/c3t4d0s2
do_scsi2_release 0

Note, again, that you can use either the DID device reference or the Solaris disk device name reference with both the 'scsi' and
the 'reserve' commands.

2. Removing SCSI-3 Persistent Reservations

Before executing the scrub operation, it is recommended that you confirm there are reservation keys registered with the device.
To confirm there are reservation keys registered with the device execute the 'scsi' command with the 'inkeys' option and then
with the 'inresv' option as shown below.

# /usr/cluster/lib/sc/scsi -c inkeys -d /dev/did/rdsk/d#s2

# /usr/cluster/lib/sc/scsi -c inresv -d /dev/did/rdsk/d#s2

To remove SCSI-3 Persistent Reservations and all the reservation keys registered with a device use the 'scsi' command with the
'scrub' option as shown below. This command does not need to be executed from the system that owns the reservation. In
other words the node does not have to have it's SCSI key registered on the device and it is not a requirement that the node is
currently part of a running cluster.

# /usr/cluster/lib/sc/reserve -c disfailfast -z /dev/did/rdsk/d#s2 (on all node boot in cluster mode)

# /usr/cluster/lib/sc/scsi -c scrub -d /dev/did/rdsk/d#s2

The '#' character is a reference to the DID device number of the disk from which you wish to remove a SCSI-3 Persistent
Reservation. The standard Solaris disk device reference, /dev/rdsk/c#t#d#s2, can also be used in place of the DID device
reference.

SCSI-3 Persistent Reservations are 'persistent' by design which means that any reservation keys registered with a storage
device or any reservation placed on a storage device are to be retained by the storage device, even if powered off, and the
only way to remove them is by issuing a specific SCSI command to do so.  In other words, there are no automatic methods
whereby a SCSI-3 Persistent Reservation will be removed, such as a reset or power cycle of the device.  Solaris Cluster
uses the PERSISTENT RESERVE OUT (PROUT) SCSI command with the PREEMPT AND ABORT service action to remove
SCSI-3 Persistent Reservations.  This PROUT SCSI command is also programmed into the 'scsi' command through the
'scrub' option for your use when absolutely necessary.  

Most storage systems also provide a method to manually remove SCSI-3 Persistent Reservations by accessing or logging
into the storage device controller and executing commands defined by the storage vendor for this purpose. These
methods, however, are implementation specific and beyond the scope of this document. So, please consult with your
storage vendor for more details.

Example #1

If DID device d27 has a SCSI-3 Persistent Reservation you wish to remove, use the following commands.

The first command displays the reservation keys registered with the device.

# /usr/cluster/lib/sc/scsi -c inkeys -d /dev/did/rdsk/d27s2

Reservation keys(3):
0x44441b0800000003
0x44441b0800000001
0x44441b0800000002

The second command displays the SCSI-3 Persistent Reservation owner and the reservation type.

# /usr/cluster/lib/sc/scsi -c inresv -d /dev/did/rdsk/d27s2

Reservations(1): 0x44441b0800000001
type ---> 5

 
 Type 5 corresponds to a Write-Exclusive Registrants-Only type of SCSI-3 Persistent Reservation. This is the SCSI-3
Persistent Reservation type used in Solaris Cluster.

The third command removes the SCSI-3 Persistent Reservation and all the reservation keys registered with the device.

# /usr/cluster/lib/sc/scsi -c scrub -d /dev/did/rdsk/d27s2

Reservation keys currently on disk:
0x44441b0800000003
0x44441b0800000001
0x44441b0800000002

Attempting to remove all keys from the disk...

Scrubbing complete, use '/usr/cluster/lib/sc/scsi -c inkeys -d /dev/did/rdsk/d27s2' to verify success

When executing the 'scsi' command with the 'scrub' option, the output first displays the keys currently registered with the
device and then recommends that you check the device again to confirm that the registered keys have been removed. This
command and the result is shown below.

# /usr/cluster/lib/sc/scsi -c inkeys -d /dev/did/rdsk/d27s2

Reservation keys(0):

You should also confirm that the SCSI-3 Persistent Reservation has also been removed.

# /usr/cluster/lib/sc/scsi -c inresv -d /dev/did/rdsk/d27s2

Reservations(0):

3. Removing PGRe Keys

When the quorum device is only SCSI-2  compliant, Solaris Cluster uses an emulation mode to store SCSI-3 Persistent
Reservation keys on the quorum device for when the CMM needs to use them. This mechanism is referred to as Persistent
Group Reservation emulation, or PGRe, and even though the reservation keys are the same ones used with SCSI-3 compliant
devices they will be referred to in the remainder of this document as emulation keys because of the special, non-SCSI, way they
are handled and stored in this case.

Emulation keys are stored in a vendor defined location on the disk and do not interfere with or take away from the storage
of user data on the quorum device.

The PGRe mechanism is used only by Solaris Cluster and the way emulation keys are handled and stored are defined only
by Solaris Cluster. The PGRe mechanism and the emulation keys are not a part of the SCSI SPC specification. In other
words, the PGRe mechanism and the way emulation keys are handled and stored do not have any operational effect as it
relates to what is defined in the SCSI Specification documents. Therefore, any desire or need to remove emulation keys
must be associated only with Solaris Cluster itself and should have nothing whatsoever to do with the storage devices
used as quorum devices. So, if you believe you need to remove emulation keys, please consult with your Oracle Services
representative.

Before executing the pgre_scrub operation, it is recommended that you confirm there are reservation keys registered with the
device. To confirm there are reservation keys registered with the device execute the 'pgre' command with the 'pgre_inkeys'
option and then with the 'pgre_inresv' option as shown below.

# /usr/cluster/lib/sc/pgre -c pgre_inkeys -d /dev/did/rdsk/d#s2

# /usr/cluster/lib/sc/pgre -c pgre_inresv -d /dev/did/rdsk/d#s2
 

To remove emulation keys use the 'pgre' command with the 'pgre_scrub' option as shown below.

# /usr/cluster/lib/sc/pgre -c pgre_scrub -d /dev/did/rdsk/d#s2

The '#' character is a reference to the DID device number of the disk from which you wish to remove emulation keys. The
standard Solaris disk device reference, /dev/rdsk/c#t#d#s2, can also be used in place of the DID device reference.

The pgre command can be used regardless if you are booted in cluster mode or not (boot -x)

Example #1

If DID device d4 has emulation keys you wish to remove, use the following commands.

The first command displays the emulation keys that have been registered with the device.

# /usr/cluster/lib/sc/pgre -c pgre_inkeys -d /dev/did/rdsk/d4s2

key[0]=0x447f129700000001.

The second command displays which emulation key is the reservation owner.

# /usr/cluster/lib/sc/pgre -c pgre_inresv -d /dev/did/rdsk/d4s2

resv[0]: key=0x447f129700000001.

The third command removes the emulation keys from the device.

# /usr/cluster/lib/sc/pgre -c pgre_scrub -d /dev/did/rdsk/d4s2

Scrubbing complete. Use '/usr/cluster/lib/sc/pgre -c pgre_inkeys -d /dev/did/rdsk/d4s2' to verify
success.

When executing the 'pgre' command with the 'pgre_scrub' option, the output displays whether the operation completed
successfully and then recommends that you check the device again to confirm that the emulation keys have been removed.
This command and the result is shown below.

# /usr/cluster/lib/sc/pgre -c pgre_inkeys -d /dev/did/rdsk/d4s2

No keys registered

You should also confirm that the reservation owner has also been removed.

# /usr/cluster/lib/sc/pgre -c pgre_inresv -d /dev/did/rdsk/d4s2

No reservations on the device.

Example #2

If the DID device has never been used as a quorum device you will receive the following error when you use the 'pgre'
command. This error means that the storage area on the disk used by Solaris Cluster to store the emulation keys has not been
initialized. Solaris Cluster initializes the area on the disk to store emulation keys when the quorum device is created.

# /usr/cluster/lib/sc/pgre -c pgre_inkeys -d /dev/did/rdsk/d7s2

quorum_scsi2_sector_read: pgre id mismatch.
The sector id is . scsi2 read returned error (22).
# /usr/cluster/lib/sc/pgre -c pgre_inkeys -d /dev/did/rdsk/d7s2
command failed errno = 22.
 

Typical action plan one can use in S11 with the seq(1M) command to automatically do all above steps on a specific set or
contiguous device id

1) disable failfast on all nodes participating in the cluster

# bash
# for i in `seq 7 197`; do /usr/cluster/lib/sc/release -c disfailfast -z /dev/did/rdsk/d${i}; done

2) remove all key on all disk

Note: this should not cause any disruption since we disabled failfast. However it is recommended to do this during a quiet
period.

On one node only:

# for i in `seq 7 197`; do /usr/cluster/lib/sc/scsi -c scrub -d /dev/did/rdsk/d${i}; done

b) perform a reconfiguration
On one node only (the command will be started automatically on the other node):
# cldev populate

c) check everything is ok
# for i in `seq 7 197`; do /usr/cluster/lib/sc/scsi-c inkeys -d /dev/did/rdsk/d${i}; done

Community Discussions

Still have questions? Come engage with the My Oracle Support - Oracle Solaris Cluster Community, to search for similar
discussions or start a new discussion on this subject.

REFERENCES

NOTE:1008224.1 - Solaris Cluster SCSI Reservations Types and Fencing Options - How To Check/Set/Configure SCSI-2, SCSI-3,
PGRE on Devices and Quorum
NOTE:1008156.1 - Solaris Cluster 3.x and 4.x How To Identify SCSI Reservations
Didn't find what you are looking for?