Session #1 Understanding Metrocluster With Continuous Access XP

HP Metrocluster with Continuous
Access P9000/XP
Session #1
Understanding Metrocluster with Continuous Access XP
© Copyright 2011 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential
Presenters
Deepthi
Velisetti
Ananth
Arumugam
Key Takeaways
Objectives for this session
•Understand the building blocks of the

Metrocluster Solution
•Understand Metrocluster package

attributes and their significance
•Understand how Metrocluster manages

replication during failover and failback
scenarios
3 © Copyright 2011 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP Confidential
AGENDA
• High Availability and Disaster Recovery Solutions
• Overview of Continuous Access
• Overview of Metrocluster CA P9000/XP
• Configuring Metrocluster CA P9000/XP
• Break 15 min
• Failover/Failback Scenarios
• Best Practices
• Troubleshooting
• Documentation for further reading
• Q&A
Feel free to ask questions at any moment.

We encourage you to use the Virtual Room Question Button
How to ask questions in the virtual room
1
4
1. Open the Questions window
2. Enter your question and click Submit
3. Use Me Too if interested in other questions
4. Click the arrows to view the responses
Overview of High Availability and
Disaster Recovery Solutions
Terminology
•Quantum of acceptable data currency upon a recovery

Recovery Point from a disaster.
Objective or RPO •Specified in a range of minutes worth of data that can
be lost on a disaster
•The acceptable time to recover from a disaster.

Recovery Time •Specified in a range minutes within which
Objective or RTO application data is recovered and the services are
online.
HP Confidential
High Availability and Disaster Recovery Solutions
Disaster
Recovery
Solutions
High
•Stretched or
Availability Redundant clusters
across different data
•Local cluster with one centers with data
site in a single data replication
center
8
Local Serviceguard Cluster
Redundant networks carrying

cluster heartbeats
(1 cluster)
HP Serviceguard Extended Distance Cluster
Local Cluster with volume manager Mirroring
Redundant networks
carrying cluster
heartbeats
Mirroring (1 cluster)
HP Confidential
HP Metrocluster
Stretched Cluster with Data Replication
Redundant networks
carrying cluster
heartbeats
(1 cluster) Synchronous or
Asynchronous
Replication
HP Confidential
HP Continentalclusters
Redundant Clusters with Data Replication
Serviceguard Cluster1 Serviceguard Cluster 2
Redundant networks
Redundant networks
carrying cluster
carrying cluster
heartbeats
heartbeats
(1 cluster)
(1 cluster)
Physical or Logical Replication
HP Confidential
Serviceguard Disaster Recovery Solutions Products
DRS Extended Metrocluster Continentalclusters
Distance Cluster
Distances Upto 100 kms Upto 300 kms More than 300 kms
Replication Type Host based mirroring Array based Array based or software
replication based replication
Supported Array N/A •EMC SRDF •EMC SRDF

replication •Continuous Access XP •Continuous Access XP and
technologies and EVA EVA
•3par remote copy
Best RPO / RTO RPO = 0 RPO = 0 RPO = 0
on complete site RTO = seconds - few RTO= few minutes RTO = few minutes
failure minutes (Automatic) (Manual)
Supported LVM/VxVM mirroring N/A Oracle Data Guard.

software based Flexibility to configure
replication additional technologies
technologies
* These are indicative numbers and not absolute numbers.

HP Confidential
Overview of Continuous Access
Data Replication for P9000 and XP Series of Disk Arrays
Continuous Access - Device Groups
Device
Device Group is a Set of XP
logical disks for replication
where CA is defined
between them Group
PVOL and SVOL are primary

and secondary volumes of a
Device Group
PVOLs SVOLS
15
Continuous Access – Types of Replication Mode
• Data transfer is acknowledged only after the data has been

Synchronous written to the remote site
• Shorter Distances
• Not resilient to link failures
• Data from one disk is passed only to a local server before an

Asynchronous acknowledgement is received
• Longer Distances
• Resilient to link failures
16
Continuous Access Synchronous Replication
• When the CA link fails, or if the entire remote (SVOL)

Fence Level
data center fails, all I/Os will be refused to the PVOL
Data
devices until the Continuous Access link is restored
• This is used when DATA CONCURRENCY is crucial

When to use
than application availability
17
Fence Levels in Synchronous Replication
• When all Continuous Access links fail, the application

Fence Level
will continue to modify the data on PVOL side, however
Never
the new data is not replicated to the SVOL side
• This is used when Application availability is crucial

When to use
than data currency
18
Understanding Async replication further
Uses Side file

CA Async
paircreate -g <devgroup> -f
async -vl -c 15
Types of Aysnc
Replication Uses Journal volumes
CA Journal paircreate -g <device_group> -f

Async async -vl -c 15 -jp <id> -js <id>
An improvement over CA Async
19
Understanding Continuous Access Async
A copy of the data with a sequence number is saved in an internal buffer,
known as the side file, for later transmission to the remote XP disk array
20
Understanding Continuous Access Journal
The pull-style replication engine contributes to resource
Uses “disk- optimization
based
journaling”
and “pull- Journal contains metadata such as both time stamp and
style sequence number to enable correct write order ensuring
replication” data consistency and integrity
21
Fence Level in Asynchronous Replication
• Allows I/O activity to continue even if the Continuous

Fence Level Access link is temporarily unavailable as the transaction
Async is considered as complete once the data is written to
the side file
When to use • This is used when write performance is crucial
22
Continuous Access Fence Level
Fence Level/Replication DATA NEVER ASYNC
Mode
CA Synchronous Used when Used when NA

DATA currency is APPLICATION
more important availability is more
than application important than
availability application availability
CA Asynchronous NA NA Used when write

performance is crucial
CA Asynchronous Journal NA NA Used when write

performance is crucial
23 Tiger Team Technical Training - HP RESTRICTED (HP/Partners)

Continuous Access Pair States
• Simplex state - there is no affinity with a PVOL or
SMPL
SVOL
• Data is synchronized between the PVOL and

PAIR
SVOL
• Data is being synchronized between the PVOL

COPY and SVOL. During this state, the data in the
SVOL is inconsistent.
• Affinity between the pairs is suspended; bitmaps

PSUS
mark changed tracks
• Affinity between the pairs is suspended due to

PSUS (SSWS) failover to the SVOL; (bitmaps mark changed tracks for
future resynchronization from SVOL to PVOL or failback.)
24
Continuous Access Pair States
• Affinity between the pairs is suspended due to a
PSUE / PDUB
hardware error
• Not all logical devices in a group have the same

INCSTG
state
EX_ENORMT • Cannot contact a host to access the array
• Cannot access the array due to I/O connection

EX_CMDIOE
problem
• The side file or journal volumes has reached

PFUL
High Water Mark, I/O delayed
• The side file or journal volumes has reached
PFUS Threshold, no I/O to SVOL, changes tracked on
bitmap
25
Overview of Metrocluster CA
P9000/XP
Metrocluster CA P9000/XP
Solution Configuration
•Software*:
• Metrocluster with Continuous Access for P9000/XP A.11.00
• Serviceguard A.11.20
• MCDREnabler A.10.01
• RAID Manager 01.24.13 (P9000) 01.23.08(XP)
•Hardware
• Equal Number of Nodes on both the sites (Unequal number of nodes not
supported)
• Quorum Server OR Arbitrator Nodes on a 3rd Site
• XP or P9000 series Disk Arrays
• DWDM or SONET/SDH Links between the sites
* All versions are recent versions on HP-UX 11i v3 as of this training

HP Confidential
Inter Site Link Requirements
Latency
Distance Between Sites = minimum of ( Cluster Latency Requirements and

Replication Latency Requirement)
Serviceguard Requires:
Round Trip Latency not exceeding 200 milli seconds.
Synchronous Replication:
Typically ~3 milli seconds
Asynchronous Replication:
Typically ~10 milli seconds
HP Confidential
Metrocluster CA P9000/XP Solution Support
Supported Arrays , Server and Workloads on HP-UX
Supported
Supported Supported Supported
Volume
Array Servers Workloads
Managers
XP512/48 Oracle 10g R2/11g
R1/11gR2 RAC *
XP1024/128 VxVM/VxFS/CVM/CFS
HP 9000
5.0/5.0.1/5.1 SP1
XP10000 SAP using SGeSAP
XP12000
XP20000 HP Integrity VM
LVM 1.0 /LVM
XP24000 HP Integrity 2.x/SLVM 1.0/SLVM
2.x
ECMT applications
P9500
HP Confidential
29 HP Restricted Information
Metrocluster CA P900/XP-Solution Architecture
3rd Location
(Site 3)
Each main data center (sites 1 and 2)
Quorum Service must have the same number of nodes
protected in a
separate cluster
Clients Serviceguard Sub-cluster B
Serviceguard Sub-cluster A IP Subnet K or L (different subnets supported)
IP Subnet K Volume Manager: SLVM or CVM
Ethernet
Ethernet
Volume Manager: SLVM or CVM Network
Network
TCP/IP DWDM
Channels
Up to 8- Up to 8-
node node
Sub- Up to 300 km Sub-
cluster w/ DWDM cluster
Data Replication
DWDM Channels
DWDM
DWDM
(Site 1) Bi-directional Data replication using (Site 2)
DATA CENTER 1 Continuous Access XP or EVA, DATA CENTER 2
or EMC SRDF
HP Confidential
30
Metrocluster CA P9000/XP – Solution Components
RAID Manager configured to start as
RM part of boot. Present command device to all nodes in
CMD
the site
MC Metrocluster Software Package with Metrocluster Module
SG SG
SG SG
1
RM RM
RM
RM RM
MC MC
MC MC
DWDM DWDM
DWDM Links Continuous Access

and SG Heartbeat
CMD PVOL CMD

Continuous Access Device Group
SVOL
HP Confidential
Metrocluster CA P9000/XP – Components
Metrocluster Software Bundle
•Metrocluster Binary
• Continuous Access storage preparation logic
•Metrocluster Module
• dts/mcxpca
• Glue that binds Continuous Access replication handling logic with Serviceguard
•Commands and Utilities
• cmdrprev
• Disk Group monitor service
• Sample RC scripts to start RAID Manager
HP Confidential
Metrocluster P9000/XP - How it Works
Metrocluster Metrocluster RAID Command
Serviceguard
Module Binary Manager Device
Start the package
Calls MC binary Queries Device
Group Status
Request
Response
Returns Device
Group Status
MC evaluates the action to be taken
Issue appropriate
takeover command
Request
Response
Returns command
Returns decision Status
Continue with starting other

package resources ( VG, FS, IP,
Services etc.)
HP Confidential
Metrocluster CA P9000/XP - Features
Solution Capabilities
•Traditional Serviceguard Failover
• Preferred and Adoptive node Failover Model
•Automatic Site Aware Failover *
• Failover within the site before failing over to remote site
•Manual Site Failover *
• Package will remain down if there are no nodes to run in the current site
• Operator must start it manually on desired site after fixing the problem
•Device Group Monitor
• A service to monitor replication and protect data at remote site.
•Data Replication preview
• Check what will happen to replication if package is to failover to the node where the command is run
•Site Aware Disaster Tolerant Architecture
• Site Controller
•Failover complex workloads configured using multiple interdependent multi-node packages across sites
• Sub Cluster
•Allow upper layer clusters to be created in a site local fashion
*Requires Sites to be configured in the cluster configuration file

HP Confidential
Automatic Failover
Serviceguard Failover
Preferred Node Affinity (configured_node)
Manual Package
Start
P P
N1 N2 N3 N4 N1 N2 N3 N4
1 2
P P
N1 N2 N3 N4 N1
N1 N2
N2 N3
N3 N4
4 3
HP Confidential
Site Aware Failover – Automatic Inter Site Automatic Failover
Site Affinity (site_preferred) Manual Package

Start
P P
N1 N2 N3 N4 N1 N2 N3 N4
1 2
P P
N1 N2 N3 N4 N1 N2 N3 N4
4 3
HP Confidential
Automatic Failover
Site Aware Failover - Manual Inter Site
Manual Package
Manual inter site failover (site_preferred_manual) Start
P P
N1 N2 N3 N4 N1 N2 N3 N4
1 2
P P
N1 N2 N3 N4 N1 N2 N3 N4
4 3
HP Confidential
Device Group Monitor
Monitor and Automate Replication Management
Notification on Replication Failures
• Monitor the status of the P9000 or XP/Continuous Access device group used in
a package and raise notification
Resynchronization on Link Recovery
• Provides the ability to perform automatic resynchronization of a device group

upon link recovery
Protects data consistency during a full copy
• Split a BC PAIR of the SVOL before resynchronization
*Requires Sites to be configured in the cluster configuration file

HP Confidential
DR Preview
Evaluate Changes to Replication
cmdrprev -p <Metrocluster package name> | -e <environment

file location>
•Simulates Metrocluster storage preparation Logic

• Uses the current CA Device Group status
• Logs the storage preparation logic steps in the package log file
• The final action that will be taken on the CA Device Group
• Whether the package will start or fail
• Before starting a package on a node, use this command on the
node to preview the replication storage preparation logic
HP Confidential
Configuring Metrocluster CA
P9000/XP
Configuring Metrocluster CA P9000/XP
High Level Steps
Install & Configure Configuring
Setup Hardware &
Application Software Application as
Install required
( Place data on the Metrocluster
Software
replicated disks) packages
Configure RAID Configuring volume

manager instance on groups over
all nodes replicated
Configure CA Device
Configure
Group in RAID
Serviceguard cluster
Manager
HP Confidential
Configuring RAID Manager Instance
Perform the steps on all nodes
• Install the Raid Manager,
• Select a RAID Manager instance number to use with Metrocluster
• Add a entry for the RAID Manager instance in /etc/services
– horcm <instance-number> <port-number> /udp
• Identify command devices to use with the RAID Manager instance
– ioscan –fnC disk | grep OPEN-CM
• Create RAID Manager configuration file with the identified instance number
– Use the template RAID Manager configuration file
– # cp /etc/horcm.conf /etc/horcm0.conf
• Specify the following sections in the RAID Manager configuration file:
– HORCM_CMD: Command Device Name (more than one is allowed)
– HORCM_MON: hostname
• Start the Raid Manager instance
– #horcmstart.sh <instance number>
HP Confidential
RAID Manager Configuration File
Basic RAID Manager without any CA Device Groups
HP Confidential
Configure CA Device Group with RAID Manager
instance
Perform the steps on all nodes
• Specify LUNS for the CA Device Group using the HORCM_DEV keyword in the
RAID manager configuration file
– ls /dev/rdsk/* | raidscan -find –fx -I<instance number>
• Specify the remote node IP and RM Manager port in the RAID Manager
configuration file under the HORCM_INST section
• Restart the RAID Manager instance
– #horcmstart.sh <instance number>
HP Confidential
RAID Manager Configuration File
With CA Device Groups Configured
HP Confidential
Creating Device Groups
To create CA Sync •# paircreate -g <devgroup> -f

Device Group <data|never> -vl -c 15
To create a CA
• # paircreate -g <devgroup> -f async
Async Device
-vl -c 15
Group
To create a CA
•# paircreate -g <device_group> -f
Async Journal
Device Group
async -vl -c 15 -jp <id> -js <id>
46
Metrocluster Cluster deployment
Configuration steps
Traditional approach Easy deployment approach
•Install OS
•Connect the network interfaces between nodes. •Install OS
Hardware
configuration
•Connect the storage HW. •Connect the network
interfaces between
nodes.
•Populate entries in cmclnodelist file.
Hardware •Connect the storage
Security
configuration
configuration HW.
•Heartbeat network configuration.

Network
configuration
•Configure /etc/nsswitch.conf
•cmdeploycl command in
CLI
System
•Configure /etc/hosts
•/etc/inetd.conf Easy
configuration
deployment
•Generate Cluster Ascii file using cmquerycl
•Edit cluster Ascii File
Final •Apply configuration file
configuration
Metrocluster Cluster Configuration
Traditional Approach
•Run cmquerycl to discover network configuration and create a basic cluster

configuration file
• EXTENDED SUBNET:
cmquerycl -n <node 1> -n < node 2> … -q <quorum server name> -C <cluster configuration
file name>
• CROSS SUBNET:
cmquerycl -w full -n <node 1> -n < node 2> … -q <quorum server name> -C <cluster
configuration file name>
•Customize the cluster configuration file
• Specify the quorum server IP address Specify the arbitrator nodes using NODE_NAME
attribute
• When using Site Aware failover policies, specify site definitions (optional)
•Check and Apply the cluster configuration file
cmapplyconf -C <cluster configuration file>
HP Confidential
Metrocluster – Cluster configuration file
• CLUSTER_NAME mcia07_cluster
• QS_HOST fep3.ind.hp.com
• QS_POLLING_INTERVAL 300000000
• SITE_NAME SITE1
• SITE_NAME SITE2
• NODE_NAME mcia07
– SITE SITE1
– NETWORK_INTERFACE lan0
– HEARTBEAT_IP 15.154.63.34
• NODE_NAME mcia08
– SITE SITE2
– NETWORK_INTERFACE lan2
– HEARTBEAT_IP 15.154.63.43
• MEMBER_TIMEOUT 14000000
Cluster Deployment Examples
Using cmdeploycl command
 Create site aware Metrocluster

 cmdeploycl -s site1 -n node1 -s site2 -n node2
-c MyAppCluster –q qs_host
Modular Package
Modules – the building blocks of a package
• cmmakepkg –m ecmt/ecmt/oracle –m dts/mcxpca

Create package_name.conf
Check • cmcheckconf -P package_name.conf
Apply • cmapplyconf -P package_name.conf
• Metrocluster XPCA package configuration must be created using the

dts/mcxpca module
• Along with:
• If an application with an ECM toolkit is used, then include the corresponding toolkit
module
Basic Metrocluster Parameters
Configuring the basic Metrocluster package parameters
Parameter Name Possible Values Significance
package_name Any valid string The unique identifier for a

package in the cluster
failover_policy min_package_node, Decides which node the

configured_node, package will failover when
site_preferred, the node where it currently
site_preferred_manual runs fails.
dts_pkg_dir Any valid directory This is the directory in

which the FORCEFLAG is
touched to start up the
metrocluster package
under special scenarios
Package Attributes
Attributes Meaning
CLUSTER_TYPE Type of clustering environment

Should be set to “metro” if this is a Metrocluster environment and “continental” if this is a
Continentalclusters environment.
DEVICE_GROUP The Raid Manager device group for this package. This device group is defined in the
/etc/horcm<#>.conf file.
FENCE Fence level (never/data/async)

Fence level is ascyn for a journal group pair.
HORCMINST Raid Manager Instance that the script will communicate with. This instance of Raid
Manager must be started on all nodes before this package can be successfully started.
HORCMPERM RAID Manager Protection Facility on the Continuous Access devices.

If the RAID Manager Protection Facility is disabled, set this variable to MGRNOINST.
(default)
HP Confidential
Package Attributes
AUTO_FENCEDATA_SPLIT
Attribute AUTO_FENCEDATA_SPLIT
Local State Remote State Fence Level Values
PVOL_PSUE SVOL_PAIR DATA Value=0(Default)

PVOL_PSUE EX_ENORMT Do not start. Exit with 1.
PVOL_PSUE EX_CMDIOE
Value = 1 or FORCEFLAG=yes
Perform a PVOL takeover. After the takeover succeeds,
package starts with a warning message about the data is
not remotely protected in the package’s control log file.
HP Confidential
Package Attribute
AUTO_NONCURDATA FENCE: NEVER or DATA
Attribute AUTO_NONCURDATA
Local State Remote State Values
PVOL_PSUS EX_ENORMT Value=0(Default)

PVOL_PSUS EX_CMDIOE Do not start. Exit with 1.
Starts with a warning message about non-current data in the
package’s control log file.
SVOL_PAIR PVOL_PSUE Value=0(Default)
SVOL_PAIR EX_ENORMT Do not start. Exit with 1.
SVOL_ PAIR EX_CMDIOE
Perform SVOL takeover, which changes SVOL to PSUS
(SSWS). After the takeover succeeds, package starts with a
warning message about non-current data in the package’s
control log file.
HP Confidential
5
Package Attribute
AUTO_NONCURDATA FENCE: Async Journal
SVOL- PVOL-PAIR; (Continuous Value=0(Default)

PAIR;MINAP=0 MINAP>0, Access- Do not start. Exit with 1.
MINAP=0 Journal)
EX_ENORMT Value = 1 or FORCEFLAG=yes
EX_CMDIOE Perform SVOL takeover, which changes SVOL to
PSUS (SSWS). After the takeover succeeds, package
starts with a warning message about non-current
data in the package's log file
HP Confidential
5
Package Attribute
AUTO_NONCURDATA and FENCE: Async
Local State Remote State Values
PVOL_PSUS EX_ENORMT Value=0(Default)

PVOL_PSUS EX_CMDIOE Do not start. Exit with 1.
PVOL_PFUS EX_ENORMT
PVOL_PFUS EX_CMDIOE Value = 1 or FORCEFLAG=yes
Starts with a warning message about non-current data in the
package’s control log file.
SVOL_PAIR PVOL_PSUE Value=0(Default)
SVOL_PAIR EX_ENORMT Do not start. Exit with 1.
SVOL_ PAIR EX_CMDIOE
SVOL_PFUL PVOL_PSUE Value = 1 or FORCEFLAG=yes
SVOL_PFUL EX_ENORMT Perform SVOL takeover, which changes SVOL to PSUS
SVOL_PFUL EX_CMDIOE (SSWS). After the takeover succeeds, package starts with a
control log file.
HP Confidential
5
Package Attributes
Attribute AUTO_SVOLPSUS
SVOL_PSUS PVOL_PSUS NEVER/DAT Value=0(Default)

SVOL_PSUS EX_ENORMT A/ ASYNC Do not start on local node. exit 2.
SVOL_PSUS EX_CMDIOE Require user intervention to either fix the problem by
resynchronizing the PVOL and SVOL or force the package
to start on this node by creating the FORCEFLAG.
control log file.
The risk of using this option is the SVOL data may be
inconsistent and the application may fail. However, there
is also a chance that the data is actually consistent, and it
is okay to startup the application.
HP Confidential
5
Package Attributes
AUTO_PSUEPSUS
Attribute AUTO_PSUEPSUS
PVOL_PSUE SVOL_PSUS(SS NEVER/DAT Value=0(Default)

WS) A/ASYNC Do not start. Exit with 1. This will protect any data that
may have been in the MCU side file or differential data in
PVOL_PFUS SVOL_PSUS(SS ASYNC the PVOL when the outage occurred.
WS)
pairresync -swapp , hortctakeover
* If pairresync-swapp works, package starts up.
*If pairresync-swapp fails, package does not start with exit

1.
This resynchronization will over write any data that was in
the MCU prior to the primary (PVOL) side failure.
HP Confidential
5
Package Attributes
Attribute AUTO_ PSUSSSWS
PVOL_PSUS SVOL_PSUS(SS NEVER/DAT Value=0(Default)

WS) A/ASYNC Do not start. Exit with 1. Require user intervention to
choose which side has the good data and resynchronizing
the PVOL and SVOL or force the package to start by
creating the FORCEFLAG file.
pairresync -swapp , hortctakeover
* If pairresync-swapp works, package starts up.
*If pairresync-swapp fails, package does not start with exit

1.
When resychronizing dtata from SVOL to PVOL side, the
changed data in the PVOL may be lost.
HP Confidential
6
Package Attributes
Attribute AUTO_ SVOLPFUS
SVOL_PFUS PVOL_PFUS ASYNC Value=0(Default)

SVOL_PFUS EX_ENORMT Do not start on local node. Exit with 2, allowing restart on
SVOL_PFUS EX_CMDIOE another node.
Require user intervention to either fix the problem by
resynchronizing the PVOL and SVOL or force the package
control log file.
SVOL data may actually be non-current and the data
written to the PVOL side after the hardware failure may be
lost.
HP Confidential
6
Package Attributes
Attribute AUTO_SVOLPSUE
SVOL_PSUE PVOL_PSUE ASYNC Value=0(Default)

Do not start on local node. Exit with 2, allowing restart on
another node.
SVOL_PSUE EX_ENORMT NEVER/DAT Require user intervention to either fix the problem by
SVOL_PSUE EX_CMDIOE A/ASYNC resynchronizing the PVOL and SVOL or force the package
control log file.
SVOL data may actually be non-current and the data
written to the PVOL side after the hardware failure may be
lost.
HP Confidential
6
Package Attributes
Attributes Meaning
HORCTIMEOUT In asynchronous mode, the timeout value in the horctakeover command, -t <timeout>,
time to wait while horctakeover re-synchronizes the delta data from the PVOL to the
SVOL.
Set the HORCTIMEOUT variable to a value greater than the Continuous Access link
timeout value. The package startup timeout value must be greater than the
HORCTIMEOUT value.
Pkg Startup Timeout > HORCTIMEOUT > Continuous Access link timeout value
It is ignored in synchronous mode.
HP Confidential
6
Setup Device Group Monitor for a Metrocluster Package
Configure DG Monitor Service in the Metrocluster Package
Attribute Values
SERVICE_NAME Specify unique Service Name
SERVICE_CMD /usr/sbin/DRMonitorXPCADevGrp
SERVICE_RESTART Unlimited
HP Confidential
Only Monitor and Notify CA Device Group Failures
Specify DG Monitor Attributes as below
Attribute Value
MON_POLL_INTERVAL 10 Minutes
MON_NOTIFICATION_FREQUENCY 3
MON_NOTIFICATION_EMAIL [email protected]
MON_NOTIFICATION_SYSLOG 1
MON_NOTIFICATION_CONSOLE 1
AUTO_RESYNC 0
This configuration will cause a notification if the CA Device Group has

remained failed for 30 minutes.
HP Confidential
Notify on CA Device Group Failures and Auto Resync on Link
Recovery
Specify DG Monitor Attributes as below
Attribute Value
MON_POLL_INTERVAL 10 Minutes
MON_NOTIFICATION_FREQUENCY 3
MON_NOTIFICATION_EMAIL [email protected]
MON_NOTIFICATION_SYSLOG 1
MON_NOTIFICATION_CONSOLE 1
AUTO_RESYNC 1 - Automatically Resync PVOL to
SVOL when Links are restored
2 – Initiate Resync only if package
directory has MON_RESYNC file
HP Confidential
Protect Data at SVOL side during a Full Copy when Synchronous replication
Problem: •Resync is Out of Order Write in Synchronous Replication

•SVOL will be Inconsistent during Resync
•A disaster at PVOL sited during a Resync will result in no consistent
data at both disk arrays
Solution: Configure Business Copy & Device Group Monitor
Device Group Monitor automatically splits the BC before resync
DWDM Links for Continuous Access
CMD PVOL Continuous Access Device Group CMD

SVOL
Mirroring
Mirroring
BC BC
HP Confidential
Failover/Failback Scenarios
HP Confidential
Failover scenario 1
Reason for failover APP
• Failover to recovery site due to application failure on all APP
nodes in primary site
Configuration
• Fence level = any supported value
• Replication type = any
PVOL SVOL
Metrocluster Behavior
• Swaps the replication role by issuing horctakeover Location State
Site 1 PVOl_PAIR
Impact of Auto Variables or Forceflag Site 2 SVOL_PAIR
• None
Location State
Operator Intervention Site 1 SVOL_PAIR
• None Site 2 PVOL_PAIR
69
Failback scenario 1
Reason for failover
• Issues at the primary site has been fixed. Application APP
can now start up in the primary site.
Configuration
• Fence level = any supported value
PVOL SVOL
• Swaps the replication role by issuing horctakeover Location State
Site 1 PVOl_PAIR
Impact of Auto Variables or Forceflag Site 2 SVOL_PAIR
• None
Location State
Operator Intervention Site 1 SVOL_PAIR
• None Site 2 PVOL_PAIR
70
Failover scenarios
Reason for failover
Failure of all nodes in the primary site
Configuration
•Fence level = any supported value
•Replication type = any except CAJ
Do not start the package
Impact of Auto Variables or Forceflag

ENORMT
SVOL
•If AUTO_NONCURDATA = “1” or FORCEFLAG is present,
•Issue horctakeover -S (SVOL-takeover)
Location State
•in case of fence = data or async, it results in swap
Site1 EX_ENORMT
takeover
Site2 SVOL_PAIR
•Incase of fence = never, it results in svol takeover*
* MC XPCA A.11.00 patches(PHSS_41663 11iv2, PHSS_41664 11iv3) results in swap S
Location State
takeover w
a Site1 EX_ENORMT
p
Site2 SVOL_SSWS
Operator Intervention
Location State
• Touch FORCEFLAG in dts_pkg_dir if AUTO_NONCURDATA=0 Site1 EX_ENORMT
71 Site2 PVOL_PAIR
Failover scenarios
Reason for failover
• Failure of all nodes in the primary site
Configuration
• Fence level = async
• Replication type = CAJ
Metrocluster Behavior SVOL

ENORMT
• Issue horctakeover when link is up
• When link is down check AUTO variable or FORCEFLAG
Location State
Impact of Auto Variables or Forceflag Site1 EX_ENORMT
Site2 SVOL_PAIR
• If AUTO_NONCURDATA = “1” or FORCEFLAG is present when link is down,
• Issue horctakeover -S (SVOL-takeover) S Location State
w
a Site1 EX_ENORMT
p
Site2 SVOL_SSWS
Location State
• Touch FORCEFLAG in dts_pkg_dir if AUTO_NONCURDATA=0, when link is down
Site1 EX_ENORMT
72 Site2 PVOL_PAIR
Failback scenarios
Reason for failback
All nodes in the primary site are up
Configuration
•Replication type = any
Issue horctakeover - swap takeover
SVOL
Impact of Auto Variables or Forceflag PVOL
None
Location State
Operator Intervention Site 2 PVOL_PAIR
Site 1 SVOL_PAIR
• None
Location State
Site 2 SVOL_PAIR
Site 1 PVOL_PAIR
73
Failback scenarios
Reason for failback
All nodes in the primary site are up
Configuration
•Replication type = Sync
Issue pairresync –swapp and issue horctakeover - swap
takeover
PVOL_PAIR
Impact of Auto Variables or Forceflag SSWS
None
Location State
Operator Intervention Site 1 PVOL_PAIR
Site 2 SVOL_SSWS
• None
Location State
Site 2 SVOL_PAIR
Site 1 PVOL_PAIR
74
Failover scenarios
Reason for failover 2
APP
• 1 Link failure followed by
• 2 application failure in all nodes APP
Configuration
• Fence level = any 1
• Replication type = any except CAJ
• do not start package PVOL_PSUE SVOL_PAIR
Impact of Auto Variables or Forceflag Location State

Site1 PVOL_PSUE
• Forceflag set, then start package
Site2 SVOL_PAIR
• If AUTO_NONCURDATA=1, Start package
Location State
Operator Intervention Site1 PVOL_PSUE
Site2 SVOL_SSWS
• Touch FORCEFLAG in dts_pkg_dir if AUTO_NONCURDATA=0
75
Failover scenarios
APP
• application failure in all nodes APP
2
Configuration
• Fence level = async 1
PVOL SVOL
• do not start package
PVOL_PSUE SVOL_PSUE
Impact of Auto Variables or Forceflag OR
• Forceflag set, then start package Location State Location State
Site1 PVOL_PSUE Site1 PVOL_PAIR
• If AUTO_SVOLPSUE=1, Start package when recovery state is
SVOL_PSUE and issue SVOL takeover Site2 SVOL_PSUE Site2 SVOL_PAIR
• If AUTO_NONCURDATA=1, Start package when recovery state

is SVOL_PAIR and issue SVOL takeover
Location State
Operator Intervention Site1 PVOL_PSUE
• Touch FORCEFLAG in dts_pkg_dir if Site2 SVOL_SSWS

AUTO_SVOLPSUE=0/AUTO_NONCURDATA=0
76
Failback scenarios
Reason for failover
• Link is restored followed by fixing of issues that halted the
application
Configuration
• Fence level = any
Metrocluster Behavior PVOL_PSUE

SVOL_SSWS
Location State
Impact of Auto Variables or Forceflag Site1 PVOl_PSUE

Site2 SVOL_SSWS
• If AUTO_PSUEPSUS = “1” or FORCEFLAG is present,
• Swap the role and Resynchronizes data from the recovery site to primary:
pairesync –swapp command
• Issues horctakeover on the primary site to replicate data to the recovery site Location State
site1 PVOl_PAIR
Operator Intervention site2 SVOL_PAIR
• Touch FORCEFLAG in dts_pkg_dir if AUTO_PSUEPSUS=0

77
Failover scenarios
• 2 failure of all nodes in primary side
Configuration
• Replication type = any except CAJ
ENORMT SVOL

Location State
• Forceflag set, then start package
Primary EX_ENORMT
• If AUTO_NONCURDATA=1, Start package
Secondary SVOL_PAIR
Operator Intervention Location State

Primary EX_ENORMT
• Touch FORCEFLAG in dts_pkg_dir if AUTO_NONCURDATA=0
Secondary SVOL_SSWS
78
Failover scenarios
• failure of all nodes in primary side
2
Configuration
1
ENORMT SVOL_PSUE
Impact of Auto Variables or Forceflag OR
• Forceflag set, then start package SVOL_PAIR

• If AUTO_SVOLPSUE=1, Start package when recovery state is Location State
SVOL_PSUE Site1 EX_ENORMT
• If AUTO_NONCURDATA=1, Start package when recovery state Site2 SVOL_PSUE/
is SVOL_PAIR SVOL_PAIR
Location State
Operator Intervention Site1 EX_ENORMT
• Touch FORCEFLAG in dts_pkg_dir if AUTO_SVOLPSUE=0/ Site2 SVOL_SSWS

AUTO_NONCURDATA=0
79
Failback scenarios
Reason for failover
• The link and the primary site nodes are restored
Configuration
PVOL_PSUE SVOL_SSWS

• If AUTO_PSUEPSUS = “1” or FORCEFLAG is present,
• Swap the role and Resynchronizes data from the recovery site to primary: Location State
• pairesync –swapp command Site1 PVOL_PSUE
• Issues horctakeover on the primary site to replicate data to the recovery site
Site2 SVOL_SSWS

Site1 PVOL_PAIR
• Touch FORCEFLAG in dts_pkg_dir if AUTO_PSUEPSUS=0
80 Site2 SVOL_PAIR
Failover/Failback scenarios
APP
• 1 Manual suspend followed by
• 2 application failure on primary side APP
Configuration
SVOL_PSUS
• do not start package PVOL_PSUS
Location State
Site1 PVOL_PSUS
• Forceflag set, then start package Site 2 SVOL_PSUS(

SSUS
• If AUTO_SVOLPSUS=1, Start package
Location State
Site1 PVOL_PSUS
• Touch FORCEFLAG in dts_pkg_dir if AUTO_SVOLPSUS=0 Site2 SVOL_SSWS
81
Failover/Failback scenarios
• 1 Manual suspend followed by
• 2 failure of all nodes in primary side
Configuration
• do not start package ENORMT SVOL_PSUS

Location State
• Forceflag set, then start package Site 1 EX_ENORMT
• If AUTO_SVOLPSUS=1, Start package Site 2 SVOL_PSUS(
SSUS

Site 1 EX_ENORMT
• Touch FORCEFLAG in dts_pkg_dir if AUTO_SVOLPSUS=0
Site 2 SVOL_SSWS
82
Failback scenarios
Reason for failover
• Fix the issues and failback
Configuration
• Fence level = any
Metrocluster Behavior PVOL_PSUE

SVOL_SSWS
Location State
Impact of Auto Variables or Forceflag Site1 PVOl_PSUE

Site2 SVOL_SSWS
• If AUTO_PSUSSSWS= “1” or FORCEFLAG is present,
• Swap the role and Resynchronizes data from the recovery site to primary:
pairesync –swapp command
• Issues horctakeover on the primary site to replicate data to the recovery site Location State
site1 PVOl_PAIR
Operator Intervention site2 SVOL_PAIR
• Touch FORCEFLAG in dts_pkg_dir if AUTO_PSUSSSWS=0

83
Best Practices
Best practices
• Increases the availability of remote array Raid
Remote Manager
Command
Device
• Prevent errors due to periodic configuration changes

• Run as a cron job and save output to a file to
Cluster
analyze errors
Verification
• Some small steps to be taken into consideration

More best
practices
Best practices
• Increases the Availability of Raid Manager (especially in 2 node
Remote Command
Device
Metrocluster)
Node 1 Node 2
Node 1 Node 3
Remote Remote Local Raid

Raid Local Raid Manager
Manager Raid Manager Instance
Instance Manager Instance
Instance
SAN
CA PAIR
CMD RCMD
CMD
RCMD CMD
CA PAIR
Remote P9000
Local P9000 Array
Array
Best practices
• Prevent errors due to periodic configuration changes
Cluster
Verification
• Run as a cron job and save output to a file to analyze errors
Checks
When to run consistency
between
environment file
•Periodically and package
•After any array configuration
software,
cluster software
or Metrocluster
software
Checks if all the
upgrades/ Checks the raid
disks are part of
manager Cluster
patches Verification the device
version
group
How to run
• as a cron Verify if the

Metrocluster
job package
parameters are
valid
Best practices
• Some small steps to be taken into consideration
More best
practices
Use supplied RC scripts to start Raid Manager Instances during machine boot
Review package logs periodically
Migrate any legacy packages to modular style packages
Configure LVM timeouts for storage errors to be detected
Use cmdrprev before manually starting a Metrocluster package on a node
Configure and use Device Group Monitor

Troubleshooting
Troubleshooting – Check Replication
Status
Progress • pairdisplay -g pkgB -fc –CLI

of Copy Group PairVol L/R Port# TID LU Seq# LDEV# P/S Status Fence % P-LDEV# M
pkgB pkgD-disk0 L CL1-C 0 3 35422 463 P-VOL COPY NEVER 79 460 –
Operati pkgB pkgD-disk0 R CL1-F 0 3 35663 3 S-VOL COPY NEVER - 0 -
ons
• pairdisplay -g pkgB -fc –CLI

Slide Group PairVol L/R Port# TID LU Seq# LDEV# P/S Status Fence % P-LDEV# M
File Size pkgB pkgD-disk0 L CL1-C 0 3 35422 463 P-VOL PAIR ASYNC 35 3 –
pkgB pkgD-disk0 R CL1-F 0 3 35663 3 S-VOL PAIR ASYNC 0 463 -
HP Confidential
Troubleshooting – where to look for
errors and warnings
• /HORCM/log<HORCM instance
RAID Manager #>/HORCC_<node name>.log
Log File • e.g. /HORCM/log0/HORCC_node1.log
• Periodically review the files syslog.log

Package log and
and pkgname.log for messages,
syslog warnings and recommended actions
HP Confidential
Troubleshooting – Looking out for Pair states that
require Manual Intervention
When CA device
• Check if the raid manager instance is down and
group is in unknown bring it up manually if it is down
state
If the package starts

• User can run a pairresync command to re-establish
up with data data replication while the package is still running
replication suspended
If the disks on primary

• Issue pairresync with -swapp/-swaps when the CA
site are still intact after link is fixed
a failover
When Pairs are in • Pairs must be manually recreated if both the primary
and secondary P9000 or XP Series disk array are in
SMPL state SMPL (simplex) state
HP Confidential
Q&A
Miscellaneous Information
HA Quarterly Update Webcast
•Two hour session

Format • Two Edition Eastern and Western TZ
•Once in every quarter
•Updates on new releases and support statements for entire
Takeaways
Serviceguard Solutions portfolio
•Demo of new features
•Overview of upcoming releases
How to get •Q2 FY 2012 is scheduled for late March 2012

•Send a mail to Sujatha Ramani to get an invite
an invite ([email protected])
HP Confidential
Documentation for further reading
• Metrocluster with Continuous Access for P9000 and XP Manuals and
White Papers
http://www.hp.com/go/hpux-serviceguard-docs > Select “ HP
Serviceguard Metrocluster with Continuous Access for P9000 and XP”
• Serviceguard Solutions Support Page
http://haweb.ind.hp.com/Support/
• Serviceguard Support Knowledge Base
http://haweb.ind.hp.com/Support/cgi-bin/QueryKDB.cgi
• Documentation for HP storage division
http://spock.corp.hp.com
Who to contact when !
• Triage Issues (Call HP Response Center)
• Support Related Queries(Serviceguard Solutions Support [email protected] )
– Which X is supported with Metrocluster ?
– Which version of Y is supported with Serviceguard ?
• Solution Configuration Queries (HA ATC [email protected] )

– Is this a valid configuration … ?
– How do I … ? How to … ?
– Best Practices, Understand Trade Off
• Roadmap Related Queries (HA Marketing [email protected] )

– When will x supported with Serviceguard ?
– Licensing information
• Not sure where to send ?

– Send it to HA ATC

Thank You
Backup
Troubleshooting – Status Checks
Pair and
Journal • pairdisplay -g oradb –fe
Group Seq#, LDEV# P/S,Status, Fence, %, P-LDEV# M CTG JID AP EM E-
Group Seq# E-LDEV#
oradb 30053 64 P-VOL PAIR Never, 75 C8 - 1 0 2
Informati oradb 30054 C8 S-VOL PAIR Never, 64 - - 1 0
on
Journal
Volumes
• raidvchkscan –v jnl 0
Informati
on

Session #1 Understanding Metrocluster With Continuous Access XP

Uploaded by

Copyright:

Available Formats

Session #1 Understanding Metrocluster With Continuous Access XP

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Session #1 Understanding Metrocluster With Continuous Access XP

Uploaded by

Copyright:

Available Formats

HP Metrocluster with Continuous

•Understand the building blocks of the

•Understand Metrocluster package

•Understand how Metrocluster manages

Feel free to ask questions at any moment.

•Quantum of acceptable data currency upon a recovery

•The acceptable time to recover from a disaster.

Redundant networks carrying

Physical or Logical Replication

Supported Array N/A •EMC SRDF •EMC SRDF

Supported LVM/VxVM mirroring N/A Oracle Data Guard.

* These are indicative numbers and not absolute numbers.

PVOL and SVOL are primary

• Data transfer is acknowledged only after the data has been

• Data from one disk is passed only to a local server before an

• When the CA link fails, or if the entire remote (SVOL)

• This is used when DATA CONCURRENCY is crucial

• When all Continuous Access links fail, the application

• This is used when Application availability is crucial

Uses Side file

CA Journal paircreate -g <device_group> -f

An improvement over CA Async

• Allows I/O activity to continue even if the Continuous

When to use • This is used when write performance is crucial

CA Synchronous Used when Used when NA

CA Asynchronous NA NA Used when write

CA Asynchronous Journal NA NA Used when write

23 Tiger Team Technical Training - HP RESTRICTED (HP/Partners)

• Data is synchronized between the PVOL and

• Data is being synchronized between the PVOL

• Affinity between the pairs is suspended; bitmaps

• Affinity between the pairs is suspended due to

• Not all logical devices in a group have the same

EX_ENORMT • Cannot contact a host to access the array

• Cannot access the array due to I/O connection

• The side file or journal volumes has reached

* All versions are recent versions on HP-UX 11i v3 as of this training

Distance Between Sites = minimum of ( Cluster Latency Requirements and

DWDM Links Continuous Access

CMD PVOL CMD

MC evaluates the action to be taken

Continue with starting other

*Requires Sites to be configured in the cluster configuration file

Site Affinity (site_preferred) Manual Package

Notification on Replication Failures

Resynchronization on Link Recovery

• Provides the ability to perform automatic resynchronization of a device group

Protects data consistency during a full copy

• Split a BC PAIR of the SVOL before resynchronization

*Requires Sites to be configured in the cluster configuration file

cmdrprev -p <Metrocluster package name> | -e <environment

•Simulates Metrocluster storage preparation Logic

Configure RAID Configuring volume

To create CA Sync •# paircreate -g <devgroup> -f

•Heartbeat network configuration.

•Run cmquerycl to discover network configuration and create a basic cluster

 Create site aware Metrocluster

• cmmakepkg –m ecmt/ecmt/oracle –m dts/mcxpca

Check • cmcheckconf -P package_name.conf