SVC 4.2.1 Advanced Copy Services 4.3.14 Deleting a Consistency Group

Front cover SVC V4.2.1 Advanced Copy Services

New Advanced Copy Services functionality

Server integration examples

Scripting examples

Jon Tate Pall Thorir Beck Tom Jahn Dan Rumney

ibm.com/redbooks

International Technical Support Organization

SVC V4.2.1 Advanced Copy Services

May 2008

SG24-7574-00 Note: Before using this information and the product it supports, read the information in “Notices” on page ix.

First Edition (May 2008)

This edition applies to the IBM System Storage SAN Volume Controller V4.2.1.

© Copyright International Business Machines Corporation 2008. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents

Notices ...... ix Trademarks ...... x

Preface ...... xi The team that wrote this book ...... xi Become a published author ...... xiii Comments welcome...... xiii

Chapter 1. Introduction to SVC V4.2.1 Advanced Copy Services...... 1 1.1 New in FlashCopy ...... 2 1.1.1 Incremental FlashCopy ...... 2 1.1.2 Cascaded FlashCopy ...... 2 1.1.3 Dynamically configurable replication services space ...... 2 1.1.4 Grain size ...... 2 1.1.5 Bitmap space ...... 3 1.1.6 Cleaning mode ...... 3 1.2 New in Remote Copy ...... 3 1.3 Other new functionality in V4.2.1 ...... 3 1.3.1 Increased storage addressing...... 3 1.3.2 Volume management ...... 4 1.3.3 Testing managed disks before adding to MDisk groups ...... 4 1.3.4 Cache ...... 4

Chapter 2. Planning for Advanced Copy Services implementation ...... 5 2.1 First questions...... 6 2.2 Reasons for copying data ...... 6 2.3 Deciding what data to copy ...... 7 2.4 Ensuring usable copied data sets ...... 7 2.4.1 Understanding caching ...... 8 2.4.2 Understanding consistency...... 12 2.5 Accessing the copied data set ...... 16 2.6 Determining how much data can be replicated ...... 16 2.6.1 VDisk capacity limit ...... 17 2.7 Understanding your current environment ...... 18 2.7.1 Storage controller information...... 19 2.7.2 Server information...... 20 2.7.3 Rate of transfer of data ...... 20 2.7.4 Intersite latency...... 21 2.8 Administration of copy services...... 21 2.8.1 SSH keys ...... 21 2.8.2 Command auditing ...... 21 2.8.3 Authorization roles ...... 21 2.9 Removing workarounds with V4.2.1 new functions...... 22

Chapter 3. Remote Copy: Metro Mirror and Global Mirror ...... 25 3.1 Remote Copy services overview...... 26 3.1.1 Metro Mirror ...... 26 3.1.2 Cluster partnership ...... 27 3.1.3 Intercluster communication ...... 28

© Copyright IBM Corp. 2008. All rights reserved. iii 3.1.4 Maintenance of the intercluster link...... 28 3.1.5 I/O channel ...... 29 3.1.6 Background copy ...... 29 3.1.7 Remote Mirror relationship ...... 30 3.1.8 Copy relationship naming convention ...... 31 3.1.9 Supported methods for synchronizing...... 32 3.1.10 Remote Mirror consistency groups ...... 33 3.2 State overview...... 34 3.2.1 Connected or disconnected ...... 34 3.2.2 Consistent or inconsistent...... 35 3.2.3 Consistent or synchronized...... 36 3.2.4 Copy states ...... 36 3.3 Metro Mirror and Global Mirror configuration limits ...... 40 3.4 Remote Copy internals ...... 40 3.4.1 Read stability ...... 40 3.4.2 Write ordering ...... 41 3.4.3 Write consistency ...... 41 3.4.4 Write completion ...... 42 3.4.5 Freeze/Run on consistency groups...... 42 3.4.6 Management of difference map ...... 43 3.4.7 Non-volatile journal ...... 44 3.4.8 Quick Recovery Copy and Reconstruct ...... 45

Chapter 4. Implementing Remote Copy ...... 47 4.1 Implementation ...... 48 4.2 Implementation using the GUI...... 49 4.2.1 Creating a cluster partnership...... 49 4.2.2 Changing a cluster partnership ...... 53 4.2.3 Deleting a cluster partnership ...... 54 4.2.4 Creating a copy relationship ...... 55 4.2.5 Changing a copy relationship ...... 59 4.2.6 Starting a copy relationship...... 60 4.2.7 Stopping a copy relationship...... 61 4.2.8 Deleting a copy relationship ...... 63 4.2.9 Creating a consistency group ...... 64 4.2.10 Starting a consistency group...... 66 4.2.11 Stopping a consistency group...... 67 4.2.12 Renaming a consistency group...... 69 4.2.13 Reversing copy direction...... 70 4.3 Implementation using the command line...... 72 4.3.1 Listing cluster candidates ...... 72 4.3.2 Creating cluster partnership ...... 73 4.3.3 Changing cluster partnership ...... 74 4.3.4 Deleting a cluster partnership ...... 74 4.3.5 Creating a copy relationship ...... 75 4.3.6 Starting a copy relationship...... 76 4.3.7 Stopping a copy relationship...... 78 4.3.8 Changing a copy relationship ...... 78 4.3.9 Deleting a copy relationship ...... 79 4.3.10 Creating a consistency group ...... 79 4.3.11 Starting a consistency group...... 81 4.3.12 Stopping a consistency group...... 82 4.3.13 Changing a consistency group ...... 83 iv SVC 4.2.1 Advanced Copy Services 4.3.14 Deleting a consistency group ...... 83 4.3.15 Reversing copy direction...... 85 4.4 Site failover example...... 86 4.4.1 Site failover when production site loses backend storage ...... 87 4.4.2 Site failover using the GUI ...... 89 4.4.3 Site failover using the CLI ...... 91 4.5 Copy commands using the CLI ...... 94

Chapter 5. FlashCopy ...... 95 5.1 Introduction to FlashCopy ...... 96 5.1.1 FlashCopy and its uses...... 96 5.1.2 Using FlashCopy...... 97 5.2 FlashCopy functions and features...... 97 5.2.1 FlashCopy mappings ...... 98 5.2.2 FlashCopy consistency groups ...... 98 5.2.3 Copy background copy ...... 100 5.2.4 FlashCopy autodelete option ...... 101 5.2.5 Multiple Target FlashCopy ...... 101 5.2.6 Cascaded FlashCopy ...... 102 5.2.7 Combined Multiple Target FlashCopy and Cascaded FlashCopy ...... 103 5.2.8 Incremental FlashCopy ...... 104 5.3 FlashCopy internals ...... 105 5.3.1 Indirection layer...... 105 5.3.2 Grains ...... 105 5.3.3 Bitmaps ...... 106 5.3.4 Usable bitmap space ...... 106 5.3.5 Number of FlashCopy mappings in a cluster ...... 107 5.3.6 Specifying and viewing bitmap space on an I/O group ...... 108 5.3.7 Bitmap storage ...... 108 5.3.8 FlashCopy characteristics...... 109 5.3.9 FlashCopy maximum configurations...... 109 5.4 FlashCopy events and states ...... 110 5.4.1 FlashCopy mapping states ...... 110 5.4.2 FlashCopy events ...... 111 5.4.3 FlashCopy consistency group states ...... 114 5.5 Dependencies between FlashCopy mappings ...... 114 5.5.1 Linked lists of mappings and resulting dependencies...... 115 5.5.2 Stopping state and cleaning ...... 120 5.5.3 Commands to obtain information about dependencies...... 120

Chapter 6. Implementing FlashCopy...... 121 6.1 Example 1: Basic FlashCopy using the GUI ...... 122 6.1.1 Creating a FlashCopy mapping...... 122 6.1.2 Starting a FlashCopy mapping ...... 129 6.1.3 Modifying a mapping...... 130 6.1.4 Stopping a mapping ...... 131 6.1.5 Creating a consistency group ...... 132 6.1.6 Creating a mapping and adding it to a consistency group ...... 135 6.1.7 Preparing a consistency group ...... 139 6.1.8 Starting a consistency group...... 141 6.1.9 Renaming a consistency group...... 145 6.1.10 Issuing a command to a mapping within a consistency group ...... 146

Contents v 6.2 Example 2: Multiple Target FlashCopy using the GUI ...... 147 6.2.1 Creating mappings and a consistency group ...... 148 6.2.2 Starting a consistency group in an MTFC tree ...... 153 6.3 Example 3: Cascaded and Incremental FC using the GUI ...... 155 6.3.1 Creating consistency groups...... 155 6.3.2 Creating Incremental FlashCopy mappings ...... 156 6.4 Example 4: FlashCopy of Metro Mirror VDisks using the CLI ...... 164 6.4.1 Checking VDisks and Metro Mirror relationship ...... 165 6.4.2 Creating mappings ...... 168 6.4.3 Creating a consistency group ...... 170 6.4.4 Adding mappings to consistency group ...... 171 6.4.5 Preparing a mapping...... 172 6.4.6 Starting a consistency group...... 173 6.4.7 Modifying a mapping...... 176 6.4.8 Stopping a mapping and a consistency group ...... 177 6.4.9 Deleting a mapping and a consistency group ...... 177

Chapter 7. Server integration...... 179 7.1 Copied data...... 180 7.2 Data on copied VDisks ...... 180 7.3 AIX specifics ...... 181 7.3.1 Creating useful copies ...... 181 7.3.2 Accessing useful copies ...... 184 7.4 Windows 2000 and Windows 2003 and copy services ...... 194 7.4.1 Creating useful copies ...... 194 7.4.2 Accessing useful copies ...... 201 7.5 Linux specifics...... 208 7.5.1 Creating useful copies ...... 208 7.5.2 Accessing useful copies ...... 211 7.6 Other operating systems ...... 213

Chapter 8. Automation ...... 215 8.1 Running commands on the cluster ...... 216 8.1.1 Connection ...... 216 8.1.2 Authentication ...... 218 8.1.3 Submission ...... 219 8.1.4 Authorization ...... 219 8.1.5 Execution ...... 219 8.2 Creating connections ...... 219 8.2.1 Transient connections...... 220 8.2.2 Persistent connections ...... 222 8.3 SVC command-line scripting...... 223 8.3.1 Submitting command-line scripts ...... 224 8.3.2 Example SVC command-line script...... 224 8.4 Server-side scripting ...... 225 8.4.1 Creating persistent SVC connections ...... 226 8.4.2 Handling the SVC response ...... 232 8.5 SVC CIMOM programming ...... 233 8.5.1 CIM concepts ...... 234 8.5.2 SVC concepts mapping to CIM concepts ...... 235 8.5.3 Creating and starting a FlashCopy mapping...... 236

vi SVC 4.2.1 Advanced Copy Services 8.6 Logging ...... 242 8.7 Auditing ...... 243

Related publications ...... 245 IBM Redbooks ...... 245 Other resources ...... 245 Referenced Web sites...... 246 How to get Redbooks...... 246 Help from IBM ...... 246

Index ...... 247

Contents vii viii SVC 4.2.1 Advanced Copy Services Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol (® or ™), indicating US registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml

The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:

Redbooks (logo) ® DS8000™ Redbooks® AIX® FlashCopy® System Storage™ DS4000™ IBM® Tivoli®

The following terms are trademarks of other companies:

QLogic, and the QLogic logo are registered trademarks of QLogic Corporation. SANblade is a registered trademark in the United States.

Snapshot, and the Network Appliance logo are trademarks or registered trademarks of Network Appliance, Inc. in the U.S. and other countries.

Java, and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Windows Server, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

x SVC 4.2.1 Advanced Copy Services Preface

This IBM Redbooks publication describes new features that were added with the release of the IBM® System Storage™ SAN Volume Controller V4.2.1 code. Readers are expected to have an advanced knowledge of the SVC and SAN environment, and we recommend these books as background reading: IBM System Storage SAN Volume Controller, SG24-6423 Introduction to Storage Area Networks, SG24-5470 SAN Volume Controller: Best Practices and Performance Guidelines, SG24-7521 Using the SVC for Business Continuity, SG24-7371

The team that wrote this book

This book was produced by a team of specialists from around the world working at the International Technical Support Organization, San Jose Center.

L-R, Dan, Pall, Tom, and Jon

Jon Tate is a Project Manager for IBM System Storage SAN Solutions at the International Technical Support Organization, San Jose Center. Before joining the ITSO in 1999, he worked in the IBM Technical Support Center, providing Level 2 support for IBM storage products. Jon has 22 years of experience in storage software and management, services,

© Copyright IBM Corp. 2008. All rights reserved. xi and support, and is both an IBM Certified IT Specialist and an IBM SAN Certified Specialist. He is also the UK Chair of the Storage Networking Industry Association (SNIA).

Pall Thorir Beck is a Technical Team Lead in IBM ITD Denmark. He has 11 years of experience working with storage and joined the IBM Denmark team in 2005. Prior to working for IBM in Denmark, he worked as an IBM CE performing hardware installations and repairs for IBM iSeries, pSeries, and zSeries in Iceland. His current position involves the administration, design, and support of large SAN environments in Denmark. He is also part of a Nordic storage team that covers storage for open systems in Sweden, Norway, and Finland. Pall has a diploma as an Electronics Technician from Odensen Tekniske Skole in Denmark and IR in Reykjavik, Iceland.

Tom Jahn is a Senior IT Specialist for IBM TSS Storage Systems Sales in Germany. He has 10 years of experience providing technical sales support in IBM. Tom provided technical sales support for networking and server consolidation on OS/390® UNIX for IBM and its customers before he joined the storage brand 8 years ago. He is engaged in providing technical support for open systems storage solutions across multiple platforms and a wide customer base, currently with a focus on storage virtualization. He holds a Dipl. Ing. degree in Computer Science from the Staatliche Studienakademie Sachsen.

Dan Rumney is the IBM Global Support Manager for the JPMC Account. Since joining IBM in 2002, he has worked in development and support of storage products. Dan worked in System Verification Test for IBM before moving to support. He has been providing Level 2 and Level 3 support of SAN Volume Controller for the past 2 years.

We extend our thanks to the many people who contributed to this project. We thank the development and PFE teams in Hursley. In particular, Matt Smith was instrumental in resolving issues and ensuring that they maintained a high profile. We would also like to thank the following people for their contributions:

John Agombar Chris Beeken Iain Bethune Trevor Boardman Carlos Fuente Gary Jarman Colin Jewell Andrew Martin Paul Merrison Steve Randle Bill Scales Matt Smith IBM Hursley

Bill Wiegand IBM Advanced Technical Support

Evelyn Wick IBM Beaverton

Dorothy Faurot IBM Raleigh

Marci Nagel IBM Rochester

xii SVC 4.2.1 Advanced Copy Services Chris Saul IBM San Jose

Sharon Wang IBM Chicago

Tom Cady Deanna Polm Sangam Racherla Bill Trimble IBM ITSO

Tom and Jenny Chang Garden Inn Hotel, Los Gatos, California

Become a published author

Join us for a two- to six-week residency program! Help write a book dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You will have the opportunity to team with IBM technical professionals, Business Partners, and Clients.

Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you will develop a network of contacts in IBM development labs, and increase your productivity and marketability.

Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html

Comments welcome

Your comments are important to us!

We want our books to be as helpful as possible. Send us your comments about this book or other IBM Redbooks® in one of the following ways: Use the online Contact us review Redbooks form found at: ibm.com/redbooks Send your comments in an e-mail to: [email protected] Mail your comments to: IBM Corporation, International Technical Support Organization Dept. HYTD Mail Station P099 2455 South Road Poughkeepsie, NY 12601-5400

Preface xiii xiv SVC 4.2.1 Advanced Copy Services 1

Chapter 1. Introduction to SVC V4.2.1 Advanced Copy Services

In this book, we discuss what is new in V4.2.1 code and how to plan for it. We focus on the new functions and features, and we provide implementation examples. We also describe server integration, and finally, we provide useful scripts to simplify some SVC tasks.

Note: IBM System Storage SAN Volume Controller Software V4.2.1 can be installed only on the IBM 2145-8G4, IBM 2145-8F4, 2145-8F2, and 2145-4F2 SAN Volume Controller Storage engines.

We do not go into any depth about business continuity and disaster recovery theories and recommend these books as resources: IBM System Storage Business Continuity: Part 1 Planning Guide, SG24-6547 IBM System Storage Business Continuity: Part 2 Solutions Guide, SG24-6548 Using the SVC for Business Continuity, SG24-7371

Considerable changes have been made to FlashCopy® functionality in this release. The major changes to FlashCopy are as follows: Incremental FlashCopy Cascaded FlashCopy Dynamically configurable replication services space Addressable storage up to 8 petabytes Grain size Bitmap space Cleaning mode

1.1.1 Incremental FlashCopy

This feature enables you to copy only the portions of the source VDisk that have been updated at either the source or target since the last time that the mapping was started, instead of having to copy the entire contents of that disk. The quantity of data that requires copying partially depends on the grain size; the smaller the grain size, the less data there may be to copy. This leads to shorter copy time and reduces the load on the SVC. To determine the difference between source and target, a “difference” value has been added.

1.1.2 Cascaded FlashCopy

This feature enables you to use a target VDisk of a FlashCopy mapping as the source VDisk of another FlashCopy mapping. The total number of targets from the “original” source can be a maximum of 16.

1.1.3 Dynamically configurable replication services space

This feature enables you to configure up to 1 PB of address space per I/O group. The default bitmap space is 40 MB per I/O group, which is evenly divided by FlashCopy and Metro Mirror and Global Mirror. SVC V4.2.1 introduces a new feature that can “borrow” some read/write cache to serve as copy service bitmap space. The bitmap space per I/O group can be up to 512 MB, which supports up to 1 PB replication service capacity.

We can dynamically configure this space among FlashCopy, Metro Mirror and Global Mirror relationships in customized sizes. This new feature includes the option to allocate the cache memory address space when needed to achieve copy services requirements. The exact amount of copy services space available to users depends upon the copy services relationships and the resulting copy services space allocated to each VDisk.

1.1.4 Grain size

We can now specify a smaller grain size of 64 KB. This smaller grain size is not restricted to incremental mappings. If a mapping is being created that will be added to a tree of cascaded or multiple target mappings, the mapping adopts the grain size used by the other mappings in the tree. The cascading of mappings provides the possibility of joining two separate trees of mappings. If two trees do not use the same grain size, it is not possible to create the mapping to join them.

2 SVC 4.2.1 Advanced Copy Services 1.1.5 Bitmap space

Bitmap space for mappings is allocated on a per I/O group basis. For a single mapping, the default I/O group from which to take bitmap space is that of the source VDisk for the mapping. For a mapping that is being added to an existing tree of mappings, the mapping adopts the I/O group being used by the other mappings in the tree.

The I/O group can be set as a parameter when creating the mapping to allow the user to force the I/O group from which to consume bitmap space. This ability can be useful when the user knows that trees of mappings may be joined in the future, or for reconstructing a tree of mappings in an order different from that of their original creation such as during a configuration restore.

1.1.6 Cleaning mode

Cleaning mode is a method of copying grains from the mapping that is to be stopped when it is in copying state (and when its target VDisk is still accessible to the host). This is achieved by adding a new “cleaning rate” parameter for a mapping that, if combined with a zero background copy rate, causes the mapping to copy grains to a downstream mapping. A new “cleaning progress” field in the query of a mapping informs the user of progress. When cleaning is complete, the mapping is stopped and transitions to the stopped state immediately. If the mapping is stopped before cleaning is complete, the remainder of the cleaning occurs while in the existing Stopping state.

1.2 New in Remote Copy

Metro Mirror now uses the same internode communication technology as Global Mirror. That means only one round-trip delay for remote I/O traffic instead of two. This improves robustness, may increase performance on the exciting applications, and may even increase the distance where Metro Mirror latency impact does not play a significant role.

1.3 Other new functionality in V4.2.1

The following sections provide an overview of other new functions in SVC V4.2.1.

1.3.1 Increased storage addressing

This feature has increased the storage addressing capability from 2 petabytes to 8 petabytes per cluster. This capability is based on an increased extent size (up to 2 GB) and applies regardless of the number of nodes in the cluster. MDisk and VDisk relations and their use are unaffected by this increase.

Chapter 1. Introduction to SVC V4.2.1 Advanced Copy Services 3 1.3.2 Volume management

This feature offers the possibility of assigning or changing the preferred node upon VDisk reallocation to a new I/O group.

1.3.3 Testing managed disks before adding to MDisk groups

This feature offers a built-in test to check MDisks before they are added to an MDisk group.

1.3.4 Cache

In this version of the software, the cache space is partitioned by managed disk group. Each MDisk group that qualifies is limited to a certain amount of the cache, depending on how many MDisk groups are running in the cluster.

4 SVC 4.2.1 Advanced Copy Services 2

Chapter 2. Planning for Advanced Copy Services implementation

Implementing Advanced Copy Services requires considerable planning to be effective. In this chapter, we discuss the issues that you should consider during your planning phase and provide guidance on how to resolve some of the issues that may arise.

Copy services come in two types: Remote Copy, comprising: – Metro Mirror – Global Mirror FlashCopy, comprising: – Single Target FlashCopy (FC) – Multiple Target FlashCopy (MTFC) – Cascaded FlashCopy (CFC) – Incremental FlashCopy (IFC)

In the early phases of planning, it is tempting to select one of these copy services functions as the first step; we strongly recommend that you wait until later in the process, By identifying the business aspects first, you are better positioned to select the most appropriate technical solution.

Prior to implementing copy services, you should ask yourself a number of questions. Why do I need multiple copies of my data? What data do I want to copy? How do I ensure that my copies are usable? Under what circumstances do I need to access this data? How much data can I copy? How are my current systems’ workloads characterized? Will copy services be controlled by server or storage administrators?

2.2 Reasons for copying data

You may want to copy your data for a number of reasons. Several business cases are discussed in the later chapters, but at a high level, these reasons may include: Disaster recovery Data is copied to prevent loss of information due to a disaster and to allow business operations to be continued until the disaster has passed. Testing and assurance Data is copied to provide testing and assurance systems with a copy of real production data to enable meaningful testing. Business operations Data is copied prior to system configuration changes to provide snapshots in case rollback is required.

The reason for copying data dictates which data will be copied. For instance, the data copied for a disaster recovery solution may differ significantly from the data copied for testing, even when the same servers are involved.

6 SVC 4.2.1 Advanced Copy Services 2.3 Deciding what data to copy

Complex systems consume and produce vast amounts of data during day-to-day operations. If the decision was made to mirror every byte of data in an environment, storage utilization would never exceed 50%. It is clear that decisions need to be made to select precisely which data must be copied to balance the cost of copying with its benefits.

Costs are associated with copy services beyond the cost of the technology. These costs include: Capital and maintenance cost of additional storage Management cost of administering copy services Opportunity cost of the bandwidth used to mirror the data

These costs are offset by the benefit of having multiple copies of your data. The benefits of these copies are often calculated by determining the cost of having to recreate this data or the cost of losing it forever. The benefit is that you do not have to meet this cost if you implement the copy services solution. In extreme cases, the data may be critical to the core function of your business, and the cost of losing it is losing your business.

We call groupings of files and other information data sets. A data set can be multiple VDisks that contain the information that you are concerned with. A data set can be smaller than a VDisk - for instance, a single partition on a larger VDisk. However, the smallest entity that SVC can replicate is a VDisk, and we do not discuss file or file system replication in this Redbook.

Some data sets have no business value but are generated during day-to-day operations. Here are some examples of data sets that you may not wish to copy: Temporary file space Temporary database tables Non-business data, such as employees’ personal files

Some data sets are easy to reproduce or, more relevantly, cheaper to reproduce than they are to copy. As such, you may decide that some data should be left uncopied. Examples of such data sets may include: Development sandboxes Test systems SAN boot volumes (reinstallation may be simpler and cheaper than replication)

Even after removing data sets such as those listed, you undoubtedly will be left with data sets that are too expensive to lose or recreate. You will want to copy the VDisks upon which these data sets reside.

2.4 Ensuring usable copied data sets

Copying the contents of a VDisk to another VDisk is all very well, but for the copy to be worth something, you must be able to access and use it at a later date. To design a system that ensures this, you must understand how write caching works and what consistency is.

Chapter 2. Planning for Advanced Copy Services implementation 7 2.4.1 Understanding caching

Caching has been used in computing for over three decades to increase the performance of access requests to storage devices. Many computer systems have multiple layers of cache, and it is important to understand some of these layers when implementing copy services.

While information is cached on both reads and writes, we are concerned only with write caching when we consider copy services

Timeline of a write cache Figure 2-1 shows a basic diagram, outlining the timeline of a write to a storage device when a write cache is in use.

Figure 2-1 Simple write operation with caching

Using a cache speeds up write operations. Without the cache, the host sends a write request directly to the storage device. The storage device responds, and data travels from client to device. When all of the data is in place on the storage device, the device signals that the process is complete. The transfer is limited by the speed at which the disks can provide data. For a storage subsystem. this operation could take tens of milliseconds to complete, in addition to the transit time over the network.

With a cache in place, the storage client need only transfer the data to the cache. The cache works much more quickly than the disks themselves, so this transfer takes less time to complete. This corresponds to the time interval between T0 and T1 in Figure 2-1. At some later time (T2), the cache performs a destaging operation and saves the cached data to the physical disk. This completes at time T3. The host, however, is completely unaware of this operation. For a storage subsystem with cache, the network transit time becomes the limiting factor, and the operation time could be less than a millisecond.

8 SVC 4.2.1 Advanced Copy Services Figure 2-1 on page 8 is a simplification. In a production environment, you have cache above cache above cache. Consider a system with SAN Volume Controller (SVC) installed, as displayed in Figure 2-2. This shows how a production environment can have many caches between the application that users see and the disks in a storage subsystem.

Figure 2-2 Layers of caching in a production environment

Chapter 2. Planning for Advanced Copy Services implementation 9 The numbering in the following list refers to the numbering of the caches in Figure 2-2 on page 9: 1. Database cache: Queries to the database may well be cached by the database process to enable rapid completion of application processes. This may be done in the form of storing queries for later execution. 2. File system cache: I/O commands to the file system may be cached in memory so that processes on the server can complete faster. These commands are passed to underlying storage at a later time or when the cache makes space for subsequent I/O commands. 3. SVC cache: SVC has a redundant write cache that allows I/O commands to the server to complete faster. The data from these commands is destaged at a later time. 4. Storage controller cache: Most modern storage controllers have a write cache of their own. When data is stored in these caches, the system reports a successful conclusion. At a later date, this information will be written to the physical disks themselves.

The diagram in Figure 2-2 on page 9 shows four layers representing a function in the system. As data flows from layer to layer, it passes through a cache. Each cache makes a commitment to the layer above that any data that it accepts eventually will be destaged. However, there is no commitment as to when the data will be destaged. This is important to remember when planning a solution that uses copy services. It should also be understood that each layer has no idea whether data is in the caches owned by higher layers.

Remember: A cache makes a commitment that completed writes will be written to the layer below but with no commitment as to when. A cache has no idea about the existence of caches above it.

Impact of write caches on copy services Copy services are provided by SAN Volume Controller at the storage layer. However, they are used to solve business challenges at the application or hosting layer. As a result, it is important to be aware of the data flow between the layers when planning your copy services solution.

Consider the FlashCopy function, which provides a point-in-time (PIT) copy of your data. When using the FlashCopy function, it is important to ensure that the VDisk that FlashCopy is copying is identical to the volume that your application sees. Because of write-caching, this is not always the case.

10 SVC 4.2.1 Advanced Copy Services In Figure 2-3 you can see where FlashCopy exists with respect to caches. The server sees a SCSI LUN presented by the SVC, corresponding to a VDisk. When an application performs certain actions, data is written to this SCSI LUN.

Figure 2-3 Position of FlashCopy with respect to caches

Because of caching, data written at some time t0 may not actually hit the VDisk until a later time t1. This usually is not a problem because: The writes will eventually get there. All reads go through the same cache, so there is never any noticeable inconsistency between what is on the SCSI LUN that the server sees and what is on the VDisk that SVC presents.

When using copy services, this does become a problem. The business problem that is being solved demands that the data on the LUN be copied. As a result, every block of data that the system has written to its LUNs must be captured by the FlashCopy. However, as previously described, SVC works with VDisks and has no idea that there is data in the caches above it.

Chapter 2. Planning for Advanced Copy Services implementation 11 To ensure that the FlashCopy Target VDisk is an identical match to the SCSI LUN that the server sees, data must be purged from the caches at all layers, and the system must be configured to place no more data in the caches until the FlashCopy has been started. This can be done by Quiescing I/O operations Setting caches to a write-through mode Setting applications to special backup modes that allow for consistent copies to be made, for example, or through other vendor-specific processes.

The same phenomenon exists for Global and Metro Mirror. In these cases, the primary and secondary VDisks remain in sync with one another, but, as with FlashCopy VDisks, there is a time lag between the SVC VDisks and the SCSI LUN that the server sees. Thus, when the relationship is broken or stopped, the data on the VDisk is not identical to the data that an application sees on the LUN.

Dealing with write caches The descriptions in the previous section concern properties of write caches and not limitations of SVC. When data is in the SVC write cache, it is handled appropriately; write commands that have been reported as having completed successfully are completed.

You must be aware of all the write caches on systems that you implement copy services on. This may involve discussions with vendors or system experts to determine what data is cached, when it is cached, and how long it is cached for. In addition, you should determine whether methods for purging the cache, disabling the cache, and checking the cache state are in place. This information is critical when planning your end-to-end solution.

2.4.2 Understanding consistency

Consistency is a critical concept when it comes to copy services. By ensuring consistency in its copy services, SVC ensures systems running on attached hosts will access copied VDisks effectively. It does so by ensuring read stability: If two reads are submitted for the same area, those reads complete successfully, no intervening (in time) or overlapping (in time) write activity occurred, the two reads must return the same data.

This may seem obvious, but it is useful to state it explicitly. In addition, this behavior is also displayed: If a write to a certain area completes successfully, any future reads from that area will return the data that was written (until a further write to the same area is completed successfully).

Again, it may seem obvious, but these two characteristics define how the ordering of reads and writes behave. They enable you to define your expectations as to what should be on a disk at any single point in time.

12 SVC 4.2.1 Advanced Copy Services Take note that writes must complete successfully before you can be certain of what a read operation will return. If you send a read (R1) to an area of disk, then send a write (W1) to the same area of the disk, and then send another read (R2) to the same area of disk, before W1 completes, the only guarantee is that R2 will match R1 or W1.

Finally, we discuss write ordering. Many applications rely on write ordering to survive failures. Write ordering means: If a write (W1) is sent and then reported as complete, and then a second write (W2) is sent and then reported as complete, the cache will submit these writes to the next level in the same order.

However, if W2 is sent before W1 completes, there is no guarantee as to the order in which the cache will submit them to the layer below. In fact, the cache may well batch these requests and submit them at the same time.

Consistency in a single copy relationship Like all SCSI storage devices, an SVC VDisk is made up of blocks. Consider a VDisk at a time t0. A process starts writing data to numerous blocks on the VDisk until time t1, at which point it stops. If you now read from all of the blocks, you have a set of data that represents the VDisk’s state at time t1. At some later point in time t2, if you read from all of the blocks again, you get an identical set of data because of read stability. Therefore, you can say that the data set from t2 is the same data set as that from t1. Because each block from t2 matches each block from t1 we say that it is a consistent copy.

Chapter 2. Planning for Advanced Copy Services implementation 13 To demonstrate this concept further, refer to Figure 2-4, which demonstrates the idea of consistency in the context of copy services. Consider two brand new VDisks: a source and a target (or a primary and a secondary in the case of Metro/Global Mirror).

Figure 2-4 Copy services consistency

At t0, we have two VDisks that are about to be put into a copy relationship. They are small VDisks with only four blocks each. The precise type of copy relationship is not relevant because the concept is the same. At this point in time, the relationship is inconsistent because the target VDisk data set does not match any version of the source VDisk data set. The subsequent points in time are described in the following list: 1. At this point, block A is copied from the source to the target. The relationship is still inconsistent. 2. At this point, block B is copied from the source to the target. The relationship is still inconsistent. 3. At this point, block C is copied from the source to the target. The relationship is still inconsistent.

14 SVC 4.2.1 Advanced Copy Services 4. At this point, block A is overwritten by the server with block E. The relationship is still inconsistent. 5. At this point, block D is copied from the source to the target. The relationship is now consistent because the target VDisk data set represents the same data set as the source VDisk at t3. 6. At this point, block E is copied from the source to the target. The relationship is still consistent and is now synchronized.

The difference in time between t3 and t5 represents the Recovery Point Objective (RPO) for this VDisk.1

Consistency for a group of copy relationships Having described the concept of consistency for a single copy relationship, we now define the concept of consistency for a group of copy relationships.

A single relationship is consistent when the data set of the target VDisk matches the data set of the source VDisk at some point in time in the past. In the same way, a group of relationships is consistent when each of the relationships in the group are themselves consistent, and each relationship represents the same point in time as all of the others.

Once consistency has been attained, it is maintained until specific user actions are taken to disrupt it.

Consistency for FlashCopy Once a FlashCopy mapping is started, the source and target VDisks are, by definition, consistent. The mapping represents a single point in time, and every FlashCopy mapping in a consistency group represents the same point in time.

Bear in mind that the target VDisks are consistent with the source VDisks that SVC is presenting. Because of write caches above the storage layer, it is possible that these VDisks differ from the LUN that the application sees. This is why the caches should be purged prior to starting FlashCopy mappings.

Consistency for Metro and Global Mirror When Metro/Global Mirror relationships are started, they are inconsistent. They must go through a synchronization stage before they become consistent (much like the process outlined in Figure 2-4 on page 14). When this process is complete, the relationships stay consistent throughout normal running.

At this point, we introduce the concept of synchronized relationships. As described previously, a relationship is consistent when the data set on the secondary VDisks represents a data set on the primary VDisks at some point in time.

A relationship is synchronized if it is consistent and the point in time that the secondary VDisks represent is the current point in time. Put another way, the secondary VDisks contain exactly the same data as the primary VDisks. This is represented by t6 in Figure 2-4 on page 14.

1 See Using the SVC for Business Continuity, SG24-7371, to learn the language of business continuity.

Chapter 2. Planning for Advanced Copy Services implementation 15 This definition is stretched a little when it comes to Global Mirror due to the asynchronous nature of the function. The Global Mirror design is such that the lag is kept as low as possible. If the lag grows too long for an extended period of time, SVC stops the relationship to prevent performance on the primary VDisk from getting too poor.

2.5 Accessing the copied data set

The decision as to when to access the data sets that have been copied is very much a business decision. It stems from the rationale behind copying the data in the first place. We do not discuss that aspect here.

The technical practicalities of presenting replicated VDisks from the SVC is straightforward and are discussed in Chapter 6, “Implementing FlashCopy” on page 121 and Chapter 4, “Implementing Remote Copy” on page 47. Accessing these data sets from your servers is more complex and is discussed in Chapter 7, “Server integration” on page 179.

2.6 Determining how much data can be replicated

A major part of copy services planning is determining how much data you actually want to replicate. As discussed in 2.3, “Deciding what data to copy” on page 7, many costs are associated with implementing copy services, and some of these costs may act to limit the amount of data that you wish to replicate.

You should be aware of technical limitations, in addition to the business limitations, when planning your solution. Copy services require the use of SVC resources and so are not limitless. However, Table 2-1 shows that these limitations are enough to allow all but the most expansive copy services solutions. As new versions of SVC have been released, these limitations have become less restraining and we list here the limitations as they have changed over past releases. In each column, we highlight the item that has changed since the previous release

Table 2-1 SVC copy services limitations Objects V3.1.0.xa V4.1.0.xa V4.1.1.x V4.2.0.x V4.2.1.x

FlashCopy mappings per cluster 2048 2048 2048 3855 3855

FlashCopy mappings per 512 512 512 512 512 consistency group

FlashCopy consistency groups 128 128 128 128 128

FlashCopy VDisk per I/O group 16 TB 16 TB 16 TB 40 TB 1024 TBb

FlashCopy targets per source1111616

Metro/Global Mirror relationships 1024 1024 1024 1024 1024 per cluster

Metro/Global Mirror consistency 256 256 256 256 256 groups

16 SVC 4.2.1 Advanced Copy Services Objects V3.1.0.xa V4.1.0.xa V4.1.1.x V4.2.0.x V4.2.1.x

Metro/Global Mirror VDisk per I/O 16 TB 16 TB 16 TB 40 TB 1024 TBb group

Metro/Global Mirror round-trip 68c/20d 68c/20d 80 80 80 latency (ms) a. Global Mirror not supported until SVC V4.1.1.x. b. The default is 40 TB. The bitmap space that is used to control copy services is shared between FlashCopy and Metro/Global Mirror. c. Fibre Channel extenders. d. SAN routers.

Refer to the latest SVC documentation to find the limitations that apply to the level of SVC that you are running.

2.6.1 VDisk capacity limit

All copy services use bitmaps to track what VDisk data must be copied from the source or primary to the target or secondary. The amount of space made available to these bitmaps is variable. This feature enables the user to trade off memory usage between FlashCopy, Metro Mirror, and Global Mirror.

Copy services use “grains” to track what must be copied. Each grain represents 256 KB worth of data that may require copying for Metro/Global Mirror. For FlashCopy, a grain is either 64 KB or 256 KB according to the user’s choice. Each bit in the bitmap represents one grain.

Bitmap space is owned by an I/O group. When a mapping or relationship is created, you can choose which I/O group will provide the bitmap space. This can be a different I/O group from the ones providing the associated VDisks.

The amount of VDisk capacity that can be copied depends on the following: Amount of memory allocated to hold the bitmap (bitmap space) Grain size When FlashCopy is being used, whether the backup is normal or incremental

The maximum size of the bitmap space for one I/O group is 512 MB, with 20 MB being the default. The bitmap space for an I/O group must be shared between FlashCopy and Metro/Global Mirror. If all of the space is used for FlashCopy, for instance, no space would be available for Metro/Global Mirror.

Bitmap space is allocated to FlashCopy mappings and Metro/Global Mirror relationships in increments of 4 KB. This corresponds to 32,768 bits of bitmap space and thus 32,768 grains. Therefore, the smallest unit of VDisk capacity that can be copied is 8 GB (for 256 KB grains) or 2 GB (for 64 KB grains). Because bitmap space is assigned in fixed increments, the amount of bitmap space required to copy a 1 GB VDisk is the same as required for copying an 8 GB VDisk.

Chapter 2. Planning for Advanced Copy Services implementation 17 With Incremental FlashCopy, two bitmaps per FlashCopy mapping are stored, which makes it consume twice the bitmap space of a normal FlashCopy mapping.

Table 2-2 shows the relationship between VDisk capacity and bitmap space depending on the size of the grain and the kind of copy service being used.

Table 2-2 Bitmap space required for copy services Copy Service Grain VDisk capacity provided by bitmap space size (KB) 1 byte 4 KB 1 MB 20 MB 512 MB

Metro/Global Mirror 256 2 MB 8 GB 2 TB 40 TB 1024 TB

FlashCopy 256 2 MB 8 GB 2 TB 40 TB 1024 TB

FlashCopy 64 512 KB 2 GB 512 GB 10 TB 256 TB

Incremental FlashCopy 256 1 MB 4 GB 1 TB 20 TB 512 TB

Incremental FlashCopy 64 256 KB 1 GB 256 GB 5 TB 128 TB

Note: For the FlashCopy mapping bitmap space, the amount of mappings has to be taken into account.

Each mapping target for a FlashCopy uses up bitmap space. This means for Multiple Target FlashCopy, even though only one source VDisk is involved, multiple targets consume the same amount of bitmap space as the same number of targets for independent FlashCopy mappings.

2.7 Understanding your current environment

Prior to implementing copy services, you should have a good understanding of your storage environment. Implementing copy services leads to increased data transfers, so you must understand your current configuration before implementing a solution that affects it.

Copy services operations are driven by two processes: Background copy processes Foreground application processes

Each process generates traffic on your SAN, and so it is important to understand how much traffic your SAN can take, without disruption, and how much traffic your application processes are generating.

18 SVC 4.2.1 Advanced Copy Services 2.7.1 Storage controller information

Controllers are designed to take as much I/O as possible before latency starts to creep up. You cannot indefinitely increase the workload on a controller, as a system with finite resources, without consequences. Figure 2-5 illustrates how latency is affected as you increase the workload on a storage controller. All commands have some latency due to transit time to the storage controller and the service time of the commands. As the number of commands sent to a controller increases each second, wait time increases due to queuing, which is the order of the service time. The dotted line on Figure 2-5 shows the maximum workload that should be driven to this storage controller.

Figure 2-5 Latency of commands to a storage controller as workload increases

If you have a well equipped test lab, you may be in a position to create this kind of graph on the same equipment as that in your production environment. Failing that, your storage controller vendor should be able to advise the maximum workload that your storage controllers are capable of, without significant increases in latency.

Once you have determine this amount, you should investigate your storage controllers to see what kind of workload they are currently handling. The method by which you do this depends on the controller that you are using. If you use IBM TotalStorage Productivity Center for Disk, and it is configured to monitor your storage controllers, you can gather this information that way. Alternatively, if you have TPC installed and it is monitoring your SVC, you can gather performance information about your MDisks and work it out that way. Bear in mind that any performance statistics gathered about your MDisks tell only part of the storage if your storage controller also is presenting volumes directly to other servers.

Chapter 2. Planning for Advanced Copy Services implementation 19 2.7.2 Server information

It is important to know what kind of workload your servers are driving to the system, including information about Data rate (MBps) read and written Workload patterns - for example, sequential or random Response requirements - that is, maximum acceptable latency

It is important to get specific information and not settle for requirements such as “As fast as possible” and “No latency.” There is no way to meet these requirements, and thus they are of no use to anyone. That is not to say that the servers do not have demanding requirements, but you need real figures to effectively plan your copy services system.

When a VDisk is being copied (either by FlashCopy or Metro/Global Mirror), writes to the VDisks drive data copies. In the case of FlashCopy, the first write to a block on the source VDisk results in data being written to the target VDisk; further writes to the same block generate no extra I/O. In the case of Metro/Global Mirror, each write to a block on the primary VDisk will be replicated to the secondary VDisk.

You must ensure that the storage controllers that the target/secondary VDisks are on are up to the workload that will be driven to them. This is particularly important for Metro/Global Mirror relationships. For instance, you may have a set of VDisks that are residing on DS8000™ storage controllers; these VDisks are comfortably handling a high write workload. You decided to replicate these VDisks to a remote site, using Metro Mirror. If the remote VDisks are residing on DS4000™ storage controllers, you can expect to see problems. The remote VDisks must be able to handle the same write workload as the local storage; otherwise, you start to see performance problems.

Under most workload patterns, Metro/Global Mirror does not cause serialization of writes that are submitted at the same time. Consider a server that sends five write operations to five different locations on a VDisk, all at the same time. Obviously, this server is not concerned about the write order because it has not waited for completion from the first before sending the second, third, fourth, or fifth. Thus for Metro Mirror, these five writes are sent to the secondary disk in any order, and completion is reported to the server when the operations are complete. For Global Mirror, these writes are completed once they are in the cache and then written to the remote cluster as soon as possible.

One workload pattern can lead to performance issues with Global Mirror. To maintain write ordering, the Global Mirror code only allows a single write to be active on any given block of a VDisk. Thus, if a write comes in from a host to a block, and that block still has data waiting to be written to the remote cluster, the new write is delayed until the copy has been completed. Applications that deliver this kind of workload do not achieve the performance that Global Mirror is intended to support.

The Per VDisk Cumulative total number of overlapping writes (gwo) statistic provided by the SVC gives a count of colliding writes. If bad performance is being seen, this statistic can be checked to determine if you are hitting this limitation.

2.7.3 Rate of transfer of data

As mentioned at the beginning of this section, Copy services are driven, in part, by foreground application processes. The following operations lead to data being copied: Every write to a grain on the primary VDisk of a Metro/Global Mirror relationship The first write to a grain on the source or target VDisk in a FlashCopy mapping The first read from a grain on the target VDisk in a FlashCopy mapping

20 SVC 4.2.1 Advanced Copy Services The rate at which your applications change their data provides an indication of the rate at which data will be transferred as part of copy services mappings and relationships.

Copy services are also driven by background copy. The rate at which data is transferred is configured on a per-mapping basis for FlashCopy. For Metro/Global Mirror, the rate of data transfer is controlled by the intercluster bandwidth setting.

2.7.4 Intersite latency

Table 2-1 on page 16 shows the latency requirements for an intercluster connection. This latency requirement should be met in all situations. At the very least, the peak workload on the intersite link should not result in the latency requirement being missed.

If the intersite link is made up of multiple links trunked together, the link should be architected so that the failure of a single link does not result in the latency requirement being missed.

2.8 Administration of copy services

To administer copy services, users need access to the SVC cluster. SVC provides multiple user roles to limit the actions that can be taken via the GUI and the command line. By assigning these roles, system administrators can access the SVC cluster in a limited capacity to administer the copy relationships associated with their servers’ storage. These commands are all logged in the audit log, which can be used for problem resolution and to satisfy auditing requirements.

2.8.1 SSH keys

Access to the SVC command line is managed through SSH keys. To provide a user with access to the SVC command line, you should create a public-private key pair. The private key should be given to the user and kept safe. The public key should be uploaded to the SVC cluster and given a descriptive label that uniquely describes that user. This label is used in the audit log (see 2.8.2, “Command auditing” on page 21).

Access to the SVC GUI is managed through ICAT users.

2.8.2 Command auditing

SVC keeps an audit log of all successfully executed commands. By appropriately configuring SSH keys and ICAT users, you can use the audit logs to track which users made what changes to the cluster and when they made the changes.

2.8.3 Authorization roles

The following roles are available: Service Monitor CopyOperator Administrator

These roles are applied to the public keys that have been uploaded to the cluster. When a user connects to the cluster, they do so with a specific private key. This corresponds to a

Chapter 2. Planning for Advanced Copy Services implementation 21 public key on the cluster. Each command that they submit is checked against that public key’s authorization role before being executed and may fail if it does not have the required level of authority.

The commands that each role is authorized to execute are give in Table 2-3.

Table 2-3 Commands available to each role Role Command

Service The following commands: svcinfo: All commnads svcservicetask: All commands

Monitor The following commands: svcinfo: All commands. svctask : finderr, dumperrlog, dumpinternallog svcservicetask : finderr, dumperrlog svcconfig: backup

CopyOperator All commands allowed for Monitor role plus the following svctask commands: prestartfcconsistgrp startfcconsistgrp, stopfcconsistgrp chfcconsistgrp prestartfcmap startfcmap, stopfcmap chfcmap startrcconsistgrp, stoprcconsistgrp switchrcconsistgrp chrcconsistgrp startrcrelationship, stoprcrelationship switchrcrelationship chrcrelationship chpartnership

Administrator All commands can be submitted by users of this role.

Up to 100 different keys can be stored on a cluster, so there is ample scope for assigning individual keys (and roles) to systems or system administrators.

Currently, you cannot limit which copy relationships a user can affect. A CopyOperator can execute commands against any FlashCopy mapping or Metro/Global Mirror relationship. However, the audit log tracks all commands that are executed so rogue operators cannot act with impunity.

Chapter 8, “Automation” on page 215 goes into detail about authorization and auditing.

2.9 Removing workarounds with V4.2.1 new functions

If you are already running SVC copy services, it is quite possible that you implemented processes to work around the limitations of the FlashCopy and Metro/Global Mirror limitations. For instance, you may have a requirement for FlashCopies to be taken of the same VDisk in a short period of time. Before Incremental FlashCopy and Multiple Target FlashCopy, this may have been impossible because the FlashCopy mappings may have required too much time to complete. With the new functions that SVC provides, you have a lot more flexibility as to when you can start new mappings involving VDisks already in a

22 SVC 4.2.1 Advanced Copy Services FlashCopy mapping. Chapter 5, “FlashCopy” on page 95 and Chapter 6, “Implementing FlashCopy” on page 121 explain the new functionality.

The changes to Metro/Global Mirror in V4.2.1 are mainly in the restrictions, which have fewer limitations. With fewer constraints, copy services should become simpler to manage.

Chapter 2. Planning for Advanced Copy Services implementation 23 24 SVC 4.2.1 Advanced Copy Services 3

Chapter 3. Remote Copy: Metro Mirror and Global Mirror

In this chapter, we introduce Remote Copy services. Remote Copy comprises two methods of copying: Metro Mirror and Global Mirror. Metro Mirror is designed for shorter distances and is a synchronous copy; Global Mirror is used for long distances and is asynchronous.

We discuss the functionality of Metro Mirror and Global Mirror, and we describe how you can use Remote Copy services in your daily operations and for business continuity.

Metro Mirror and Global Mirror are IBM branded terms for the functions Synchronous Remote Copy and Asynchronous Remote Copy, respectively. Throughout this book the term “Remote Copy” is used to refer to both functions where the text applies to each equally.

In the topics that follow we discuss some of the many features and functionality of Metro Mirror and Global Mirror.

3.1.1 Metro Mirror

Metro Mirror works by establishing a synchronous copy relationship between two VDisks of equal size. This can be an intracluster relationship that is within the same I/O group of one SVC cluster, or an intercluster relationship, which means a relationship between two SVC clusters that are separated by distance. Those relationships can be standalone or in a consistency group.

Figure 3-1 shows the function of a Metro Mirror relationship.

(1)Write (4) Ack

(2) Mirror write

Remote copy Local copy CACHE CACHE

(3) Ack write

Vdisk Vdisk (Primary) (Secondary)

Metro Mirror Relationship

Figure 3-1 Functionality of Metro Mirror write operation

Metro Mirror functionality ensures that I/Os are committed to both the primary and secondary VDisks before sending confirmation of the completion to the server. This ensures that the secondary VDisk is synchronized with the primary VDisk in case of a failure. The secondary VDisk is in a read-only state, and manual intervention is required to change that access to read/write state. The server administrator also has to mount the secondary disk so the application can start to use that VDisk.

Global Mirror Global Mirror copy relationships work in a similar way as Metro Mirror does but by establishing an asynchronous copy relationship between two VDisks of equal size. This is mostly intended for intercluster relationships over long distances.

26 SVC Advanced Copy Services for Business Continuity With Global Mirror, a confirmation is sent to the server before it has received good completion at the secondary VDisk. When a write is sent to a primary VDisk, it is assigned a sequence number. Mirror writes sent to the secondary VDisk are committed in sequential number order. If a write is issued while another write is outstanding, it may be given the same sequence number.

Figure 3-2 shows the sequence.

(1)Write (2) Ack

(3) Mirror write

Remote copy Local copy CACHE CACHE

(4) Ack write

Vdisk Vdisk (Primary) (Secondary)

Global Mirror Relationship

Figure 3-2 Global mirror write operation

Therefore, if a host has submitted concurrent writes, the SVC batches them together (because the host does not care in what order they are written). The secondary VDisk is therefore consistent with the primary. If a host is writing to the same block before the previous write is committed, the next write is held until the previous write is completed.

3.1.2 Cluster partnership

When creating an SVC cluster copy partnership, we are connecting two SVC clusters that are separated by distance. After the SVC cluster partnership creation has been configured on both clusters, further communication between the nodes in each of the clusters is established and maintained by the SAN network. The intercluster link controls the traffic between the nodes and helps coordinate activity between the clusters (see 3.1.3, “Intercluster communication” on page 28). All intercluster communication goes through the Fibre Channel network.

The SVC does not require a control network or fabric to be installed to manage Remote Copy services. The control link controls the copy states and coordinates updates at either end. The control link is implemented on top of the same FC connection that the SVC uses for Remote Copy I/Os. If communication between the SVC clusters is interrupted or lost, an error is logged, and all copy relationships stop automatically.

Chapter 3. Remote Copy: Metro Mirror and Global Mirror 27 3.1.3 Intercluster communication

Each SVC cluster can be configured with one other SVC cluster as its partner. When both clusters are correctly configured and can communicate with each other over the fabric, they establish further communication facilities between the nodes on each of the clusters.This is called the intercluster link.

The intercluster link keeps information about the total bandwidth available from the local cluster to the remote cluster that is to be assigned to background copy process. This is used to govern the rate at which the background copy is performed.

The intercluster link can be changed during normal operation.The default value for this parameter is 50 MBps.

3.1.4 Maintenance of the intercluster link

All SVC nodes maintain a database of the other devices that are visible on the fabric. The database is updated as devices appear and disappear.

Devices that are identified as SVC nodes are categorized according to the SVC cluster to which they belong. SVC nodes that belong to the same cluster establish communication channels among themselves and begin to exchange messages to implement the clustering and functional protocols of the SVC.

Nodes that are in different clusters do not exchange messages after the initial discovery is complete, unless they have been configured in a Remote Copy relationship.

The intercluster link carries the control traffic to coordinate activity between the two clusters. Through this intercluster link, a message is sent between the nodes that checks whether a connection is established, and it is referred to as a heartbeat between the clusters. It is formed between one node in each cluster, which is termed the focal point. The traffic between the focal point nodes is distributed among the logins that exist between those nodes.

If the focal point node should fail (or all its logins to the remote cluster fail), a new focal point is chosen to run the control traffic.

Note: If we change the focal point node, we cause I/O to pause, but copy relationships do not enter the Consistent Stopped state.

The software that controls the intercluster link monitors for link errors and excludes the link if the error rate gets too high: An intermittent hardware problem on a nonredundant link that leads to a link reset for the SVC cluster. Lost frames or delay exceeding 1.5 seconds affecting the heartbeats between the two clusters. A problem leading to excessive data frames being dropped (even where non-data frames are passed successfully). A series of failures that mean progress is not made - for example, a message is not delivered for more than 15 seconds.

If errors lead to the heartbeat timeout being detected on one of the clusters, that cluster closes all process logins between the focal point nodes.

28 SVC Advanced Copy Services for Business Continuity The SVC cluster monitors for errors on the intercluster link. and if an error reoccurs within 30 seconds, it is ignored. But if more than three 30-second periods experience an error event, the intercluster link is excluded. Then the focal point node deliberately prevents an intercluster link being created between the local cluster and the remote cluster. The other cluster reports Partially Configured in its configuration state for the remote cluster. It is likely the other cluster also has detected the same set of errors and excluded the link on the same basis; in this case, both clusters report Partially Configured. If only one cluster detects an error and excludes the link, the cluster that does not detect the error continues to report Fully Configured.

3.1.5 I/O channel

I/O channel is the term used to describe the connections between clusters that are used to deliver mirror writes between the clusters when using Remote Copy (intercluster).

Intracluster Remote Copy uses connectivity within a node to deliver I/O to secondary virtual disks. I/Os are never routed between nodes in different I/O groups.

The I/O channel uses the same protocols for communication between the nodes in different clusters as that used between the nodes within a local cluster. This requires only one round trip to deliver the write and receive its completion.

The I/O channel traffic is monitored using the same scheme as described in 3.1.4, “Maintenance of the intercluster link” on page 28.

3.1.6 Background copy

The bandwidth that background copy can use is configured on the intercluster link as stated in 3.1.3, “Intercluster communication” on page 28. That bandwidth is divided evenly between the nodes that are performing background copy for active copy relationships. This bandwidth allocation is independent from the number of disks that a node is responsible for.

Each node, in turn, divides its bandwidth evenly between the multiple relationships performing background copy. The default value for this feature is 50 MBps, and this value should be set less or equal to the bandwidth that can be sustained by the intercluster link.

For intracluster relationships, each node is assigned 25 MBps for background copy.

Note: Background copy proceeds from low LBA to high LBA in sequence.

Bandwidth impact on foreground I/O latency The background copy bandwidth determines the rate at which the background copy will be attempted. The background copy bandwidth can affect foreground I/O latency in one of three ways: The following results can occur if the background copy bandwidth is set too high compared to the intercluster link capacity: – Delay in the synchronous secondary writes of foreground I/Os – Increase in foreground I/O latency as perceived by applications

Chapter 3. Remote Copy: Metro Mirror and Global Mirror 29 If the background copy bandwidth is set too high for the storage at the primary site, background copy read I/Os overload the primary storage and delay foreground I/Os. If the background copy bandwidth is set too high for the storage at the secondary site, background copy writes at the secondary overload the secondary storage and again delay the synchronous secondary writes of foreground I/Os.

To set the background copy bandwidth optimally, make sure that you take into consideration all aspects of your environments. The three biggest contributing resources are the primary storage, the intercluster link bandwidth, and the secondary storage. Provision the most restrictive of these three resources between the background copy bandwidth and the peak foreground I/O workload.

3.1.7 Remote Mirror relationship

A Remote Mirror relationship is a relationship between two individual VDisks of the same size, and if both VDisks are in the same cluster, they must both be within the same I/O group. These disks are called a master VDisk and an auxiliary VDisk - for more detail, see 3.1.8, “Copy relationship naming convention” on page 31.

A primary VDisk contains a valid copy of host data and is accessible for host write I/Os. A secondary VDisk, if consistent, contains a valid copy of host data but is not available for host write I/Os.

In an ordinary relationship, (not idling) with one primary VDisk and one secondary VDisk, Remote Copy tries to make the data on the secondary VDisk match the data on the primary VDisk by performing writes to the secondary virtual disk.

The following types of VDisk make a copy relationship: Primary VDisk - enabled for read and write I/Os Secondary VDisk - enabled only for read I/Os

Note: When the copy relationship is in the IDLE state, both primary and secondary are enabled for read and write I/Os.

When a Remote Copy relationship is first created, the master VDisk is always assigned the role of primary, and the auxiliary VDisk is assigned the role of secondary. Hence, the copying direction is from master to auxiliary.

These roles might be reversed, and the copying direction will then be from auxiliary to master. The direction of copying is reflected in the state of the system. A Global Mirror and Metro Mirror copy relationship has the same characteristics and uses the same set of commands to create and control the relationship. The relationship between the VDisks persists until it is deleted.

30 SVC Advanced Copy Services for Business Continuity 3.1.8 Copy relationship naming convention

The SVC introduces the following terms in Remote Copy. An explanation of those commonly encountered terms follows: Master VDisk is created as master, and it is always the master. When creating a master VDisk, it becomes the master/primary VDisk open for write I/Os from the application by default. Auxiliary VDisk is a partner VDisk to the master VDisk. When the copy relationship is created all previous data on that VDisk is destroyed. This disk always is the auxiliary VDisk in this relationship. Primary VDisk is the production VDisk and allows read/writes. Updates to this VDisk are mirrored to the secondary VDisk. This state can be changed when changing copy directions. When changing the copy direction, we enable write I/Os to the secondary VDisk, and the primary is set to read-only. Secondary VDisk in an active relationship is always in read-only mode when it contains a consistent image of the primary VDisk. This role changes when you change the copy direction.

Figure 3-3 shows how the role name follows the copy direction.

Master Auxiliary VDisk VDisk

Role: ´Copy direction Role: Primary Secondary

Role: Role: Secondary ´Copy direction Primary

Figure 3-3 Naming conventions for VDisks in copy relationship.

When you create a copy relationship, you need to state which VDisk is the master VDisk, and the other VDisk in that relationship becomes the auxiliary. This relationship does not change even if you change copy directions.

Note: When creating the SVC cluster partnership, neither cluster is master/primary nor auxiliary/secondary. They are both equal in this partnership, and when you use the mkpartnership command, you are using the only Remote Copy command that must be issued to both clusters, all other Remote Copy commands can be issued from only one of the clusters in the relationship.

Chapter 3. Remote Copy: Metro Mirror and Global Mirror 31 3.1.9 Supported methods for synchronizing This section describes three methods that can be used to establish a relationship.

Full synchronization after Create This is the default method. It provides a full copy of the primary VDisk to the secondary VDisk. It is the simplest in that it requires no administrative activity apart from issuing the necessary commands. However, in some environments, the bandwidth available will make this method unsuitable.

The sequence for a single relationship is as follows: A mkrcrelationship is issued without specifying the -sync flag. A startrcrelationship is issued without the -clean flag.

Synchronized before Create In this method, the administrator must ensure that the primary and secondary virtual disks contain identical data before creating the relationship. The administrator can do so in two ways: Both disks are created with the -fmtdisk feature to make all data zero. A complete tape image (or other method of moving data) is copied from one disk to the other.

In either technique, no write I/O must take place to either primary or secondary before the relationship is established.

Then, the administrator must ensure that: A mkrcrelationship is issued with the -sync flag. A startrcrelationship is issued without the -clean flag.

If these steps are not performed correctly, the relationship is reported as being consistent when it is not. This is likely to make any secondary disk useless. This method has an advantage over the full synchronization in that it does not require all the data to be copied over a constrained link. However, if the data must be copied, the master and auxiliary disks cannot be used until the copy is complete, which might be unacceptable.

Quick synchronization after Create In this method, the administrator must still copy data from primary to secondary, but it can be used without stopping the application at the primary. The administrator must ensure that: A mkrcrelationship is issued with the -sync flag. A stoprcrelationship is issued with the -access flag. A tape image (or other method of transferring data) is used to copy the entire primary disk to the secondary disk.

Once the copy is complete, the administrator must ensure that a startrcrelationship is issued with the -clean flag.

With this technique, only the data that has changed since the relationship was created is copied from the primary and secondary. As described in “Synchronized before Create” on page 32, the copy step must be performed correctly; otherwise, the secondary will be useless, although the copy will report it as being synchronized.

Note: In an SVC cluster relationship, it is a good rule to use the same SVC cluster for your master VDisks, even if the master SVC cluster contains the secondary VDisk.

32 SVC Advanced Copy Services for Business Continuity 3.1.10 Remote Mirror consistency groups

A consistency group consist of one or more copy relationships. By grouping the relationship, you can ensure that these relationship are managed so that data within the group is in a Consistent state. If the consistency group is empty, it acquires the type of the first relationship that is added to it, Metro Mirror or Global Mirror. Therefore, each relationship that you add to an already in-use consistency group must be of the same type.

Consistency groups address the issue where the objective is to preserve data consistency across multiple Remote Mirrored VDisks because the applications have related data that spans multiple VDisks. A requirement for preserving the integrity of data being written is to ensure that “dependent writes” are executed in the application’s intended sequence.

Copy commands can be issued to either Metro Mirror or Global Mirror consistency groups, which affects all Remote Mirror relationships in that consistency group, or to a single standalone Remote Mirror relationship, if it is not part of a consistency group.

In summary: Remote Mirror relationships can be part of a consistency group, or they can be standalone and therefore handled as single instances. A consistency group can contain zero or more relationships. An empty consistency group, with zero relationships in it, has little purpose until it is assigned its first relationship, except that it has a name. All the relationships in a consistency group must have matching master and auxiliary SVC clusters. All relationships must be of the same type.

The rules behind consistency mean that certain configuration commands are prohibited, where this would not be the case if the relationship were not part of a consistency group.

Chapter 3. Remote Copy: Metro Mirror and Global Mirror 33 3.2 State overview

After creation of a Remote Copy relationship, that relationship has a state at any given time. Standalone relationships and consistency groups share a command configuration and state modes. We discuss those states and start with the three most common ones as shown in Figure 3-4: Connected versus Disconnected, Consistent versus Inconsistent, and Consistent versus Synchronized.

inconsistent

Start INCONSISTENT INCONSISTENT STOPPED COPYING

Stop Forced Start out of sync create_unsync

IDLING Forced Start Background copy complete Stop enable access consistent

create_sync Start Start CONSISTENT CONSISTENT in sync STOPPED SYNCHRONIZED

Stop

Stop Legend enable access

REMOTE COPY STATE

Remote Copy Event

Figure 3-4 State overview

3.2.1 Connected or disconnected

This distinction can arise when a Remote Copy relationship is created with two virtual disks in different clusters. It is possible that the communication line between those clusters gets broken, and therefore communication between the two clusters is lost.

When the two clusters can communicate, the clusters and the relationships spanning them are described as connected. When they cannot communicate, the clusters and the relationships spanning them are described as disconnected.

34 SVC Advanced Copy Services for Business Continuity If the cluster cannot communicate, each of the clusters is left with half the relationship and has only a portion of the information that was available to it before. Some limited configuration activity is possible but not all.

When the clusters can communicate again, the relationships become connected again. Remote Copy automatically reconciles the two state fragments, taking into account any configuration or other event that took place while the relationship was disconnected.

As a result, either the relationship can return to the state it was in when it became disconnected, or it can enter a different connected state.

Note: Relationships that are configured between virtual disks in the same cluster are never described as being in a disconnected state since only one cluster is involved.

3.2.2 Consistent or inconsistent

A copy relationship between a primary VDisk and a secondary VDisk can be described as being consistent or inconsistent. Consistency groups containing copy relationships can also be described as being consistent or inconsistent. The consistent/inconsistent attribute describes the relationship of the data that exists on the secondary VDisk to that existing on the primary VDisk. It can be considered a property of the secondary VDisk itself.

A secondary VDisk is described as consistent if it contains data that could have been read by a host system from the primary - when, for example, power fails at some point in time while I/O is in progress and the power is later restored.

This imaginary point in time is defined as the recovery point. The requirements for consistency are expressed with respect to activity at the primary up to the recovery point. The secondary VDisk contains the data from all writes to the primary for which the host had received good completion and that data had not been overwritten by a subsequent write (before the recovery point).

If we look at it from the host site, consistency means that a secondary VDisk contains the same data as the primary VDisk at the recovery point (the time at which the power failure, for example, occurred).

If an application is designed to cope with unexpected power failure, this guarantee of consistency means that the application will be able to use the secondary and begin operation just as though it had been restarted after the hypothetical power failure.

Again, the application is dependent on the key properties of consistency: Write ordering Read stability for correct operation at the secondary

If a relationship, or set of relationships, is inconsistent and an attempt is made to start an application using the data in the secondaries, a number of outcomes are possible: The application might decide that the data is corrupt and crash or exit with an error code. The application might fail to detect the data is corrupt and return erroneous data. The application might work without a problem.

Because of the risk of data corruption and, in particular, undetected data corruption, Metro Mirror strongly enforces the concept of consistency and prohibits access to inconsistent data.

Chapter 3. Remote Copy: Metro Mirror and Global Mirror 35 Consistency as a concept can be applied to a single relationship or a set of relationships in a consistency group. The concept of write ordering is that an application can maintain consistency across a number of disks that are accessed through multiple systems.

When deciding how to use consistency groups, the administrator must consider the scope of the application data, taking into account the interdependent systems that communicate and exchange information.

If two programs or systems communicate and store details as a result of the information exchanged, either of the following actions might occur: All the data accessed by the group of systems must be placed into a single consistency group. The systems must be recovered independently (each within its own consistency group). Then each system must perform recovery with the other applications to become consistent with them.

3.2.3 Consistent or synchronized

A copy that is consistent and up-to-date is described as synchronized. In a synchronized relationship, the primary and secondary VDisks are different only in regions where writes from the host are outstanding.

Consistency does not mean that the data is up-to-date. A copy can be consistent and yet contain data that was frozen at some point in time in the past. Write I/O might have continued to a primary and not have been copied to the secondary. This state arises when it becomes impossible to keep up-to-date and maintain consistency - for example, a loss of communication between clusters when writing to the secondary.

When communication is lost for an extended period of time, Metro Mirror tracks the changes that happen at the primary, but not the order or details (write data) of such changes. When communication is restored, it is impossible to synchronize the secondary without sending write data to the secondary out of order, and therefore losing consistency.

Two policies can be used to cope with this: Take a point-in-time copy of the consistent secondary before allowing the secondary to become inconsistent. In the event of a disaster before consistency is achieved again, the point-in-time copy target provides a consistent, though out-of-date, image. Accept the loss of consistency during resync and the loss of a useful secondary, while making it synchronized.

3.2.4 Copy states

The available states are described in detail in this section, and they are: Consistent Stopped Consistent Synchronized Inconsistent Stopped Inconsistent Copying Idling Idling disconnected Inconsistent disconnected Consistent disconnected Empty (applies only to consistency groups)

36 SVC Advanced Copy Services for Business Continuity Consistent Stopped This is a connected state. In this state, the secondary contains a consistent image, but it might be out-of-date with respect to the primary.

This state can arise when a relationship is in Consistent Synchronized state and suffers an error that forces a consistency freeze. It can also arise when a relationship is created with a sync parameter set.

Normally, following an I/O error, subsequent write activity causes updates to the primary, and the secondary is no longer synchronized. In this case, to reestablish synchronization, consistency must be given up for a period. A start command with the Force option must be used to acknowledge this, and the relationship or consistency group transitions to inconsistent copying. Do this only after all outstanding errors are repaired.

In the unusual case where the primary and secondary are still synchronized (perhaps following a user stop, and no further write I/O was received), a start command takes the relationship to consistent synchronized. The No Force option is required. Also in this unusual case, a switch command is permitted, which moves the relationship or consistency group to Consistent Synchronized and reverses the roles of the primary and secondary (switches copy directions).

If the relationship or consistency group becomes disconnected, the secondary side transitions to consistent disconnected. The primary side transitions to Idling Disconnected.

An informational status log is generated every time a relationship or consistency group enters the Consistent Stopped with a status of Online state.

Note: It is possible to create a form of automation that monitors the status log. The SVC cluster can send an SNMP trap that triggers an automation script, which issues the start command in a case of loss of synchronization when certain log entries appear.

Consistent Synchronized This is a connected state. In this state, the primary VDisk is accessible for read and write I/O, and the secondary VDisk is accessible for read-only I/O. This state is also called “normal copy state” and indicates that your relationship is both consistent and synchronized.

Writes sent to the primary VDisk are sent to both primary and secondary VDisks. Either good completion must be received for both writes, the write must be failed to the host, or a state transition out of Consistent Synchronized must take place before a write is completed to the host.

A stop command takes the relationship to Consistent Stopped state. A stoprcrelationship command with the -access argument takes the relationship to the Idling state.

A switch copy direction command leaves the relationship in the Consistent Synchronized state but reverses the primary and secondary roles and therefore the copy direction.

A startrcrelationship or startrcconsistgrp command is accepted, but has no effect.

If the relationship or consistency group becomes disconnected, the same transitions are made as for Consistent Stopped.

Chapter 3. Remote Copy: Metro Mirror and Global Mirror 37 Inconsistent Stopped This is a connected state. In this state, the primary is accessible for read and write I/O, but the secondary is not accessible for either. The command start copy process must be given to make the secondary consistent.

This state is entered when the relationship or consistency group was Inconsistent Copying and has either suffered a persistent error or received a stop command, which has caused the copy process to stop.

A start command causes the relationship or consistency group to move to the Inconsistent Copying state again. A stop command is accepted but has no effect.

If the relationship or consistency group becomes disconnected, the secondary side transitions to Inconsistent Disconnected. The primary side transitions to Idling Disconnected.

Inconsistent Copying This is a connected state. In this state, the primary is accessible for read and write I/O, but the secondary is not accessible for either read or write I/O.

Inconsistent Copying runs as a background copy process, which copies data from the primary to the secondary VDisk.

This state is entered after a start command is issued to an Inconsistent Stopped relationship or consistency group. It is also entered when a forced start is issued to an Idling or Consistent Stopped relationship or consistency group.

In the absence of errors, an Inconsistent Copying relationship is active, and the copy progress increases until the copy process completes. In some error situations, the copy progress might freeze or even return to a known state.

A persistent error or stop copy process command places the relationship or consistency group into an Inconsistent Stopped state. A start command is accepted but has no effect.

If the background copy process completes on a standalone relationship, or on all relationships for a consistency group, the relationship or consistency group transitions to Consistent Synchronized.

If the relationship or consistency group becomes disconnected, the secondary side transitions to Inconsistent Disconnected. The primary side transitions to Idling Disconnected.

Idling This is a connected state. Both master and auxiliary disks are operating in the “primary” role. Consequently both are accessible for write I/O.

In this state, the relationship or consistency group accepts a start command. Metro Mirror maintains a record of regions on each disk that received write I/O while Idling. This is used to determine what areas must be copied following a start command.

The start command must specify the new copy direction. A start command can cause a loss of consistency if either VDisk in any relationship has received write I/O. This is indicated by the synchronized status. If the start command could lead to a loss of consistency, the Force parameter must be specified.

Following a start command, the relationship or consistency group transitions to Consistent Synchronized if no loss of consistency occurs, or to Inconsistent Copying if a loss of consistency does occur.

38 SVC Advanced Copy Services for Business Continuity Also, while in this state, the relationship or consistency group accepts a Clean option on the start command. If the relationship or consistency group becomes disconnected, both sides change their state to Idling Disconnected.

Idling Disconnected This is a disconnected state. The VDisk or disks in this half of the relationship or consistency group are all in the “primary” role and accept read and write I/O.

The main priority in this state is to recover the link and make the relationship or consistency group connected once more.

No configuration activity is possible (except for deletes or stops) until the relationship becomes connected again. At that point, the relationship transitions to a connected state. The exact connected state that is entered depends on the state of the other half of the relationship or consistency group, which depends on: The state when it became disconnected The write activity since it was disconnected The configuration activity since it was disconnected

If both halves are Idling Disconnected, the relationship becomes Idling when reconnected.

While Idling Disconnected, if a write I/O is received that causes loss of synchronization and the relationship was not already stopped (either through user stop or a persistent error), an error log entry is raised to indicate this. This error log is the same as that raised when the same situation arises when Consistent Synchronized.

Inconsistent Disconnected This is a disconnected state. The VDisks in this half of the relationship or consistency group are all in the secondary role and do not accept read or write I/O.

No configuration activity except for deletes is permitted until the relationship becomes connected again.

When the relationship or consistency group becomes connected again, the relationship automatically becomes Inconsistent Copying unless either: The relationship was Inconsistent Stopped when it became disconnected. The user issued a stop command while disconnected

In either case, the relationship or consistency group becomes inconsistent stopped.

Consistent Disconnected This is a disconnected state. The VDisks in this half of the relationship or consistency group are all in the secondary role and accept read I/O but not write I/O.

This state is entered from Consistent Synchronized or Consistent Stopped when the secondary side of a relationship becomes disconnected.

In this state, the relationship or consistency group displays an attribute of FreezeTime (see 3.4.5, “Freeze/Run on consistency groups” on page 42), which is the point in time when consistency was frozen. When entered from Consistent Stopped, it retains the time it had in that state. When entered from Consistent Synchronized, the FreezeTime shows the last time when the relationship or consistency group was known to be consistent. This corresponds to the time of the last successful heartbeat to the other cluster.

Chapter 3. Remote Copy: Metro Mirror and Global Mirror 39 A stop command with enable write access to secondary transitions the relationship or consistency group to an Idling Disconnected state. This allows write I/O to be performed to the VDisks and is used as part of a disaster recovery scenario.

When the relationship or consistency group becomes connected again, the relationship or consistency group becomes Consistent Synchronized only if it does not lead to a loss of consistency under these criteria: The relationship was Consistent Synchronized when it became disconnected. No writes received successful completion at the primary while disconnected.

Otherwise the relationship become Consistent Stopped. The FreezeTime setting is retained.

Empty This state only applies to consistency groups. It is the state of a consistency group that has no relationships and no other state information to show.

It is entered when a consistency group is first created or all relationships are removed from the consistency group. It is exited when the first relationship is added to the consistency group at which point the state of the relationship becomes the state of the consistency group.

3.3 Metro Mirror and Global Mirror configuration limits

Table 3-1 shows the Metro Mirror and Global Mirror configuration limits. All the limits in this table apply to the aggregate of the whole Remote Copy configuration, regardless of whether an individual relationship, or group, is Metro or Global Mirror.

Table 3-1 Metro Mirror and Global Mirror configuration limits Parameter Value

Number of consistency groups 256 per SVC cluster.

Number of relationships 1024 per SVC cluster.

Total VDisk size per I/O group on 1024 TB is the per I/O group limit on the quantity of primary and both clusters secondary VDisk address spaces that can participate in relationships; the default is 40 TB.

3.4 Remote Copy internals

The following sections describe aspects of the internal functions of Metro Mirror and Global Mirror.

3.4.1 Read stability

As mentioned in Chapter 2, “Planning for Advanced Copy Services implementation” on page 5, read stability is defined as: If two reads are submitted for the same area, those reads complete successfully, and no intervening (in time) or overlapping (in time) write activity occurred, the two reads must return the same data.

40 SVC Advanced Copy Services for Business Continuity A number of applications that perform recoveries following power failure or crashes depend on this property for correct behavior.

Many implementations of mirroring systems fail to maintain read stability. Following a power loss or reset scenario this non-volatile record is used to ensure that both mirrors are made consistent by copying data from one disk to the other.

Reads are allowed only when stability is assured.

Note: Read stability is also known as “mirror write consistency.”

Our definition of read stability makes no guarantee of data returned to reads while writes are outstanding, only when reads are issued after writes have completed. It is possible to conceive of an application that issues writes and, without waiting for write completion, issues reads to the same region. Such an application, on seeing new data being returned for its read, might conclude that the write data is indeed committed. Such an application is not satisfied by this definition of read stability. Remote Copy guarantees read stability only following write completion.

3.4.2 Write ordering

Many applications that use block storage are required to survive failures such as loss of power, or a software crash, and not lose data that existed prior to the failure. Because many applications must perform large numbers of update operations in parallel to that storage, maintaining write ordering is key to ensuring the correct operation of applications following a disruption.

If an application is performing numerous updates, it must be designed with the concept of allowing dependent writes. These are writes where it is important to ensure that an earlier write has completed before a later write is started. Reversing the order of dependent writes can undermine the application algorithms and can lead to problems such as detected, or undetected, data corruption.

3.4.3 Write consistency

Remote Copy maintains write consistency where it ensures that, while primary VDisk and secondary VDisk are synchronized, the VDisks stays in sync even in the case of failure in the primary cluster or other failures that cause the results of writes to be uncertain. After such a failure, it is important to ensure that any data reads at the primary VDisk are also read at the secondary VDisk because subsequent writes are dependent on the results of the read data for correctness.

It has been argued that Remote Copy does not require this support during normal operation because only the primary disk is “active” and the secondary is in a read-only state. Remote Copy requires this support only when an unusual event forces you to use the secondary VDisk, and the content of that disk is examined.

SVC Remote Copy services uses mirrored non-volatile record marks to protect against this, which leads to a small increase in processor work because extra protocol is involved. However, it should have no impact on I/O latency (unless I/O throughput is constrained by SVC) or bandwidth.

Chapter 3. Remote Copy: Metro Mirror and Global Mirror 41 3.4.4 Write completion

To ensure read stability (see 3.4.1, “Read stability” on page 40), Remote Copy must not allow reads to complete successfully to areas that are out of sync except when a write to that region is currently outstanding.

During normal operation, only regions that have outstanding writes are out of sync, and thus reads do not have to be delayed or tracked. When an I/O receives an error, processing of new read I/Os is suspended; this is achieved before the failed write is completed. This situation is resolved before any read is released, either by forcing the relationship Consistent Stopped or by failing new and pending I/O.

3.4.5 Freeze/Run on consistency groups

The algorithm that consistency is based on is called Freeze/Run. This algorithm freezes I/O in the consistency group before allowing the next I/O to commit, and when restarted, it allows I/Os only to the primary VDisk.

A Freeze/Run cycle is required when the following has occurred: A stop command is received from the user. An I/O command to either primary or secondary fails so that a host write is not known to be copied to both disks and the primary disk has not become Path Offline. (In such circumstances, a Freeze is performed on the primary of the affected relationships.) New host write I/O is suspended without issuing writes to either primary or secondary. Host writes where the data has not been written, with successful completion, to both primary and secondary are Stalled - that is, completion to the host is delayed. All ongoing write activity is allowed to complete until it has either become Stalled or completed to the host only if both primary and secondary writes have completed. New read I/O is suspended.

FreezeTime When the freeze has been achieved across all relationships in a consistency group and any system transients (such as Path Offline status associated with connectivity to the controllers) have completed, a decision is made as to how to proceed. An attempt might be made to perform a recovery copy that, if successful, allows I/O to resume without loss of synchronization. Otherwise, assuming the primary is still online, a Run is performed with the relationship Consistent Stopped. As a result, a run I/O is resumed, which means: Host I/Os are allowed to complete. New I/Os are allowed to proceed issuing writes only to the primary. Read I/Os are allowed to proceed.

When a consistency group is in the state of Consistent Stopped, the freeze_time parameter (see Example 3-1) shows the time in YY/MM/DD/HH/MM format, and the time it shows is the last known time when the state between primary and secondary was Consistent Synchronized.

Example 3-1 freeze_time parameter IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp 254 id 254 name ITSO_SERVER_GRP master_cluster_id 000002006B603A38 master_cluster_name ITSOCL2 aux_cluster_id 0000020068403A42

42 SVC Advanced Copy Services for Business Continuity aux_cluster_name ITSOCL3 primary master state consistent_stopped relationship_count 2 freeze_time 2007/11/08/20/14/54 status online sync in_sync copy_type metro RC_rel_id 2 RC_rel_name ITSO_SERVER_01 RC_rel_id 9 RC_rel_name ITSO_SERVER_02

The most useful value of FreezeTime is on the secondary VDisk because the primary sets its value when it first determines that the relationship is consistent.

3.4.6 Management of difference map

The difference map performs change recording. This records which areas of the disk were not copied from primary to secondary or were changed since they were copied, as well as recording which areas have writes in flight when synchronized or copying.

Space management A difference map is maintained on all online nodes that are members of the I/O group of a primary VDisk. Space for a different map is permanently reserved for the secondary VDisk for use when it needs to act as a primary.

The difference map is implemented as a non-volatile bitmap, one bit corresponding to a ‘grain of data and each grain corresponding to 256 KB of aligned disk space. This is the unit of background copy (as well as recovery copy).

The difference map space is allocated in units corresponding to 8 GB of disk space (4 KB * 8 bits * 256 KB). Whole units are allocated to each VDisk, with any excess being lost. The limit is expressed in terms of VDisk space in each I/O group.

Note: When creating a copy relationship with VDisks that are less than 8 GB, you still reserve bitmap space for 8 GB. When making a copy relationship, it is a good practice to scale the size of the VDisks in increments of 8 GB so as not to waste bitmap space.

Each I/O group includes both primary and secondary VDisks.

Difference map state The difference map (grain state) may be in one of three global states: Clean No I/O is in flight, and the primary and secondary are synchronized. Out of sync The primary and secondary are synchronized, but I/O is in flight and must complete before being marked clean. It could also mean that after a disruption or failure, Recovery Copy is required to complete successfully before declaring it clean.

Chapter 3. Remote Copy: Metro Mirror and Global Mirror 43 Dirty The primary and secondary are not synchronized.

Each grain may be in a state no worse than the difference map (that is to say, dirty difference maps can have grains that are clean or dirty, but a clean difference map has all grains clean, and hence no I/O in flight). Out of sync and dirty are recorded the same in the non-volatile bitmap. They are distinguished by the global state.

Messaging When two nodes are present in an I/O group, they exchange messages to ensure that each is kept up-to-date with respect to changes required in the difference map: When a write takes place to a grain that has been synchronized, the map must be marked Out of Sync on both nodes before the I/O is allowed to proceed. The Unlock message is not sent until both primary and secondary I/Os have completed successfully. If either write fails, an UnlockDoneFailed message is sent. When a background copy or recovery copy operation has successfully synchronized a grain, the grain must be marked Clean on both nodes. The node that owns the grain sends a message to the partner node to mark the grain Clean. A CleanDone message is sent in reply. When an I/O touches a grain that is synchronized but it is not going to perform a write to the secondary (for example, it is in Consistent Stopped state), the grain must be marked Dirty on both nodes. A node sends a DirtyGrain message and receives a DirtyDone reply from the partner.

When multiple write I/Os are active in the same synchronized grain, each requires its own Lock/LockDone/Unlock exchange with the partner node. Only when the last I/O completes within a grain and no I/O that was active in that grain suffered a write failure is the grain marked Clean in each node.

Effect of node failure When nodes appear and disappear, the difference maps must be resynchronized so that both nodes start (or restart) I/O operation with identical maps. When a node disappears for a long period of time and is survived by its partner node, the partner node must delete all the failed node’s maps because they are invalid. This invalidation must complete before allowing any I/O to progress on the assumption that it has an assured difference map mark in place.

An I/O proceeds only when all up-to-date difference maps have been marked as required (that is, just the local difference map if only one node is operational, but both difference maps if two nodes are operational). For instance, an I/O that dirties a grain must mark both maps dirty before allowing any write I/O to take place.

It is possible that through a sequence of node appearances and disappearances, an up-to-date copy of the difference map is not accessible. This is possible even though a node in the I/O group has an online path to the VDisk. In this case, Remote Copy holds the path offline and rejects host I/O. Either the missing node must be recovered or the Remote Copy relationship must be deleted.

3.4.7 Non-volatile journal

The non-volatile journal data structure is used to maintain details of all writes that are active on Consistent Synchronized relationships. These details include the VDisk, LBA, and count of sectors, and for Global Mirror, the sequence number that was assigned. These details are used by the Quick Recovery Copy and Reconstruct processes (see 3.4.8, “Quick Recovery Copy and Reconstruct” on page 45).

44 SVC Advanced Copy Services for Business Continuity The contents of the non-volatile journal are maintained on the nodes in the I/O group of the primary VDisk for the relationship and are updated at the same time as the bitmap bit in the difference map. The same messages that maintain the locks are used to drive the maintenance of the non-volatile journal. The non-volatile journal has the same availability as any of the difference maps associated with the virtual disks in the same I/O group. In the event that the nodes of the I/O group are offline, all relationships using the non-volatile journal change from Consistent Synchronized to Consistent Stopped, at which point the non-volatile journal is deallocated and is reestablished when the nodes reappear. No administrative intervention is required.

3.4.8 Quick Recovery Copy and Reconstruct

Quick Recovery Copy and Reconstruct are used to maintain synchronization where some disruption has taken place and it is not certain whether I/Os have completed successfully to both primary and secondary.

Quick Recovery Copy and Reconstruct operate during the freeze state described previously. If successful, then I/O can be resumed without having to change to Consistent Stopped state. These processes are essential to making the Consistent Synchronized state fault tolerant. Note the following: Quick Recovery Copy is used for Metro Mirror. Reconstruct is used for Global Mirror.

Each process uses a record of outstanding I/O; the record of I/Os that must be copied to perform Quick Recovery Copy or Reconstruct are maintained in the non-volatile journal. Each entry details the range of sectors that were subject to a write I/O that was in flight at the point that the disruption occurred. This record is used to drive a process to copy data from the primary to secondary disks. This copy is similar to the process that is used for background copy.

While Reconstruct or Quick Recovery Copy is in progress, I/O is suspended for the relationship. Essentially Recovery Copy forms part of the Freeze described earlier. Recovery Copy completes when either all areas in doubt are copied, primary or secondary are offline, or a persistent error occurs during a read.

Quick Recovery Copy and Reconstruct are performed by the owner node if two nodes within the I/O group have online paths to the VDisk. Or, if there is only one node, Quick Recovery Copy and Reconstruct are performed by the one node with an online path within the I/O group.

Recovery Copy uses the non-volatile difference map to record which grains were subject to an outstanding write I/O. Each grain is copied completely from primary to secondary. This process is performed by the current software only if a recovery is required in the short period after a software upgrade completes, but before the conversion to use the non-volatile journal is complete.

Chapter 3. Remote Copy: Metro Mirror and Global Mirror 45 46 SVC Advanced Copy Services for Business Continuity 4

Chapter 4. Implementing Remote Copy

In this chapter, we show you how to implement Remote Copy using the GUI and the command-line interface. We demonstrate the basics of using Remote Copy, including some of the advanced functionality of Remote Copy services.

In this chapter, we show how we implemented Remote Copy in our environment. We use both the GUI and the command line interface to assist us in creating Metro and Global Mirror relationships and consistency groups. Our environment consists of the equipment shown in Table 4-1

Table 4-1 SVC and subsystem Production site SVC Subsystem

Devices ITSOCL2 DS4500

Disaster site SVC Subsystem

Devices ITSOCL3 DS4700

We use the Metro Mirror intercluster relationship between the clusters as shown in Figure 4-1.

Figure 4-1 Demo setup

Our SVC clusters are ITSOCL2 and ITSOCL3, and we are going to create a copy relationship between those two clusters. Because Metro Mirror and Global mirror use the same set of commands, we show you in detail how we implement Metro Mirror. We provide additional commentary when Global Mirror differs from Metro Mirror.

To start the copy relationship, the two clusters should have an established connection through the SAN network. In this chapter, we assume that this has been done. For information about how to install the SVC, refer to the IBM System Storage SAN Volume Controller, SG24-6423.

48 SVC 4.2.1 Advanced Copy Services 4.2 Implementation using the GUI

When using the GUI to establish a copy relationship between two VDisks, we have to log on to both SVC clusters (as stated, we will be using intercluster copy relationship). The only copy relationship command we submit from both clusters is the create partnership command. After submitting the create partnership command, we can perform all our copy functions from one cluster because this is an active/active partnership and both clusters can control the copy process.

4.2.1 Creating a cluster partnership

We start by logging on to the ITSOCL2 cluster, and click the Create button to create the new partnership, as shown in Figure 4-2.

Figure 4-2 Create a partnership between the clusters

Chapter 4. Implementing Remote Copy 49 After choosing to create our new partnership, we see our cluster candidates, as in Figure 4-3.

Figure 4-3 Select ITSOCL3 as cluster partner

The candidates are the clusters that are already configured on the SAN and are zoned to our production cluster ITSOCL2.

When we create the partnership, we need to configure the link between the clusters. If we are creating a Metro Mirror partnership, we have to think only of the bandwidth for the link. If we are creating a Global Mirror partnership, we have to be aware of the link tolerance between the two SVC clusters. These parameters are as follows: Bandwidth – Specifies the bandwidth that is used by the background copy process between the clusters. It adjusts the bandwidth that is used by Metro Mirror or Global Mirror for the initial background copy process. The bandwidth is set to default to 50 MBps. Set the bandwidth to a value that is less than or equal to the bandwidth that can be sustained by the intercluster link. Link Tolerance (Global mirror only) – Defines how sensitive the cluster is to inter-link overload tolerance. The value specifies the seconds that the cluster will tolerate continued difficulties before the cluster suspends the copy function to prevent host I/O impact. The parameter accepts values from 60 to 86400 seconds in steps of 10 seconds. The default is 300 seconds (5 minutes). You can disable link tolerance by entering a value of zero for this parameter. Inter-Cluster Delay (Global Mirror only) – An optional feature for Global Mirror permits a delay simulation to be applied on writes sent to secondary virtual disks. This allows testing to be performed and can be used to

50 SVC 4.2.1 Advanced Copy Services test an application before full deployment of the feature. A value of 0 disables this feature. Intra-Cluster Delay (Global Mirror only) – An optional feature for Global Mirror permits a delay simulation to be applied on writes sent to secondary virtual disks. This allows testing to be performed and can be used to test an application before full deployment of a feature. A value of 0 disables this feature.

We use the default values in our creation of the partnership. In Figure 4-4, we established a partial partnership between ITSOCL2 and ITSOCL3.

Figure 4-4 Partially configured partnership

Note: When creating a partnership be aware of setting the copy rate settings to a value appropriate to the link and secondary back-end storage.

Chapter 4. Implementing Remote Copy 51 We log on to ITSOCL3 to complete the creation of the partnership (see Figure 4-5).

Figure 4-5 Complete creation of the partnership on remote cluster

After clicking OK, we can see that the partnership is fully configured as shown in Figure 4-6.

Figure 4-6 Fully configured cluster partnership

We close the Metro & Global Mirror Cluster Partnerships window, and now we have a fully configured partnership between the clusters. All copy commands from now on can be entered from either of the two clusters but not from both. We continue using ITSOCL2 as our production SVC cluster.

52 SVC 4.2.1 Advanced Copy Services 4.2.2 Changing a cluster partnership

When changing a cluster partnership, we select our partnership from the main menu as shown in Figure 4-7.

Figure 4-7 Modify cluster partnership

If changes are made on the partnership link, it does not stop active mirroring. In Figure 4-8 we can change the same settings that we inserted when we created the partnership (shown previously in Figure 4-2 on page 49).

Figure 4-8 Parameter can be changed for cluster partnership

Chapter 4. Implementing Remote Copy 53 4.2.3 Deleting a cluster partnership

To delete a partnership, we have to run the delete command on both clusters, similar to what we did when we created the partnership (see Figure 4-9).

Figure 4-9 Delete cluster partnership

Figure 4-10 shows the partnership delete query.

Figure 4-10 Confirm deletion of partnership

After we have clicked the Delete button, the cluster partnership is partially configured as shown in Figure 4-11. We can see the state on the remote cluster and on the cluster where we complete the deletion of the partnership.

Figure 4-11 After deleting the local partner, deletion is partially complete

As we did when we created the partnership, we have to log on to the remote cluster to finish the deletion. As shown in Figure 4-11, we have a partially configured link that we will permanently delete by clicking Delete.

54 SVC 4.2.1 Advanced Copy Services 4.2.4 Creating a copy relationship

We have a VDisk on our production SVC cluster, and we are going to mirror its contents to the remote SVC cluster. After creating the VDisks on both clusters, we create the relationship between those VDisks as shown in Figure 4-12; then we import that relationship into the consistency group.

Figure 4-12 Create copy relationship.

After we have selected Create a Relationship and clicked Go, an overview of the steps ahead is displayed (see Figure 4-13).

Figure 4-13 Overview of selections for creating the relationship

Chapter 4. Implementing Remote Copy 55 We name the relationship, creating a meaningful name, as shown in Figure 4-14.

Figure 4-14 Name the relationship

We selected Metro Mirror as the copy type because we want the copy to be synchronous. We can also select the relationship to be intercluster or intracluster. We select intercluster because we are mirroring from production to recovery clusters that are separated by distance.

Figure 4-15 Search for master VDisk candidates

If we know the name of the VDisk we are going to use as our master VDisk, we can filter for that name. Or to see all VDisk candidates on the master cluster, we can use an asterisk to see all the available VDisk candidates as shown previously in Figure 4-15.

56 SVC 4.2.1 Advanced Copy Services After sorting our available candidates, we choose ITSO_MM_S001 as our Master Vdisk as shown in Figure 4-16.

Figure 4-16 Select master VDisk

At this point, we choose our auxiliary VDisk from the other SVC cluster. The SVC clusters are already in a partnership, and therefore the GUI provides a list of candidates for the auxiliary VDisk, without requiring us to log on to the remote cluster.

The GUI shows only VDisks that are the same size as the master VDisk and are available for a copy relationship (see Figure 4-17).

Figure 4-17 Select VDisk as our auxiliary VDisk

Because we are creating a copy relationship with unused VDisks, we know the status of the disks: Both VDisks exist and have not been mapped to any host.

Chapter 4. Implementing Remote Copy 57 From an operating system point of view, VDisks are not initialized and empty. Therefore we can chose to select them as synchronized. This is the most common way to perform this action when you are creating a new relationship between unused VDisks, as shown in Figure 4-18.

Figure 4-18 Select Synchronized so primary and secondary are in sync when copy process starts

We do not add this relationship to a consistency group at this time; we do that later in this chapter (see 4.2.9, “Creating a consistency group” on page 64). Being able to add a relationship directly to a consistency group is useful when adding disks to a host that already has VDisks in an consistency group.

Our relationship is called ITSO_SERVER_01, and it is now created as shown in Figure 4-19.

Figure 4-19 Verify relationship

58 SVC 4.2.1 Advanced Copy Services 4.2.5 Changing a copy relationship

If we need to make modifications to our relationship, we can do so by first selecting the relationship. From the drop-down menu, select Modify a Relationship, as shown in Figure 4-20.

Figure 4-20 Modify relationship

When we modify a relationship, we can change the name of the relationship and add or remove it from a consistency group, as shown in Figure 4-21.

Figure 4-21 Change name of relationship or add or remove it from consistency group

Chapter 4. Implementing Remote Copy 59 4.2.6 Starting a copy relationship

After the successful creation of the relationship, the relationship is in a Consistent Stopped state, as shown in Figure 4-22.

Figure 4-22 Relationship created and ready to start

In Figure 4-22, even though we have created the relationship and although we have not started any copy process, the relationship is in a Consistent Stopped state. It is consistent because we selected the synchronized parameter when creating the relationship.

The final step toward a fully configured relationship is to start the copy process between the VDisks.

We choose Start Copy Process from the drop-down menu as shown in Figure 4-23.

Figure 4-23 Choose Start Copy Process

60 SVC 4.2.1 Advanced Copy Services In Figure 4-24 we do not have to check Forced start or Mark as clean because we are starting a new relationship, and we have already selected it to be synchronous from the start.

Figure 4-24 Start copy process

We see the final state of our relationship in Figure 4-25. Our relationship is in a Consistent Synchronized state so both primary and secondary VDisks should have the same content.

Figure 4-25 Final state of relationship creation

At this point, we have created a relationship between two VDisks on two SVC clusters that are in an intercluster partnership. Our primary VDisk is the master VDisk and is therefore our production disk.

4.2.7 Stopping a copy relationship

We can stop a copy relationship only when it is a standalone relationship. If the relationship is in a consistency group, we must stop the consistency group and therefore stop all relationships in that group as well.

Chapter 4. Implementing Remote Copy 61 As shown in Figure 4-27, we select our relationship, and from the drop-down menu, we select Stop Copy Process.

Figure 4-26 Stop copy relationship.

In Figure 4-27 we can select whether we want to enable write access to the secondary VDisk. Because we are only stopping the copy process and do not want to enable write access to the secondary, we select OK without setting a check mark to enable write access.

Figure 4-27 Stop copy process

62 SVC 4.2.1 Advanced Copy Services In Figure 4-28 we can see the result of stopping the copy process. Our copy state is now Consistent Stopped.

Figure 4-28 Copy state is now Consistent Stopped

4.2.8 Deleting a copy relationship

When we need to delete a relationship, we select the relationship that we want to delete, and from the drop-down menu, we select Delete a Relationship, as shown in Figure 4-29.

Figure 4-29 Delete relationship

As shown in Figure 4-30, we can choose to remove the relationship from the consistency group before we delete it. This option still deletes the relationship, but it removes the relationship from the consistency group before deleting it.

Figure 4-30 Remove from consistency group

Chapter 4. Implementing Remote Copy 63 4.2.9 Creating a consistency group

When we create our relationship, we have the opportunity to add the relationship to a consistency group at the same time. If we have not already created the consistency group, we need to create one and import the relationship to that consistency group. Many administrators start with creating the consistency group and then add the relationship to it during creation.

In this section, we show you how you create a consistency group and how to perform different tasks with it, such as adding and removing the relationship, and stopping and starting the copy process. 1. As shown in Figure 4-31 we select Manage Copy Services from the main menu and then we select Metro & Global Mirror Consistency Groups.

Figure 4-31 Select Create a Consistency Group

2. After we select Create a Consistency Group, we click Go. 3. As shown in Figure 4-32 we name our consistency group and select Create Inter-cluster Mirror consistency group.

Figure 4-32 Name and configure consistency group

We named our consistency group ITSO_SERVER_GRP, and it chose to make it an intercluster relationship because we know the relationship is between two SVC clusters.

64 SVC 4.2.1 Advanced Copy Services After configuring the consistency group, we have the opportunity to add already configured relationships to it. The relationship and the consistency group must have the same type of configuration, which is intercluster or intracluster (in our case, intercluster). 4. In Figure 4-33, you can see that we already have a relationship that is not in a consistency group. We choose it to add it as a member in our ITSO_SERVER_GRP consistency group.

Figure 4-33 Select relationship to consistency group

If we do not select any relationship for our consistency group, it will be defined as empty. We can then add relationship(s) to it at our convenience. In Figure 4-34 we see a summary of settings for our consistency group.

Figure 4-34 Summary of settings for consistency group

Chapter 4. Implementing Remote Copy 65 After clicking Finish, our consistency group is created, but as we can see in Figure 4-35, the state of the consistency group is Consistent Stopped.

Figure 4-35 Consistency group created in Consistent Stopped state

The state is Consistent Stopped because the relationship we added to the consistency group was in a Consistent Stopped state. Consistency groups enter the state of the first relationship that is added to the group, and after that you cannot add a relationship of a different state than that of the consistency group.

4.2.10 Starting a consistency group

Because the nature of consistency groups is to maintain consistency in all member relationships, our actions therefore affect all relationships in the given group. Figure 4-36 shows where we can select to start the copy process from the drop-down menu.

Figure 4-36 Start copy process

66 SVC 4.2.1 Advanced Copy Services We start by selecting the consistency group that we want to start the copy process on and select the options shown in Figure 4-37.

Figure 4-37 Select options for starting the copy process

As before, this copy relationship and the VDisks in that relationship are in sync. Figure 4-38 shows a successful start of copy process on the ITSO_SERVER_GRP consistency group.

Figure 4-38 Consistency group successfully started

4.2.11 Stopping a consistency group

When we stop a copy process, we are not affecting the primary VDisk in any way. We can enable write access to the secondary VDisk when we select the stop copy process (we discuss this in 4.4.2, “Site failover using the GUI” on page 89).

In this scenario we stop the copy process on all relationships in the consistency group, leaving the secondary in read-only mode.

Chapter 4. Implementing Remote Copy 67 In Figure 4-39 from the Metro & Global Mirror Consistency Groups main menu, we select the consistency group that we would like to stop.

Figure 4-39 Select consistency group to stop copy process

After we select Stop Copy Process, we are presented with the window shown in Figure 4-40.

Figure 4-40 Stop copy process

68 SVC 4.2.1 Advanced Copy Services In this case, we do not enable write access to the secondary; we click Go to stop the copy process (see Figure 4-41).

Figure 4-41 Consistency group is Consistent Stopped

4.2.12 Renaming a consistency group

When we rename a consistency group, it does not affect in any way the operation of the relationships that are within that group. To rename a consistency group, follow the steps provided in this section. 1. As shown in Figure 4-42, we start by selecting the group we want to rename; then from the drop-down menu, we select Rename a Consistency Group.

Figure 4-42 Rename consistency group

Chapter 4. Implementing Remote Copy 69 2. After clicking Go, we enter the new name into the New Name field, as shown in Figure 4-43.

Figure 4-43 Enter new name for consistency group

4.2.13 Reversing copy direction

When the consistency group is in a Consistent Synchronized state, we can switch copy directions. This action results in a full reverse of the copy direction - that is, the primary becomes the secondary and vice versa. 1. First we choose the Metro and Global mirror consistency groups, to determine which consistency groups are available and in what state they are (see Figure 4-44).

Figure 4-44 Metro & Global Mirror Consistency Groups

We verify that the status is Consistent Synchronized and we can also see that the primary is the master VDisk. Thus we know that the copy direction is from master to auxiliary.

70 SVC 4.2.1 Advanced Copy Services 2. We select the consistency group on which we want to change the copy direction, choose Switch Copy Direction (see Figure 4-45), and click Go.

Figure 4-45 Select Switch Copy Direction

The next window (shown in Figure 4-46) shows us that our selected consistency group is the primary VDisk and asks us whether we want to change the copy direction. If we choose the same current copy direction, this command has no effect on the copy state. 3. We want to change the primary VDisk from master to auxiliary and therefore select the primary VDisk to be the auxiliary under new copy direction as shown in Figure 4-46.

Figure 4-46 Change auxiliary to primary VDisk

Chapter 4. Implementing Remote Copy 71 We have now changed our Remote copy direction. The primary is now auxiliary as shown in Figure 4-47.

Figure 4-47 Primary VDisk is now auxiliary

This command does not check whether the secondary VDisk is mapped to a host or the host is actually sending I/Os to the primary VDisk when we make the change. We therefore recommend that the system administrator verify that all I/O is stopped before issuing the switch copy direction command.

4.3 Implementation using the command line

When we implement Remote Copy using the command-line interface, we do not have to use all the command parameters because many of the parameters are optional.

See Chapter 8, “Automation” on page 215 where we discuss some of the commands you can issue when creating a relationship and importing it to a consistency group. More detailed descriptions of each command can be found in IBM Total Storage SAN Volume Controller Command-Line interface User’s Guide, SC26-7903.

As was the case when we implemented GUI, our demo environment consists of two SVC clusters that are separated by distance. We have already installed both SVCs, following the instructions provided in IBM System Storage SAN Volume Controller, SG24-6423.

We are using SVC clusters ITSOCL2 and ITSOCL3. ITSOCL2 is our production cluster, and ITSOCL3 is our disaster or recovery cluster. We start by checking for available candidates.

4.3.1 Listing cluster candidates

We start by checking whether we have a cluster candidate for our partnership as shown in Example 4-1.

Example 4-1 List cluster candidates IBM_2145:ITSOCL2:admin>svcinfo lsclustercandidate id configured cluster_name 0000020068203A42 no ITSOCL3

We can see that this SVC cluster is not configured, meaning our SVC cluster candidate is not already configured as our partner. Only one cluster partnership is available for each cluster.

72 SVC 4.2.1 Advanced Copy Services 4.3.2 Creating cluster partnership

As shown in Example 4-2, we create a partnership by issuing the mkpartnership command.

Example 4-2 mkpartnership command >>- svctask -- -- mkpartnership ------>

>--+------+------> '- -bandwidth -- bandwidth_in_mbps -'

>--+- remote_cluster_id ---+------>< '- remote_cluster_name -'

This command creates a one-way partnership between our production cluster and the disaster/recovery cluster. To complete the two-way partnership, the equivalent mkpartnership command must be issued on the disaster/recovery cluster.

We create the partnership using the default bandwidth - that is, 50 MBps - and our remote cluster is called ITSOCL3. Example 4-3 shows the command used on the production SVC cluster.

Example 4-3 Create partnership with ITSOCL3 IBM_2145:ITSOCL2:admin>svctask mkpartnership ITSOCL3

At this point, we have partially created the partnership by issuing the command lscluster -delim : ITSOCL3 (see Example 4-4).

Example 4-4 Partially configured partnership IBM_2145:ITSOCL2:admin>svcinfo lscluster -delim : ITSOCL3 id:0000020068403A42 name:ITSOCL3 location:remote partnership:partially_configured_local bandwidth:50

We log on to ITSOCL3 (the remote cluster) and complete the creation. We issue the same command as before, svctask mkpartnership ITSOCL2, and then svcinfo lscluster -delim : ITSOCL2 to see the status of the partnership as shown in Example 4-5.

Example 4-5 Create partnership with ITSOCL2 IBM_2145:ITSOCL3:admin>svctask mkpartnership ITSOCL2 IBM_2145:ITSOCL3:admin>svcinfo lscluster -delim : ITSOCL2 id:000002006B603A38 name:ITSOCL2 location:remote partnership:fully_configured bandwidth:50

We have created a fully configured partnership between ITSOCL2 and ITSOCL3 with a bandwidth of 50 MBps.

Chapter 4. Implementing Remote Copy 73 4.3.3 Changing cluster partnership

As shown in Example 4-6 we change the cluster partnership.

Example 4-6 chpartnership command >>- svctask -- -- chpartnership ------>

>-- -bandwidth -- bandwidth_in_mbps ------>

>--+- remote_cluster_id ---+------>< '- remote_cluster_name -'

We can change the parameters of the cluster link without stopping the active copy process. We might want to increase the bandwidth the background copy can use from the default value of 50 Mbps to 75 MBps (see Example 4-7).

Example 4-7 Change bandwidth between clusters from 50 to 75 MBps IBM_2145:ITSOCL2:admin>svctask chpartnership -bandwidth 75 ITSOCL3

IBM_2145:ITSOCL2:admin>svcinfo lscluster ITSOCL3 id 0000020068403A42 name ITSOCL3 location remote partnership fully_configured bandwidth 75

4.3.4 Deleting a cluster partnership

As shown in Example 4-8, we show how to delete the cluster partnership.

Example 4-8 The rmpartnership command >>- svctask -- -- rmpartnership ------>

>--+- remote_cluster_id ---+------>< '- remote_cluster_name -'

When deleting a copy partnership, we have to delete the partnership from both the local and remote clusters. From the cluster where we are logged on, we issue the rmparntership command, using the remote cluster name in the command as shown in Example 4-9.

Example 4-9 Remove partnership IBM_2145:ITSOCL2:admin>svctask rmpartnership ITSOCL3

IBM_2145:ITSOCL3:admin>svctask rmpartnership ITSOCL2

After issuing the rmpartnership command, we log on to the remote cluster to complete deletion of the partnership.

74 SVC 4.2.1 Advanced Copy Services 4.3.5 Creating a copy relationship

After we created the intercluster partnership between our clusters, we have to start a copy relationship between two VDisks. We have already created VDisks on both clusters of the same size, which is required when creating a copy relationship. The disks we have created are as follows: On our production cluster ITSOCL2 – ITSO_MM_S01 – ITSO_MM_S02 On our Disaster/Recovery cluster ITSOCL3 –ITSO_MM_T01 –ITSO_MM_T02

As shown in Example 4-10, we create the relationship.

Example 4-10 mkrcrelationship command >>- svctask -- -- mkrcrelationship ------>

>-- -master --+- master_vdisk_id ---+------> '- master_vdisk_name -'

>-- -aux --+- aux_vdisk_id ---+------> '- aux_vdisk_name -'

>-- -cluster --+- cluster_id ---+------> '- cluster_name -'

>--+------+------> '- -name -- new_name_id -'

>--+------+------> '- -consistgrp --+- consist_group_id ---+-' '- consist_group_name -'

>--+------+-- --+------+------>< '- -sync -' '- -global -'

Note: If a consistency group is already created, we can add a relationship to a consistency group in the same step, using the -consistgrp parameter.

Chapter 4. Implementing Remote Copy 75 When using the mkrelationship command, we have to provide the names or IDs of the VDisks and clusters we use. For more detailed information about particular aspects of the command, refer to the CLI command guide (IBM Total Storage SAN Volume Controller Command Line Interface User’s Guide, SC26-7903-02). Our command looks like that shown in Example 4-11.

Example 4-11 Create relationship IBM_2145:ITSOCL3:admin>svctask mkrcrelationship -master ITSO_MM_S01 -aux ITSO_MM_T01 -cluster ITSOCL3 -name ITSO_SERVER_01 -sync

RC Relationship, id [0], successfully created

At this point, we have created a fully configured relationship between ITSO_MM_S01 and ITSO_MM_T01, but the copy process is not started. The VDisks are in sync and are only waiting for the copy process to be started (see Example 4-12).

Example 4-12 State of relationship IBM_2145:ITSOCL3:admin>svcinfo lsrcrelationship 0 id 0 name ITSO_SERVER_02 master_cluster_id 000002006B603A38 master_cluster_name ITSOCL2 master_vdisk_id 1 master_vdisk_name ITSO_MM_S02 aux_cluster_id 0000020068403A42 aux_cluster_name ITSOCL3 aux_vdisk_id 0 aux_vdisk_name ITSO_MM_T02 primary master consistency_group_id consistency_group_name state consistent_stopped bg_copy_priority 50 progress freeze_time status online sync in_sync copy_type metro

4.3.6 Starting a copy relationship

As shown in Example 4-13, we start the copy relationship.

Example 4-13 startrcrelationship command >>- svctask -- -- startrcrelationship ------>

>--+------+-- --+------+------> '- -primary --+- master -+-' '- -force -' '- aux ----'

>--+------+-- --+- rc_rel_id ---+------>< '- -clean -' '- rc_rel_name -'

76 SVC 4.2.1 Advanced Copy Services To start the copy process for our relationship, containing the VDisks ITSO_MM_S01 and ITSO_MM_T01, we use the startrcrelationship command. If we have already put the relationship into a consistency group, we have to start the consistency group with all the relationships in that group by using the startrcconsistgrp command.

We have created the relationship and now we want to start copying; but we apply the -force parameter because this parameter is required if consistency would be lost by starting a copy operation. Because we do not know whether an I/O has occurred to the primary VDisk after we created the relationship, we need to issue the command with the -force parameter. In general, the -force parameter is required when the relationship is in one of the following states: Consistent Stopped but not synchronized Idling but not synchronized

The -force parameter is not required when the relationship is in one of the following states: Inconsistent Stopped Inconsistent Copying Consistent Synchronized

Example 4-14 shows where our relationship ITSO_SERVER_01 is started.

Example 4-14 Start copy process within relationship IBM_2145:ITSOCL2:admin>svctask startrcrelationship -force ITSO_SERVER_01

After we have started the relationship, it becomes Consistent Synchronized as shown in Example 4-15.

Example 4-15 State of relationship Consistent Synchronized IBM_2145:ITSOCL2:admin>svcinfo lsrcrelationship ITSO_SERVER_01 id 2 name ITSO_SERVER_01 master_cluster_id 000002006B603A38 master_cluster_name ITSOCL2 master_vdisk_id 2 master_vdisk_name ITSO_MM_S001 aux_cluster_id 0000020068403A42 aux_cluster_name ITSOCL3 aux_vdisk_id 4 aux_vdisk_name ITSO_MM_T001 primary master consistency_group_id consistency_group_name state consistent_synchronized bg_copy_priority 50 progress freeze_time status online sync copy_type metro

Chapter 4. Implementing Remote Copy 77 4.3.7 Stopping a copy relationship

In Example 4-16 we show how to stop a copy relationship.

Example 4-16 stoprcrelationship command >>- svctask -- -- stoprcrelationship -- --+------+------> '- -access -' >--+- rc_rel_id ---+------>< '- rc_rel_name -'

The stoprcrelationship command applies only to a standalone relationship. If the relationship is part of a consistency group, the group, and therefore all relationships within that group, has to be stopped. We can issue the stoprcrelationship command to stop a relationship that is copying from primary to secondary VDisks.

If the relationship is in a Consistent state (that is, Consistent Stopped, Consistent Synchronized, or Consistent Disconnected), we use the -access parameter to enable write access to the secondary VDisk.

4.3.8 Changing a copy relationship

In Example 4-17 we show how to change a copy relationship. To change the relationship we use the command svctask chrcrelationship.

Example 4-17 chrcrelationship command >>- svctask -- -- chrcrelationship ------>

>--+- -name -- new_name_arg ------+------> +- -force ------+ '- -consistgrp --+- consist_group_id ---+-' '- consist_group_name -'

>--+- rc_rel_id ---+------>< '- rc_rel_name -'

We can change the name and add or remove relationships from the consistency group using this command. We can remove a relationship from a consistency group by specifying the -force parameter and the name and ID of the relationship.

We start by adding a relationship to an existing consistency group with the name of ITSO_SERVER_GRP, as shown in Example 4-18.

Example 4-18 Add standalone relationship to consistency group IBM_2145:ITSOCL2:admin>svctask chrcrelationship -consistgrp ITSO_SERVER_GRP ITSO_SERVER_01

Then we remove the relationship from the consistency group, as shown in Example 4-19.

Example 4-19 Remove relationship from consistency group IBM_2145:ITSOCL2:admin>svctask chrcrelationship -force ITSO_SERVER_01

78 SVC 4.2.1 Advanced Copy Services This form of the modify relationship command succeeds in the connected or disconnected states. If the clusters are disconnected, the relationship is only removed from the consistency group on the local cluster at the time the command is issued. When the clusters are reconnected, the relationship is automatically removed from the consistency group on the other cluster. Alternatively, you can issue a modify command (chrcrelationship) to remove the relationship from the group on the other cluster while it is still disconnected.

4.3.9 Deleting a copy relationship

In Example 4-20 we show how to delete the copy relationship.

Example 4-20 rmrcrelationship command. >>- svctask -- -- rmrcrelationship -- --+- rc_rel_id ---+------>< '- rc_rel_name -'

When we delete a relationship, we are only deleting the relationship between those two VDisks; the deletion does not affect the VDisks themselves. It is not possible to delete a relationship if it is a member of a consistency group. We have to remove it from the consistency group first and then delete it, as shown in Example 4-21.

Example 4-21 Delete copy relationship IBM_2145:ITSOCL2:admin>svctask chrcrelationship -force ITSO_SERVER_01

IBM_2145:ITSOCL2:admin>svctask rmrcrelationship ITSO_SERVER_01

4.3.10 Creating a consistency group

In Example 4-22 we show how to create a consistency group.

Example 4-22 mkrcconsistgrp command >>- svctask -- -- mkrcconsistgrp -- --+------+---> '- -name -- new_name -'

>-- --+------+------>< '- -cluster --+- cluster_id ---+-' '- cluster_name -'

Note: If we make a Metro Mirror or Global Mirror consistency group without the -cluster parameter, the consistency group is created only on the local cluster.

We want to create a consistency group between ITSOCL2 and ITSOCL3 and associate the copy relationship (that we have already created) to that consistency group. We created the consistency group as shown in Example 4-23.

Example 4-23 Create consistency group IBM_2145:ITSOCL2:admin>svctask mkrcconsistgrp -cluster ITSOCL3 -name ITSO_SERVER_GRP RC Consistency Group, id [255], successfully created

Chapter 4. Implementing Remote Copy 79 This consistency group is now created, but it has no relationship. We can see that by looking closer at the relationship as shown in Example 4-24.

Example 4-24 Show content of the consistency group IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp ITSO_SERVER_GRP id 255 name ITSO_SERVER_GRP master_cluster_id 000002006B603A38 master_cluster_name ITSOCL2 aux_cluster_id 0000020068403A42 aux_cluster_name ITSOCL3 primary state empty relationship_count 0 freeze_time status sync copy_type empty_group

To move a relationship between two consistency groups, you must issue the chrcrelationship command twice. Use the -force parameter to remove the relationship from its current group, and then use the -consistgrp parameter with the name of the new consistency group.

Because our relationship was created as a standalone relationship, it is not a member of a consistency group, so we only need to use the -consistgrp parameter as in Example 4-25.

Example 4-25 Add relationship to consistency group IBM_2145:ITSOCL2:admin>svctask chrcrelationship -consistgrp ITSO_SERVER_GRP ITSO_SERVER_01

The content of our consistency group has change, as shown in Example 4-26.

Example 4-26 Consistency group listing with relationship details IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp ITSO_SERVER_GRP id 255 name ITSO_SERVER_GRP master_cluster_id 000002006B603A38 master_cluster_name ITSOCL2 aux_cluster_id 0000020068403A42 aux_cluster_name ITSOCL3 primary master state consistent_stopped relationship_count 1 freeze_time status online sync in_sync copy_type metro RC_rel_id 0 RC_rel_name ITSO_SERVER_01

80 SVC 4.2.1 Advanced Copy Services Note: If we try to add a relationship to a consistency group that has a different copy state - such as the relationship is stopped and the consistency group is synchronized - we get an error message: The relationship cannot be added because the states of the relationship and the consistency group do not match.

4.3.11 Starting a consistency group

In Example 4-27 we show how to start the consistency group.

Example 4-27 startrcconsistgrp command >>- svctask -- -- startrcconsistgrp ------>

>--+------+-- --+------+------> '- -primary --+- master -+-' '- -force -' '- aux ----' >--+------+-- --+- rc_consist_group_id ---+------>< '- -clean -' '- rc_consist_group_name -'

When we start or stop a consistency group, we affect all relationships in that group; hence the term “consistency.” It is possible to make changes to the consistency group within the start and stop commands.

When issuing the startrcconsisgrp command, we can use the following parameters: -primary With this parameter we state the VDisk that will become the primary VDisk in this relationship. We can define whether it is the master or the auxiliary that we want to be the active disk. -force This allows us to force the copy process to start. If we know the state of the secondary VDisk, we can start the copy process - even if we know that we will lose consistency while the secondary is catching up with the primary. -clean This decides if it is a clean start. Clean in this sense means that any changes that have been made at the secondary are ignored, and only changes made at the primary are considered when synchronizing the primary and secondary VDisks.

In this example, we start the copy process by issuing the startrcconsistgrp command and stating that the primary will be the master VDisk (see Example 4-28).

Example 4-28 Start copy process in consistency group with master as primary disk IBM_2145:ITSOCL2:admin>svctask startrcconsistgrp -primary master ITSO_SERVER_GRP

Chapter 4. Implementing Remote Copy 81 primary master state consistent_synchronized relationship_count 1 freeze_time status online sync copy_type metro RC_rel_id 0 RC_rel_name ITSO_SERVER_01

At this point, we have started the ITSO_SERVER_01 copy relationship in the ITSO_SERVER_GRP consistency group, and the VDisks are in the state Consistent Synchronized.

4.3.12 Stopping a consistency group

In Example 4-29 we show how to stop the relationship.

Example 4-29 stoprcrelationship command. >>- svctask -- -- stoprcrelationship -- --+------+------> '- -access -' >--+- rc_rel_id ---+------>< '- rc_rel_name -'

By issuing the stoprcconsistgrp command with the -access parameter, we stop the copy process and enable write access to the secondary, This command does not change the copy direction, but the state of the consistency group is Idling, which means that both VDisks have read/write access enabled and no copy process is active.

If we issue the command stoprcconsistgrp only, the consistency group stops mirroring between VDisks, but the secondary still is in a read-only state.

Example 4-30 shows how we used the stoprcconsistgrp with the -access command to stop the copy process and enable write access to the secondary VDisk.

Example 4-30 Stop copy process and enable write access IBM_2145:ITSOCL2:admin>svctask stoprcconsistgrp -access ITSO_SERVER_GRP

IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp ITSO_SERVER_GRP id 255 name ITSO_SERVER_GRP master_cluster_id 000002006B603A38 master_cluster_name ITSOCL2 aux_cluster_id 0000020068403A42 aux_cluster_name ITSOCL3 primary state idling relationship_count 1 freeze_time status sync in_sync copy_type metro

82 SVC 4.2.1 Advanced Copy Services RC_rel_id 0 RC_rel_name ITSO_SERVER_01

To change the copy direction, follow the process described in 4.3.15, “Reversing copy direction” on page 85.

4.3.13 Changing a consistency group

Example 4-31 shows how we change the consistency group.

Example 4-31 chrcconsistgrp command >>- svctask -- -- chrcconsistgrp -- -- -name -- new_name_arg --->

>-- --+- rc_consist_group_name -+------>< '- rc_consist_group_id ---'

When changing the consistency group, we are changing only the name. A consistency group is only a name until it has a relationship created.

In Example 4-32 we change the consistency group name from ITSO_SERVER_GRP to ITSO_SILENT.

Example 4-32 Change name of consistency group IBM_2145:ITSOCL2:admin>svctask chrcconsistgrp -name ITSO_SILENT ITSO_SERVER_GRP

At this point, our consistency group has the name ITSO_SILENT.

4.3.14 Deleting a consistency group

In Example 4-33 we show how to delete the consistency group with active relationships in it, and how the relationships are not affected by this deletion.

Example 4-33 rmrcconsistgrp command >>- svctask -- -- rmrcconsistgrp -- --+------+------> '- -force -' >--+- rc_consist_group_id ---+------>< '- rc_consist_group_name -'

When deleting a consistency group that is not empty, we can use the -force parameter to delete the consistency group. Issuing this command and parameter changes all relationships in that consistency group to a standalone relationship. The state of the relationship is not changed by deleting the consistency group.

In the examples that follow, we delete a consistency group with two Consistent Synchronized relationships in it.

Chapter 4. Implementing Remote Copy 83 First we check the status of the consistency group called ITSO_SILENT; in this group we have the two relationships ITSO_SERVER_01 and ITSO_SERVER_02. As shown in Example 4-34, the copy status is Consistent Synchronized.

Example 4-34 Check status of relationships within consistency group IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp ITSO_SILENT id 254 name ITSO_SILENT master_cluster_id 000002006B603A38 master_cluster_name ITSOCL2 aux_cluster_id 0000020068403A42 aux_cluster_name ITSOCL3 primary master state consistent_synchronized relationship_count 2 freeze_time status online sync copy_type metro RC_rel_id 2 RC_rel_name ITSO_SERVER_01 RC_rel_id 9 RC_rel_name ITSO_SERVER_02

At this point, we delete the consistency group as in Example 4-35.

Example 4-35 Delete consistency group. IBM_2145:ITSOCL2:admin>svctask rmrcconsistgrp -force ITSO_SILENT

By checking the status of the relationships, we see that they have not changed state; both are Consistent Synchronized (see Example 4-36 and Example 4-37 on page 85).

Example 4-36 Status of ITSO_SERVER_01 copy relationship IBM_2145:ITSOCL2:admin>svcinfo lsrcrelationship ITSO_SERVER_01 id 2 name ITSO_SERVER_01 master_cluster_id 000002006B603A38 master_cluster_name ITSOCL2 master_vdisk_id 2 master_vdisk_name ITSO_MM_S001 aux_cluster_id 0000020068403A42 aux_cluster_name ITSOCL3 aux_vdisk_id 4 aux_vdisk_name ITSO_MM_T001 primary master consistency_group_id consistency_group_name state consistent_synchronized bg_copy_priority 50 progress freeze_time status online

84 SVC 4.2.1 Advanced Copy Services sync copy_type metro

Example 4-37 Status of ITSO_SERVER_02 copy relationship IBM_2145:ITSOCL2:admin>svcinfo lsrcrelationship ITSO_SERVER_02 id 9 name ITSO_SERVER_02 master_cluster_id 000002006B603A38 master_cluster_name ITSOCL2 master_vdisk_id 9 master_vdisk_name ITSO_GLOBAL aux_cluster_id 0000020068403A42 aux_cluster_name ITSOCL3 aux_vdisk_id 5 aux_vdisk_name ITSO_GLOBAL primary master consistency_group_id consistency_group_name state consistent_synchronized bg_copy_priority 50 progress freeze_time status online sync copy_type metro

4.3.15 Reversing copy direction

When a consistency group is in a Consistent Synchronized state, we can reverse the copy direction for that group by using the switchrcconsistgrp command with the -primary parameter (see Example 4-38).

Example 4-38 State of consistency group before switching copy direction IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp ITSO_SERVER_GRP id 255 name ITSO_SERVER_GRP master_cluster_id 000002006B603A38 master_cluster_name ITSOCL2 aux_cluster_id 0000020068403A42 aux_cluster_name ITSOCL3 primary master state consistent_synchronized relationship_count 1 freeze_time status online sync copy_type metro RC_rel_id 0 RC_rel_name ITSO_SERVER_01

Chapter 4. Implementing Remote Copy 85 And now we issue the switch command (Example 4-39).

Example 4-39 Issue the switch copy direction command IBM_2145:ITSOCL2:admin>svctask switchrcconsistgrp -primary aux ITSO_SERVER_GRP

And then check the status after issuing the command, see Example 4-40.

Example 4-40 State of consistency group after issuing switch command IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp ITSO_SERVER_GRP id 255 name ITSO_SERVER_GRP master_cluster_id 000002006B603A38 master_cluster_name ITSOCL2 aux_cluster_id 0000020068403A42 aux_cluster_name ITSOCL3 primary aux state consistent_synchronized relationship_count 1 freeze_time status online sync copy_type metro RC_rel_id 0 RC_rel_name ITSO_SERVER_01

At this point, we have switched copy direction, and the primary disk is now the auxiliary Vdisk.

4.4 Site failover example

When a disaster occurs, it is often the case that we are not prepared for it. It is good practice to have a detailed plan of what has to be done, how, and by whom. In a storage environment, we may have to make decisions about whether and when to declare a disaster, and when or if we do so, it is important to have a detailed action plan.

A detailed action plan should be written in such a way that a storage administrator can carry out those tasks without having the person who wrote the plan explaining each line.

When controlling the copy functionality with either with Metro Mirror or Global Mirror, you can control it from either cluster in the copy relationship. That means if one SVC cluster is taken down, the other SVC cluster can stop the mirroring process and can enable read/write access to the secondary VDisks.

Note: Using Metro Mirror or Global Mirror, the secondary VDisks are always in read-only mode and should not be mapped to a server until the copy direction is changed or the copy process is stopped and the VDisk is set to read/write mode.

If we need to perform a site failover, the following suggestions will help us minimize the downtime of servers: Make sure that all VDisks are in a copy relationship in consistency groups. Make sure that you have created a script that helps you perform a site failover. Chapter 8, “Automation” on page 215 provides script examples.

86 SVC 4.2.1 Advanced Copy Services All servers on the recovery site should be zoned and created, but not mapped to the target SVC cluster VDisks. If a disaster occurs and you are going to perform a site failover, stop all copy processes if they have not already stopped. This is because the secondary VDisk cannot be set to read/write without stopping the copy process.

4.4.1 Site failover when production site loses backend storage

In this example, two SVC clusters are performing a Metro Mirror copy.

As shown in Figure 4-48 both server A and server B are connected to the SAN and defined on both SVC clusters, but only primary server A has allocated VDisks from SVC cluster 1. SVC cluster 2 is the remote SVC, and therefore we create the disks and Metro Mirror relationship, but we do not allocate the VDisks to server B. The target VDisks are left unallocated.

Figure 4-48 Overview of environment

Chapter 4. Implementing Remote Copy 87 The server loses access to its LUNs, and the SVC cannot see its MDisks from that subsystem, as shown in Figure 4-49.

Figure 4-49 Subsystem on production site fails

If the disks on Server A become unavailable, we can perform a failover to the standby server at the disaster/recovery site and start using the secondary VDisks.

This is the process we followed: 1. Terminate all connections from Server A to SVC cluster 1 VDisks. If we can access SVC cluster 1, it is best to terminate all connections between Server A and the SVC cluster. When the repair is complete, we do not have to worry about conflicts with the “old” disks. 2. Stop all copy processes that involve this server. a. Log on to the remote SVC cluster and stop all copy services. This enables us to grant write access to the secondary VDisk. b. If we lose connection to our MDisks, the copy service automatically stops because the SVC perceives it as a hardware failure. You could also reverse the mirror and by that give read/write access to the target LUNs, but if the source Managed Disk Group (MDG) is offline or has a hardware failure, you cannot change the copy direction. 3. Map VDisks from SVC cluster 2 to Server B. 4. Mount VDisks and restart the application.

Note: Automation using the CLI is discussed in Chapter 8, “Automation” on page 215.

88 SVC 4.2.1 Advanced Copy Services 4.4.2 Site failover using the GUI

Our example scenario is a dual-site environment, production and disaster/recovery; the production site (which we refer to as site A) consists of server, subsystem, and SVC. The disaster/recovery site (site B) has identical hardware. In site B we have standby Server B and a Metro Mirror partner to our production SVC.

In Figure 4-50 we can see a Metro Mirror consistency group called ITSO_SERVER_GRP, which is running on our production SVC. In this consistency group we have two active relationships, and the state is Consistent Synchronized.

Figure 4-50 Consistency group in healthy state

In our scenario, the subsystem on site A went offline so all our MDisks were unavailable at that site. Our copy process then automatically stops as shown in Figure 4-51.

Figure 4-51 Consistency group in status Consistent Stopped

At this point the server cannot see its disks at the production site, and we want to perform a failover to the disaster/recovery site. We log on to the SVC cluster located at the disaster/recovery site and make the secondary VDisks available for read/write access.

Even though the status of the relationship is Consistent Stopped, we need to stop the copy process to enable read/write access to the VDisks in the consistency group. The state of the consistency group prevents us from using the Switch Copy Directions command.

Chapter 4. Implementing Remote Copy 89 In Figure 4-52 we selected the Metro & Global Mirror Consistency Groups from the main menu, and there we can see the current consistency groups this cluster partnership has. From the drop-down menu, we select to stop the copy process on the selected consistency group.

Figure 4-52 Stop copy process

When we select to stop the copy process and click Go, we have the opportunity to change the mode of the secondary VDisks to enable write access, as shown in Figure 4-53.

Figure 4-53 Stop copy process and enable write access to secondary

By placing a check mark in the Enable write access check box, and clicking OK, we enable write access to all VDisks in this consistency group. All relationships in the consistency group will become idle and, therefore, the consistency group itself.

90 SVC 4.2.1 Advanced Copy Services Figure 4-54 shows the status of the consistency group after we have enabled write access.

Figure 4-54 Consistency group in Idling state

At this point, we can map the secondary VDisks to our disaster/recovery server. After mapping the disk, a server administrator needs to check whether the SDD driver can see the disks and then restart the application.

After fixing the problem on the production site, all errors regarding the copy relationship must be cleared before you can start the copy process again. When we have enabled write access to the secondary VDisk, and all errors are fixed, we can copy from the disaster/recovery site to the production site and have the auxiliary as our primary disk.

When the production environment is stabilized, we need a service window for the server to switch the copy direction and run production on the production site.

4.4.3 Site failover using the CLI

When we perform a site failover using the CLI, we can implement those commands in a script (see the examples in Chapter 8, “Automation” on page 215).

In this section, we show the same steps as we did in 4.4.2, “Site failover using the GUI” on page 89; we describe the process using the CLI.

The CLI sometimes provides more detailed information about what is going on in our environment.

Our production cluster is called ITSOCL3, and our consistency group is called senegal.

By checking the status of the consistency group, we can see that the state is Consistent Synchronized and the status is online. We can also see that our primary is our master VDisk (see Example 4-41).

Example 4-41 Status of senegal consistency group IBM_2145:ITSOCL3:admin>svcinfo lsrcconsistgrp 255 id 255 name senegal master_cluster_id 0000020068203A42 master_cluster_name ITSOCL3 aux_cluster_id 000002006B403A38 aux_cluster_name ITSOCL2 primary master

Chapter 4. Implementing Remote Copy 91 state consistent_synchronized relationship_count 2 freeze_time status online sync copy_type metro RC_rel_id 0 RC_rel_name senegal_r01 RC_rel_id 1 RC_rel_name senegal_r02

Everything appears to be normal on the server, but then an unexpected event occurs on our subsystem, and we notice that the state and status have changed. The CLI shows us which VDisk (primary or secondary) has a problem, and as soon as this unexpected event occurs to either disk, the copy process stops.

Example 4-42 Copy stops IBM_2145:ITSOCL3:admin>svcinfo lsrcconsistgrp 255 id 255 name senegal master_cluster_id 0000020068203A42 master_cluster_name ITSOCL3 aux_cluster_id 000002006B403A38 aux_cluster_name ITSOCL2 primary master state consistent_stopped relationship_count 2 freeze_time 2007/10/23/14/03/02 status primary_offline sync in_sync copy_type metro RC_rel_id 0 RC_rel_name senegal_r01 RC_rel_id 1 RC_rel_name senegal_r02

We need to enable write access to the secondary VDisk in this relationship, so we log on to the disaster/recovery cluster = in our case, ITSOCL2. We issue the stoprcconsistgrp with the -access parameter.

Note: Use the stoprcconsistgrp command with the -access parameter to enable write access to the secondary VDisks in the group when the group is in a Consistent state.

When a consistency group is in a Consistent state (for example, in the Consistent Stopped, Consistent Synchronized, or Consistent Disconnected state), you can issue the -access parameter with the stoprcconsistgrp command to enable write access to the secondary virtual disks within that group. See Table 4-2 for examples of when to use the -access parameter.

Table 4-2 Criteria for using -access parameter Initial state Final state Notes

Inconsistent Stopped Inconsistent Stopped -

92 SVC 4.2.1 Advanced Copy Services Initial state Final state Notes

Inconsistent Copying Inconsistent Stopped -

Consistent Stopped Consistent Stopped -access permitted

Consistent Synchronized Consistent Stopped -access permitted

Idling Consistent Stopped -access permitted

Idling Disconnected Unchanged A relationship may move to Stopped state when reconnected.

Inconsistent Disconnected Inconsistent Stopped On the cluster issuing the svctask stoprcconsistgrp command

Inconsistent Disconnected Unchanged On the disconnected cluster

Consistent Disconnected Consistent Stopped On the cluster issuing the svctask stoprcconsistgrp command, -access permitted

Consistent Disconnected Unchanged On the disconnected cluster, -access permitted in our case we issue svctask stoprcconsistgrp -access senegal, and the outcome is shown in Example 4-43. Notice that the consistency group is in Idling state.

Example 4-43 Idling state IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp 255 id 255 name senegal master_cluster_id 0000020068203A42 master_cluster_name ITSOCL3 aux_cluster_id 000002006B403A38 aux_cluster_name ITSOCL2 primary state idling relationship_count 2 freeze_time status sync in_sync copy_type metro RC_rel_id 4 RC_rel_name senegal_r01 RC_rel_id 22 RC_rel_name senegal_r02

At this point, we have enabled write access to the secondary VDisk, so we can map the VDisk to the disaster/recovery server with svctask mkvdiskhostmap. The server administrator can check whether the disks can be seen from the SDD driver.

Chapter 4. Implementing Remote Copy 93 4.5 Copy commands using the CLI

You can issue several commands from the CLI, and we list them here. All CLI commands can be found, along with a detailed descriptions, in the IBM System Storage SAN Volume Controller Command-Line Interface User’s Guide, SC26-7903. chpartnership chrcconsistgrp chrcrelationship mkpartnership mkrcconsistgrp mkrcrelationship rmpartnership rmrcconsistgrp rmrcrelationship startrcconsistgrp startrcrelationship stoprcconsistgrp stoprcrelationship switchrcconsistgrp switchrcrelationship

94 SVC 4.2.1 Advanced Copy Services 5

Chapter 5. FlashCopy

In this chapter we discuss SVC FlashCopy, which is the point-in-time copy capability of the SVC. FlashCopy is used to create instant copies of VDisks. We describe the characteristics of FlashCopy and how it can be implemented to ensure that we meet our business continuity objectives.

FlashCopy creates a rapid, complete, and consistent copy from a source VDisk to a target VDisk. Often this functionality is called Time-Zero copy, point-In-time copy or Snapshot™ copy. Its main characteristic is to present a clone of a VDisk for a specific point in time, very fast - that is, it is being presented as complete before the copy is “really” created.

5.1.1 FlashCopy and its uses

Without a function such as FlashCopy, to achieve a self-consistent copy of data, the I/O of the application that manipulates the data has to be quiesced for the entire time the physical copy process takes place. In such a case, the time that the copy process requires is defined by the amount of data to be copied and the capabilities of the infrastructure to copy the data. Only after the copy process is finished can the application that manipulates the data start to access the VDisk that was involved. Only then is it ensured that the data on the copy target is self-consistent and identical to the data on the source for a given point in time.

With FlashCopy, this process is different. FlashCopy enables the creation of a copy of a source VDisk to a target VDisk in a very short time. Thus, the application has to be prevented from changing the data only for a short period of time.

After the FlashCopy has been started, the target VDisk represents the contents of the source VDisk for the point in time when the FlashCopy was invoked. The target VDisk does not yet contain all the data of the source VDisk physically. It can be seen as a “virtual” copy, created using bitmaps.

After FlashCopy has started but before it has finished physically copying the data to the target, the copy can be accessed in read/write mode. From that point on, data that is changed on the source VDisk (by the applications that manipulates the source VDisk) is written to the target VDisk, thus ensuring that the representation of the data on the target VDisk for that point in time is valid.

It is also possible to copy all the data of the source VDisk to the target VDisk through a background copy process. The target VDisk, although not fully copied yet, represents a clone of the source VDisk as long as the relationship between the source and the target exists. Once all data has been copied to the target VDisk, the relationship between the VDisks can be removed, and both VDisks become normal. The former target VDisk is now a physical clone of the source VDisk for the point in time that the FlashCopy was invoked.

To create consistent copies of data that spans multiple VDisks, consistency groups can be used. Consistency groups are sets of FlashCopy mappings, which get copied at the same point in time, thus creating a consistent snapshot of the data across all VDisks.

FlashCopy is very flexible. It is possible for a VDisk to be a source VDisk in one FlashCopy mapping and for the VDisk to be the target VDisk in another FlashCopy mapping. Also, one VDisk can have the role as source VDisk in multiple FlashCopy mappings with different target VDisks. Another feature of FlashCopy is the capability of incrementally “updating” a fully copied target VDisk with only the changes that have been made to the source VDisk of the same mapping.

96 SVC 4.2.1 Advanced Copy Services 5.1.2 Using FlashCopy

It is possible to create multiple target VDisks from one source VDisk. While it is the same source VDisk and different target VDisks, for every source-target VDisk combination, a dedicated mapping for each combination is required.

FlashCopy has many uses. One obvious use is for backing up a consistent set of data without requiring a long backup window. The application manipulating the data must make sure the data is consistent, and the application must be suspended for a short period of time. Once the copy is started, the backup application can access the target while the applications can resume manipulating the live data. No full volume copy is needed.

One very useful case for FlashCopy in our business continuity strategy is to create a full consistent copy of production data for a given point in time at a remote location. In this case, we combine Remote Copy and FlashCopy, and we take a FlashCopy from the Remote Copy secondary VDisks. We can take a consistent backup of our production data on the second location, or create a clone of the data so it is available if anything should happen to our production data.

FlashCopy can also be used as a “safety net” for operations that make copies of data inconsistent for longer-than-normal periods of time. For example, if Global Mirror was to get out of synchronization, the auxiliary VDisk is still consistent in itself, but the process of resynchronizing renders the auxiliary VDisk inconsistent as long as it is not finished. To obtain a consistent copy of the data of the auxiliary VDisk while it is being synchronized, a FlashCopy of this VDisk can be created. i

Another use for FlashCopy is to create clones of data for application development testing or for application integration testing. FlashCopy is also useful when a set of data has to be used for different purposes - for example, a FlashCopy database can be used for data mining.

5.2 FlashCopy functions and features

Characteristics and features of FlashCopy are important to understand to be able to successfully use it. Some features increase the flexibility of FlashCopy. They are FlashCopy consistency groups, Multiple Target FlashCopy (MTFC), Cascaded FlashCopy (CFC), and Incremental FlashCopy (IFC).

Chapter 5. FlashCopy 97 5.2.1 FlashCopy mappings

A FlashCopy mapping is created when two ordinary VDisks get “mapped” together for the purpose of creating a point-in-time copy, a process that is depicted in Figure 5-1.

Source Vdisk m1

Mapping m1

Target Vdisk m1

time T0

Figure 5-1 Single VDisk mapping

Once associated with the mapping, the VDisks assume new responsibilities: One VDisk becomes a source VDisk, and one VDisk becomes a target VDisk. We provide a FlashCopy mapping name for this VDisks mapping, which can be seen as an alias that we use to manage the source and target VDisk pairs for FlashCopy tasks instead of using the names of the VDisks themselves.

The source VDisk provides the data to be copied to the target VDisk. The target VDisk represents the data of the source VDisk for the point in time when the FlashCopy was started as long as the mapping exists.

Additionally, the entire source VDisk can be copied to the target VDisk using a background copy process. Once this copy is finished, the mapping is no longer needed for the target VDisk to present the data.

For a single VDisk, a limit of 16 mappings can be shared between the VDisk in mappings for MTFC and CFC. A combination of mappings representing MTFC and CFC in a “cascading tree” may not exceed 16.

5.2.2 FlashCopy consistency groups

A consistency group consists of a set of FlashCopy mappings and has a FlashCopy consistency group name. This name can be considered an alias, and FlashCopy actions can be directed to the consistency group. This enables the execution of commands on multiple

98 SVC 4.2.1 Advanced Copy Services mappings and ensures consistency across data that spans multiple source VDisks for all mappings in the consistency group. A consistency group consisting of two mappings is illustrated in Figure 5-2.

Source Source Vdisk m1 Vdisk m2

Mapping m1 Mapping m2

Consistency Group cg1

Target Target Vdisk m1 Vdisk m2

Start Consistency Group cg1

time T0

Figure 5-2 FlashCopy consistency group

Data that must be consistent can span multiple VDisks. For instance, when database management system (DBMS) logs reside on a different VDisk than the database itself, we have the problem of “dependent writes. Dependent writes occur, for instance, when database logs are written before (to indicate a database update is about to take place) and after (to log a successful write operation to the database) the database itself gets updated.

If we were to start a FlashCopy of the involved VDisks without ensuring consistency across VDisks, the FlashCopy target could contain both written logs but not the database update. The logs in the FlashCopy target indicate that the write I/O has occurred, but the actual write I/O is missing from the target VDisk that holds the database. This renders the FlashCopy target data inconsistent from a DBMS point of view.

Note: Mappings that are not placed in a consistency group are reported to be members of the pseudo consistency group 0.

Chapter 5. FlashCopy 99 5.2.3 Copy background copy

The FlashCopy background feature enables you to copy all data in a source VDisk to the corresponding target VDisk. Without FlashCopy background, only data that changed on the source VDisk can be copied to the target VDisk. The benefit of using FlashCopy mapping with background copy enabled is that the target VDisk becomes a real clone (independent from the source VDisk) of the FlashCopy mapping source VDisk.

The background copy rate is a property of a FlashCopy mapping that is expressed as a value between 0% and 100%. It can be changed in any FlashCopy mapping state and can be different in the mappings of one consistency group. A value of 0 disables background copy.

Table 5-1 shows the relationship of the background copy rate value to the attempted number of grains to be split per second. Grains are discussed in 5.3.2, “Grains” on page 105.

Table 5-1 Relationship of background copy rate to copy speed Background copy Data copied/sec 256 KB grains 64 KB grains rate value in % split/sec split/sec

1-10 128 KB 0.5 2

11-20 256 KB 1 4

21-30 512 KB 2 8

31-40 1 MB 4 16

41-50 2 MB 8 32

51-60 4 MB 16 64

61-70 8 MB 32 128

71-80 16 MB 64 256

81-90 32 MB 128 512

91-100 64 MB 256 1024

The SVC will attempt to achieve these copy rates. Be aware that the background copy rate and the foreground copies share the same infrastructure. If the required bandwidth (needed to copy data to the underlying MDisks provided by the storage subsystems) is not available, a latency increase and a reduction in throughput on both the foreground copies and the background copy can occur. The degradation is graceful however, and the background copy is performed by both nodes of the I/O group providing the source VDisk.

100 SVC 4.2.1 Advanced Copy Services 5.2.4 FlashCopy autodelete option

A new FlashCopy option enables the automatic deletion of a FlashCopy mapping once the background copy has finished copying all data to the target VDisk. This feature is especially useful in scripts.

5.2.5 Multiple Target FlashCopy

In Multiple Target FlashCopy (MTFC) a VDisk serves as the source VDisk in multiple mappings, while the target is a different VDisk, as shown in Figure 5-3.

Source Vdisk m1 - m16

Mapping 16 Mapping 1 Mapping 2

Target Target Target Vdisk Vdisk ... Vdisk m1 m2 m16

Order of FlashCopy Start (time)

Figure 5-3 Multiple Target FlashCopy (MTFC)

For each source VDisk up to 16 different mappings are possible, and these mappings are independently controllable from each other. For instance, the background copy priority can be different for these mappings. The implementation of MTFC introduces some dependencies, which are discussed in 5.5, “Dependencies between FlashCopy mappings” on page 114.

MTFC mappings can be members of the same or different consistency groups. In a case where all the mappings are in the same consistency group, the result of starting the consistency group will be multiple identical target VDisks.

Chapter 5. FlashCopy 101 5.2.6 Cascaded FlashCopy

A new feature of FlashCopy is Cascaded FlashCopy (CFC). With CFC a VDisk is a source for one FlashCopy mapping and the target for another FlashCopy mapping, as shown in Figure 5-4. These FlashCopy mappings are generally independent of each other; however, dependencies of how to perform actions on them are in place (we discuss this in 5.5, “Dependencies between FlashCopy mappings” on page 114), and for each cascade, 16 mappings are possible.

Mapping m1 Mapping m2 Mappings m3-16

Source Vdisk Vdisk ... Vdisk Vdisk m1 t m1 t m2 t m3-16 s m2 s m3-16

Order of FlashCopy Start (time)

Figure 5-4 Cascaded FlashCopy (CFC)

The use of consistency groups is restricted when using CFC. A consistency group serves the purpose of invoking FlashCopy mappings at the same point in time. Within the same consistency group, it is not possible to have mappings where: The source VDisk of one mapping is the target of another mapping. The target VDisk of one mapping is the source VDisk for another mapping.

These combinations are not useful because, within a consistency group, mappings cannot be established in a certain order, which renders the content of the target VDisks undefined. For instance, it is not possible to determine whether the first mapping was established before the target VDisk of the first mapping that acts as a source VDisk for the second mapping.

Even if it was possible to ensure an order in which the mappings are established within a consistency group, the result would be equal to MTFC (that is, two VDisks holding the same target data for one source VDisk). In other words, a cascade is useful when we want to copy VDisks in a certain order (and copying the changed content targets of FlashCopies) rather than at the same time in an undefined order (from within one single consistency group).

102 SVC 4.2.1 Advanced Copy Services 5.2.7 Combined Multiple Target FlashCopy and Cascaded FlashCopy

MTFC and CFC can be combined in any way, as long as combinations of both do not exceed 16 mappings in one tree. Combining MTFC and CFC is depicted in Figure 5-5.

Source Vdisk m1 – m2

Mapping m1 Mapping m2 Mapping m3

Target Target Target Vdisk m1 Vdisk m2 / Vdisk m3 Source Vdisk m3

Order of FlashCopy Start (time)

Figure 5-5 A tree of combined MTFC and CFC

As we can see in Figure 5-5, what we call a “target VDisk” or a “source VDisk” is, in reality, a VDisk that can be the source, the target, or a source and a target - if it is a member of one or more FlashCopy mappings.

Chapter 5. FlashCopy 103 5.2.8 Incremental FlashCopy

Without Incremental FlashCopy (IFC), once all data in a FlashCopy mapping has been copied to a target VDisk, and the FlashCopy mapping is restarted, all data has to be copied again to create an independent copy. By creating a FlashCopy mapping with the “incremental” flag, only the data that has been changed since the last FlashCopy was started is written to the target VDisk, as shown in Figure 5-6.

Data change Source Source Vdisk m1 Vdisk m1

Start Mapping m1: copy all data Restart Mapping m1: copy changed data only

Target Target Vdisk m1 Vdisk m1

time T0 (1) T0 (2)

Figure 5-6 Incremental FlashCopy (IFC)

The amount of data to be copied depends on the grain size as well. If the grain size is 64 KB (as compared to 256 KB), there might be less data to copy to get a fully independent copy of the source again.

A “difference” value is provided in the query of a mapping, which makes it possible to know how much data has changed. This data must be copied when the IFC is restarted. The difference value is the percentage (0-100%) of grains that have been changed and must be copied to the target VDisk to get a fully independent copy of the source VDisk.

This functionality is needed in cases where we want, for example, a full copy of a VDisk for disaster tolerance, application testing, or data mining. It greatly reduces the time required to establish a full copy of the source data as a new snapshot when the first background copy is completed. In cases where customers maintain fully independent copies of data as part of their disaster tolerance strategy, using IFC can be help them meet their business continuity goals.

104 SVC 4.2.1 Advanced Copy Services 5.3 FlashCopy internals

Understanding how FlashCopy works internally helps us configure it the way we want and enables us to get the most benefit from it.

5.3.1 Indirection layer

For FlashCopy to function, the I/O path of the SVC is altered with the introduction of a facility called the indirection layer. The indirection layer is an additional layer of functionality in the I/O path that causes the I/O to make a detour to enable the FlashCopy function.

The indirection layer becomes active in the I/O path at the start of a FlashCopy mapping and intercepts I/Os directed to both the source VDisk and the target VDisk. It decides for each I/O what actions can be performed. It either: Allows the I/O to go directly to the VDisk Redirects the I/O from the target VDisk to another VDisk Stalls the I/O until it arranges the copying of data from the source VDisk to the target VDisk

The indirection layer is placed below the cache, and this means any latency introduced by its function does not affect interaction with the layer above it (the storage client and the SVC cache). If latency occurs because of stalled I/Os, it affects only the I/O generated by destaging the cache. The decision on how to handle I/Os is based upon three attributes: Destination of the I/O (VDisk and its Logical Block Address - (LBA) Direction of the I/O (that is, either it is a write or a read operation) State of the FlashCopy bitmap

Read operations Read operations targeted to the source VDisk are always passed directly to the source VDisk. Read operations from the target VDisk use the information stored in the FlashCopy bitmap. In a case where the data to be read has not yet been copied to the target VDisk, it is read from the source VDisk, or from another target VDisk in the case of MTFC. In a case where it has been copied, the data is read from the target VDisk.

Write operations Write operations for which the grain containing the data to be written to has been already split go to the VDisk. Write operations to source VDisks and target VDisks targeting an area that has not yet been copied result in stalling the write operation. A copy operation is then started to copy the data from the source VDisk to the target VDisk, and after this operation completes, the stalled write operation resumes.

5.3.2 Grains

Data is copied between VDisks in a FlashCopy mapping in units of the FlashCopy mapping address space called a grain (the granularity of the data to be copied from the source VDisk to the target VDisk is one grain). The grain size can be 256 KB (default) or 64 KB.

The grain size can affect the amount of data copied for Incremental FlashCopy. With IFC, the smaller grain size of 64 KB can lead to less data being copied. If a new FlashCopy mapping is added to a tree, it uses the grain size that is already in use for the mappings in the tree. Joining two existing trees of FlashCopy mappings is also possible. To create the mapping to join the trees, both trees must use the same grain size.

Chapter 5. FlashCopy 105 5.3.3 Bitmaps

Bitmaps are an internal data structure used to track which grains in FlashCopy mappings have been copied from the source VDisk to the target VDisk (that is, which grains have been “split“). One bit in each bitmap represents the state of one grain, and either the bitmap is split or it is not.

A FlashCopy bitmap uses up bitmap space in the memory of the SVC. The maximum size of the bitmap space is 512 MB per I/O group, which has to be shared between FlashCopy bitmaps and Remote Copy bitmaps. We assign the bitmap space to the copy functions using a variable.

The default is 20 MB for FlashCopy and 20 MB for Remote Copy (Metro/Global Mirror). The maximum shared bitmap space is 512 MB per I/O group (that is, 256 MB for FlashCopy and 0 MB for Remote Copy or any similar combination).

This feature enables us to trade off memory between the cache, FlashCopy, and Remote Copy (Metro Mirror/Global Mirror).

5.3.4 Usable bitmap space

Table 5-2 shows the relationship of bitmap space to FlashCopy address space, depending on the size of the grain and the kind of copy service being used.

Table 5-2 Relationship of bitmap space to FlashCopy address space for the specified I/O group Copy Grain 1 byte of 4 KB of 1MB of 20 MB of 512 MB of Service size bitmap bitmap bitmap bitmap bitmap in KB space space space space space gives a gives a gives a gives a gives a total of total of total of total of total of

Metro Mirror 256 2 MB of 8GB of 2TB of 40 TB of 1024 TB of and Global VDisk VDisk VDisk VDisk VDisk Mirror capacity capacity capacity capacity capacity

FlashCopy 256 2 MB of 8GB of 2TB of 40 TB of 1024 TB of target VDisk target VDisk target VDisk target VDisk target VDisk capacity capacity capacity capacity capacity

FlashCopy 64 512 KB of 2GB of 512 GB of 10 TB of 256 TB of target VDisk target VDisk target VDisk target VDisk target VDisk capacity capacity capacity capacity capacity

Incremental 256 1 MB of 4GB of 1TB of 20 TB of 512 TB of FlashCopy target VDisk target VDisk target VDisk target VDisk target VDisk capacity capacity capacity capacity capacity

Incremental 64 256 KB of 1GB of 256 GB of 5TB of 128 TB of FlashCopy target VDisk target VDisk target VDisk target VDisk target VDisk capacity capacity capacity capacity capacity

106 SVC 4.2.1 Advanced Copy Services Note: For the FlashCopy mapping address space, the amount of mappings has to be taken into account.

Each mapping target for a FlashCopy uses up bitmap space. For MTFC, this means that even if only one source VDisk is involved, multiple targets consume the same amount of bitmap space as the number of targets of independent FlashCopy mappings use.

5.3.5 Number of FlashCopy mappings in a cluster

The new maximum number of FlashCopy mappings is 3855 for the entire cluster. Each single mapping (which might be part of a MTFC or CFC) contributes to the maximum number of mappings.

Before MTFC and CFC, when a VDisk could take part in only a single one-on-one relationship, the maximum number of mappings was the maximum number of VDisks per cluster (4096) divided by two (2048).

The limit of 4096 VDisks per cluster still exists, and it is still responsible for the maximum number of FlashCopy mappings. This time though, the maximum number depends on how we use FlashCopy, and to what extent we use MTFC and CFC. With the introduction of MTFC, the maximum limit for mappings increased even though the maximum number of VDisks per cluster stayed the same. This is because with MTFC, one VDisk (which obviously takes only one spot in the maximum VDisk limit of 4096) can be the source VDisk in up to 16 mappings. With the introduction of CFC, a VDisk can be a member of two mappings, and CFC also allows for an increase in the number of mappings possible in a cluster.

Although it is beneficial to have the increased flexibility of MTFC and CFC and, possibly, the increased number of mappings, using this flexibility comes at an expense. The more VDisks we use as target VDisks for MTFC, or the more we use for CFC, then the number of VDisks in a cluster that can be used as source VDisks for MTFC, (or as a cascade as a “root VDisk“), declines accordingly.

For instance, in the extremely unlikely case (this is a worst-case scenario) that we try to copy every VDisk as a source VDisk in as many mappings as possible (mapped with 16 different target VDisks), we can have only 240 VDisks as source VDisks mapped to 16 target VDisks and 1 remaining VDisk mapped to 15 target VDisks.

The calculation here is as follows: 17 VDisks (1 source VDisk and 16 target VDisks) multiplied by 240 equals 4080 VDisks. The maximum number of VDisks supported by an SVC cluster is 4096. That leaves 16 VDisks to be used in mappings, and we could have 1 VDisk as the sourceVDisk and 15 acting as target VDisks.

So, as we can see, the practical number of VDisks to act as source (root) VDisks in FlashCopy mappings is between 241 (assuming maximum usage of MTFC) and 2048 (assuming no usage of MTFC and CFC).

Note: These numbers apply to a cluster consisting of four I/O groups supporting 4096 VDisks.

A cluster consisting of two nodes in one I/O group supports 1024 VDisks. This cluster could support between 60 source (root) VDisks, with each of these in 16 mappings and 1 source VDisk in 3 mappings (worst case) and 512 source VDisks (assuming no use of MTFC and CFC).

Chapter 5. FlashCopy 107 5.3.6 Specifying and viewing bitmap space on an I/O group

The user can increase and decrease the amount of memory used on nodes for keeping FlashCopy and Remote Copy bitmaps at runtime.

For example, for FlashCopy we specify the amount of bitmap space of 20 MB for io_grp0, shown in Example 5-1, and list it.

Example 5-1 Specifying and viewing the bitmap space located on an I/O group admin>svctask chiogrp -feature flash -size 20 io_grp0 admin> admin>svcinfo lsiogrp 0 id 0 name io_grp1_r node_count 2 vdisk_count 382 host_count 4 flash_copy_total_memory 20.0MB flash_copy_free_memory 20.0MB remote_copy_total_memory 20.0MB remote_copy_free_memory 19.9MB

5.3.7 Bitmap storage

The bitmap for FlashCopy mappings is stored in a particular I/O group. When we create a new FlashCopy mapping, we can force which I/O group we want the bitmap to reside in. Using the CLI this can be achieved with the -iogrp parameter. This feature can be useful if changes are planned - for example, creating, joining, or reorganizing trees.

This ability can be useful if the user knows that trees of mappings may be joined in the future, or for reconstructing a tree of mappings in a different order to their original creation order during a configuration restore. If we specify the I/O group of the “target vdisk” when creating the mapping, the bitmap space goes toward the I/O group of the target VDisk. If we do not specify the I/O group for the bitmap space, the bitmap is placed on the I/O group that owns the source VDisk. If we add a mapping to a Cascaded FlashCopy or a Multiple Target FlashCopy or a “cascaded tree” (a combination of both), discussed in 5.3.8, “FlashCopy characteristics” on page 109, the bitmap is stored on the I/O group where the bitmap for the other mappings is stored already.

Note: This feature can lead to FlashCopy mapping behavior that should be taken into consideration.

If, for example, we assume the bitmap of a tree is stored in I/O group 0, and some VDisks (that are members of mappings in that tree) are owned by I/O group 1, when I/O group 0 (where the bitmap for the tree is stored) is taken offline, the FlashCopy mappings for the VDisks owned by I/O group 1 (which is still online, and whose VDisks are still online) stops.

108 SVC 4.2.1 Advanced Copy Services 5.3.8 FlashCopy characteristics

Following is a list of FlashCopy characteristics: FlashCopy is a copy process involving two VDisks associated with a FlashCopy mapping. Once used in a FlashCopy mapping, a VDisk becomes a source or target VDisk. A FlashCopy copies an entire source VDisk to a target VDisk within one cluster. The VDisks chosen as the source and target VDisks can be provided by different I/O groups. A VDisk can be the source VDisk in multiple FlashCopy mappings. A VDisk can be the target VDisk in one FlashCopy mapping. A VDisk can be the source in one mapping and the target in another at the same time. The VDisks used for one FlashCopy mapping must be of the same size. The size of VDisks that are members in a FlashCopy mapping cannot be changed. The content of the source VDisk does not get changed by the FlashCopy process. The original content of the target VDisk gets destroyed by the FlashCopy process. After the FlashCopy is established, the target VDisk represents a clone of the source VDisk. After the FlashCopy is established, the target VDisk can be accessed as read/write. Until all the data is copied, the target VDisk presents the data as long as the mapping exists. Once all the data is copied, the mapping can be deleted, and the VDisks are independent again. Once all the data is copied, the former target VDisk holds the data even without the mapping.

5.3.9 FlashCopy maximum configurations

To plan for and implement FlashCopy, some limits have to be checked and adhered to. The limits for an SVC cluster are shown in Table 5-3.

Table 5-3 Configuration limits for FlashCopy Property Maximum Notes

FlashCopy targets per source 16 The maximum number of FC mappings that can exist with the same source VDisk.

FlashCopy mappings per 3855 The maximum number of FC cluster mappings is calculated by noticing that 240 source VDisks with 16 FC mappings plus one more with 15 mappings uses up all 4096 VDisks per cluster.

FlashCopy consistency groups 128 An arbitrary limit policed by the per cluster software.

FlashCopy VDisk space per I/O 1024 TB This is a limit on the quantity of group FC mappings using bitmap space from this I/O group. This maximum configuration consumes all 512 MB of bitmap space for the I/O group and allows no Metro and Global Mirror bitmap space. The default is 40 TB.

Chapter 5. FlashCopy 109 Property Maximum Notes

FlashCopy mappings per 512 Due to the time taken to consistency group prepare a consistency group with a large number of mappings.

5.4 FlashCopy events and states

The state of a FlashCopy mapping is managed on a per FlashCopy mapping basis (commands that prepare and start a FlashCopy mapping are targeted to a consistency group).

5.4.1 FlashCopy mapping states

A FlashCopy mapping can have states, and it is put in these states by FlashCopy mapping events.

Copying The FlashCopy has been started. From now on, the target VDisk presents the content of the source VDisk for the point in time when the FlashCopy was started (but is not holding all the data yet).

Idle/Copied VDisks are members of a mapping but behave independently. The target represents a full copy of the source for the point in time when the FlashCopy was started. On both VDisks, write caching and read caching is enabled.

Stopping The mapping is in the process of transferring data to a mapping that depends on it.

Stopped The data on the target VDisk is lost and the target VDisk is offline. To regain access to the target VDisk, either the mapping must be deleted or it must be started again.

Suspended The FlashCopy mapping is in the Copying or the Stopping state and the target has been “flashed” from the source. The access to the metadata has been lost, resulting in both the source VDisk and the target VDisk going offline.

Preparing The FlashCopy mapping gets prepared for a FlashCopy start while I/O still continues to the source VDisk. This includes flushing the modified write data of the source VDisk from the cache, placing the cache for the source VDisk into write through mode, and discarding any data associated with the target VDisk from the cache.

Prepared The FlashCopy mapping is ready to be started.

110 SVC 4.2.1 Advanced Copy Services Table 5-4 summarizes the FlashCopy mapping states and the corresponding cache states for the VDisks in a FlashCopy mapping.

Table 5-4 FlashCopy mapping states Source VDisk Target VDisk

State Mapping state Cache state Mapping state Cache state

Idling/Copied online write-back online write-back

Copying online write-back online write-back

Stopped online write-back offline n/a

Stopping online write-back online if copy n/a complete; offline if not

Suspended offline write-back offline n/a

Preparing online write-through online and n/a inaccessible

Prepared online write-through online, but n/a inaccessible

5.4.2 FlashCopy events

In Figure 5-7 on page 113, we show the events that alter the state a FlashCopy mapping. We describe them in this section.

Create Places the FlashCopy mapping into the Idle/Copied state. It creates a new FlashCopy mapping between two VDisks, assigning roles to them, one now being a source VDisk and one being the target VDisk for the new mapping.

Prepare Places the FlashCopy mapping in the Preparing state. The prepare command is directed to a consistency group (single mappings are members of consistency group 0 - in this case, the command is targeted at the mapping name for the given FlashCopy mapping).

Note: Even though the FlashCopy mapping has not been started yet, the target VDisk has to be considered “corrupt” from this point on, because from the beginning of the prepare, cached writes are discarded.

Flush Done Places the FlashCopy mapping in the Prepared state. This is done once all cached data from the source VDisk has been flushed and all cached data of the target VDisk has been invalidated.

Start Places the FlashCopy mapping in the Copying state. The FlashCopy mapping is being started. The start of the FlashCopy mapping is synchronized (ensuring consistency across all involved mappings in the consistency group).

Chapter 5. FlashCopy 111 Stop Places the FlashCopy mapping in either the Stopped or Stopping state. A FlashCopy mapping can be stopped by a user command or by an I/O error.

Stop Complete Places the FlashCopy mapping in the Stopped state. Stop Complete occurs as a transition from Stopping to Stopped as a result of a complete copy operation to break dependencies between FlashCopy mappings.

Flush Failed Places the FlashCopy mapping in the Stopped state. This happens when the flush of data from the cache fails.

Copy Complete Places the FlashCopy mapping in the Idle/Copied state. This happens once every grain has been split. With the use of the autodelete feature, the FlashCopy mapping is deleted once it completes the copy. Without it, the FlashCopy mapping remains and can be prepared and started again. The copy complete event does not occur while mappings that depend on it are still in the Copying state.

Bitmap Offline/Bitmap Online If all nodes in the I/O group that is storing the bitmap for a FlashCopy mapping go offline, the bitmap goes offline. When the last node to go offline comes back online, the bitmap comes back online. Bitmap Offline places the FlashCopy mapping in the Suspended state. Bitmap Online places the FlashCopy mapping in the Copying or Stopping state.

Modify This event does not place the FlashCopy mapping in another state. Many properties can now be modified - name, consistency group, background copy rate, cleaning rate, and the autodelete attribute.

Delete The Delete event deletes the FlashCopy mapping.

112 SVC 4.2.1 Advanced Copy Services Figure 5-7 is the FlashCopy mapping state diagram. It illustrates which state a mapping can be in, and what events are responsible for a state change.

Stop Completes (target is not complete)

Stop Completes (target is complete) Stop Bitmap STOPPED Stop STOPPING Online SUSPENDED Completes (target is not complete) Bitmap Offline Delete (Force)

Create Flush Stop Bitmap Bitmap Prepare Failed Stop Online Offline Delete, Autodelete

IDLE_OR_COPIED PREPARING PREPARED COPYING Flush Prepare Start Done

Copy Complete

Legend Stop Completes (target is complete)

FLASHCOPY MAPPING STATE

FlashCopy Event

Figure 5-7 FlashCopy mapping states diagram

Chapter 5. FlashCopy 113 5.4.3 FlashCopy consistency group states

For reference, the FlashCopy consistency group state diagram is shown in Figure 5-8. This diagram shows the state transitions for consistency groups and the possible states of the mappings within the consistency groups.

STOPPED STOPPING

STOPPED STOPPING IDLE_OR_ COPIED STOPPED IDLE_OR_COPIED

SUSPENDED

COPYING SUSPENDED STOPPING STOPPED IDLE_COPIED

PREPARING PREPARED COPYING IDLE_OR_COPIED COPYING IDLE_OR_COPIED STOPPED PREPARED STOPPING PREPARED STOPPED PREPARING IDLE_OR_COPIED IDLE_OR_COPIED

Legend

CONSISTENCY_GROUP STATE

VALID MAPPING STATE

Figure 5-8 FlashCopy consistency group states

5.5 Dependencies between FlashCopy mappings

FlashCopy mappings, which are members in a tree, can depend on each other. A dependency is when one or more mappings are responsible for providing grains for another mapping. In such a case, actions such as the deletion of one mapping can affect other mappings that depend on the mapping being deleted.

A dependency between mappings exists only as long as a source VDisk of a mapping does not physically hold all grains. This can happen only when a tree contains preceding mappings that have not yet completed the background copy process to establish an independent clone of the source VDisk.

114 SVC 4.2.1 Advanced Copy Services 5.5.1 Linked lists of mappings and resulting dependencies

Linked lists Linked lists are dynamic internal data structures that hold information about active FlashCopy mappings in an MTFC setup, a FlashCopy cascade, or a combination of both. A mapping is inserted into a linked list of mappings when it is being started, not when the mapping is created. You can set up a cycle of mappings, but all of them cannot be active at the same time; no cycle is allowed in a linked list.

Dependencies and effect on mappings Dependencies in a dependency chain exist as long as a single mapping contains unsplit grains. When a target VDisk in a mapping has received all grains from the source VDisk, the target VDisk is no longer dependent on the source VDisk. Similarly, in a linked list of mappings, when all grains from a target VDisk (acting as a source) have been copied to a dependent target VDisk, the dependency ceases to exist.

Dependencies between mappings have effects such as the dependent mappings stop when a mapping they depend on stops (either because it is force-stopped or has an error). Also, the target VDisks of the dependent mappings go offline if the target VDisk of the mapping they depend on goes offline. This happens when the source VDisk goes offline while the background copy is incomplete.

Dependencies with Cascaded FlashCopy In the case of CFC, the dependency exists because the source VDisk of a mapping can be the target VDisk of another mapping that does not hold all the data physically. This is because the background copy process of the preceding mapping is not complete. Thus grains can still reside on the source VDisk of the preceding mapping in the cascade. A cascaded mapping cannot be prepared in case active mappings exist further down the cascade. The command to display these dependent mappings is shown in 5.5.3, “Commands to obtain information about dependencies” on page 120.

For Cascaded Mappings, if mappings in the linked list below the current mapping are active (not stopped or Idle Copied), this mapping cannot be started. To start this mapping, the downstream mappings must be stopped.

Dependencies with Multiple Target FlashCopy With MTFC there is no obvious “preceding” mapping because all target VDisks share the same source VDisk, which holds all data physically. Nonetheless, because of the implementation of FlashCopy using linked lists, an order has been established, which makes mappings precede other mappings. In the case of many mappings with the same source, to avoid too much load on the source VDisk, MTFC is implemented in such a way that mappings try to get grains from preceding mappings in the list, which are newer members of the list (that is, mappings started later).

Position of mappings and resulting dependencies The first mapping in a linked list simply maps one source VDisk to a target VDisk (first member, oldest mapping). New mappings are inserted into the linked list (when the mapping is being started) using the following rules: A new cascaded mapping is inserted behind the mapping whose target VDisk is being used as the source VDisk.

Chapter 5. FlashCopy 115 A new multiple target mapping is inserted in front of the next oldest mapping of the same source VDisk. If multiple mappings are started at the same time, because they are in the same consistency group, an unspecified ordering is chosen.

The mapping in the first position, at the “head” of the inked list (not necessarily the one being inserted first), does not depend on other mappings. Every other succeeding mapping depends on the preceding mappings in the list.

The illustrations in this section show how mappings are inserted into a linked list, depending on their being set up as an MTFC, or a CFC, or as both.

A simple MTFC is shown in Figure 5-9.

Vdisk 1

s m1, s m2

Mapping m1 Mapping m2 started at t1 started at t2

Vdisk 2 Vdisk 3

t m1 t m2

time

Figure 5-9 Multiple Target FlashCopy - m2 started after m1

Both mappings, sharing the same source VDisk, result in an MTFC, which results in a linked list consisting of two mappings, shown in Figure 5-10. These lists start with the leftmost mapping, which is called the head of the list.

m2 m1 started at t2 started at t1

Figure 5-10 Linked list of MTFC mappings m1 and m2

The arrow in Figure 5-10 originates from the mapping that is dependent on the mapping to which the arrow points.

116 SVC 4.2.1 Advanced Copy Services Mapping m2, which was started after mapping m1, precedes mapping m1 in the list. Mapping m1 now depends on mapping m2. This means, for any grains not located in mapping m1, the target VDisk of mapping m2 will be used. Figure 5-11 shows which VDisks provide grains for VDisk 2.

Vdisk 1 provides grains to Vdisk 2 in case Vdisk 3 could not provide the grain

Vdisk 1 Vdisk 3 provides grains to provides grains to Vdisk 3 Vdisk 2

Vdisk 1 Vdisk 3 Vdisk 2

s m1, s m2 t m2 t m1

Figure 5-11 VDisks provide grains

A simple CFC is shown in Figure 5-12.

Mapping m1 Mapping m2 started at t1 started at t2

Vdisk 1 Vdisk 2 Vdisk 3

s m1 t m1 t m2 s m2

time

Figure 5-12 Cascaded FlashCopy (CFC) - m2 started after m1

Chapter 5. FlashCopy 117 Figure 5-13 shows the resulting linked list.

m1 m2 started at t1 started at t2

Figure 5-13 Linked list of CFC mappings m1 and m2

Figure 5-14 shows which VDisks provide grains.

Vdisk 1 provides grains to Vdisk 3 in case Vdisk 2 could not provide the grain

Vdisk 1 Vdisk 2 provides grains to provides grains to Vdisk 2 Vdisk 3

Vdisk 1 Vdisk 2 Vdisk 3

t m1 s m1 t m2 s m2

Figure 5-14 VDisks providing grains

118 SVC 4.2.1 Advanced Copy Services The rules for inserting mappings to the list also apply when combining MTFC and CFC. An example of such a tree is shown in Figure 5-15.

Vdisk 1

s m1, s m2

Mapping m1 Mapping m2 Mapping m3 Mapping m4 started at t1 started at t2 started at t3 started at t4

Vdisk 2 Vdisk 3 Vdisk 4 Vdisk 5

t m1 t m2 t m3 t m4 s m3 s m4

time

Figure 5-15 Combined MTFC and CFC

The resulting linked list is shown in Figure 5-16.

m2 m3 m4 m1 started at t2 started at t3 started at t4 started at t1

Figure 5-16 Linked list of combined MTFC and CFC

Writes to target VDisks in a linked list For MTFC, writes to a target VDisk cause writes to target VDisks of the next dependent mapping. When an application writes to a target VDisk gets written, its content changes compared to the content of the source VDisk for the specified point in time.

This change, however, is not supposed to happen on other mappings. To retain the data of the point in time for other dependent mappings, the target VDisk of the preceding mapping copies the grains to be changed to the target VDisk of the dependent mapping. For CFC, a write to a target VDisk causes a write to the target VDisk of the next mapping in the cascade. This write to an unsplit grain on this target VDisk merges the data from the target VDisk of the previous mapping in the cascade with the storage client write data and writes the result to the next in the cascade.

Chapter 5. FlashCopy 119 5.5.2 Stopping state and cleaning

When a mapping must be stopped and deleted, you have to deal with dependencies. Stopping and deleting a mapping has an impact on dependent mappings for which a VDisk in the deleted mapping held grains that were still not split. This, of course, must be prevented from happening.

To avoid impacting mappings that depend on the mapping to be stopped, the Stopping state is used. During the Stopping state, a copy process finds and copies all grains that are unique on the mapping to be stopped; the grains are copied to another mapping in the tree. When this process is complete, the mapping can enter the Stopped state and be deleted. The target VDisk of a FlashCopy mapping in the Stopping state is offline if the target is not an independent copy and is, therefore, not accessible to storage clients for an what may be an unacceptably long time. If it is an independent copy, it is online while the mapping is stopping.

To resolve this situation, a new method of copying grains is being implemented in SVC V4.2.1 - Cleaning mode. In this mode, grains are copied from the mapping to be stopped while it is still in the Copying state (and while its target VDisk is still accessible to the storage client) to an older mapping.

This mode is implemented using a new Cleaning Rate parameter. If this parameter is combined with a zero copy background rate, it causes the mapping to copy the grains to a downstream mapping.

Information about the progress is contained in a new Cleaning Progress field introduced in the query output of the mapping.

After the cleaning is complete (all unique grains are copied), the mapping can be stopped, and it will be in a Stopping state immediately. If the mapping is stopped before the cleaning process completed, the mapping enters the Stopping state first, and the remaining grains are copied. After this completes, the mapping enters the Stopped state.

5.5.3 Commands to obtain information about dependencies

Using the command svcinfo lsvdiskdependentmaps, we can display all mappings with target VDisks that are dependent upon data held on the specified VDisk. Cascaded mappings cannot be prepared (and then started) in case active mappings exist further down the cascade, which can be displayed using this command. The syntax is shown in Example 5-2.

Example 5-2 svcinfo lsvdiskdependentmaps >>- svcinfo -- -- lsvdiskdependentmaps --+- vdisk_id ---+------>< '- vdisk_name -'

In addition, we can specify the mapping using the command svcinfo lsfcmapdependentmaps, which displays all the FlashCopy mappings that are dependent on the user-specified mapping. The syntax is shown in Example 5-3.

Example 5-3 svcinfo lsfcmapdependentmaps >>- svcinfo -- -- lsfcmapdependentmaps -- --+------+------> '- -nohdr -'

>--+------+-- --+- fc_id ---+------>< '- -delim -- delimiter -' '- fc_name -'

120 SVC 4.2.1 Advanced Copy Services 6

Chapter 6. Implementing FlashCopy

In this chapter we show examples of how to use FlashCopy by utilizing the GUI and the CLI.

The first example we show is basic FlashCopy consisting of one mapping where not all data of the VDisk must be copied physically.

The goal is to back up the production data using a small backup window, which is one of the most basic problems. As the data to backup increases, the time required to back up the data increases. In cases where a full consistent copy is required, the application manipulating this data must be suspended from changing the data for the entire time the backup is being created.

Our solution is to use FlashCopy to create a snapshot of the data. We do so by creating a virtual copy of the data for the given point in time. Because the FlashCopy needs only a short time to complete its copy, the application has to be prevented from changing the data only for a short time.

Of course, the data should be consistent and all involved caches must be flushed and written to the VDisk. The target copy can instantly be accessed by the backup application while the source is ready to be accessed and used (and to be changed) again.

6.1.1 Creating a FlashCopy mapping

We can start by creating a single FlashCopy as shown in Figure 6-1, on the Viewing FlashCopy Mappings window. Existing mappings are displayed (none for now). The only action possible at this point is to create a new mapping.

Figure 6-1 View FlashCopy mappings - no mapping exists

122 SVC 4.2.1 Advanced Copy Services The first window of the Create a Mapping wizard is shown in Figure 6-2. It provides an overview of the steps we go through to create the mapping.

Figure 6-2 Create a FlashCopy Mapping - Introduction

Chapter 6. Implementing FlashCopy 123 The wizard is very straightforward and consists of setting the properties of the FlashCopy mapping, choosing the VDisks to be used, and verifying the settings. The first step, setting the properties of the FlashCopy mapping, is shown in Figure 6-3.

Figure 6-3 Create a FlashCopy fcm - Set Properties

We can choose the FlashCopy mapping name or let the system decide. We have not defined a consistency group yet, so the related input field Consistency Group Name is empty. Therefore, this mapping is associated with consistency group 0, which is used for standalone mappings.

The Background Copy Priority field specifies the speed of the background copy the SVC attempts to achieve, in case we want the source copied to the target in full. Possible speeds are from 128 KBps at 10% priority up to 64 MBps at 100% priority. Priority 50 has a target copy rate of 2 MBps. Refer to 5.2.3, “Copy background copy” on page 100 for a discussion of background copy.

For our purpose, we chose NOCOPY, which means no data gets copied by the background copy process. You do not have to set the Cleaning Priority field because no mapping exists yet that depends on this mapping (this new parameter is discussed in 5.5, “Dependencies between FlashCopy mappings” on page 114).

The type of mapping is Standard, as opposed to Incremental. We do not need Incremental because we do not want a full copy of the source VDisk, which could benefit from being incrementally refreshed. Also, Incremental FlashCopy uses up to twice the amount of bitmap space compared to standard FlashCopy (Incremental FlashCopy is discussed in 5.2.8, “Incremental FlashCopy” on page 104).

124 SVC 4.2.1 Advanced Copy Services The grain size we use is 256 KB. The new option of a grain size of 64 KB uses four times the amount of bitmap space compared to a grain size of 256 KB. A grain size of 64 KB can be used to decrease the amount of data to be copied when restarting an Incremental FlashCopy mapping and thus the time it takes to reestablish a full independent clone of a source VDisk. The last option on this window is our choice of whether we want the system to automatically delete the FlashCopy mapping when the background copy completes. We are not creating a full copy here, so we do not use this option.

Once we click the Next button, we view a window where we can filter the source VDisk candidates we want displayed in the next step, shown in Figure 6-4.

Figure 6-4 Create a FlashCopy mapping - Filtering Source VDisk Candidates

In the Filtering Source VDisk Candidates window, we can choose to filter by Virtual Disk (VDisk) Name and Managed Disk Group Name (using a wildcard), Virtual Disk Type (any, sequential, striped, image), I/O Group, and State (offline, online, degraded).

In our example we choose to filter by VDisk name because our naming scheme indicates the production VDisks to be used as source VDisk with the characters “prod” as part of the VDisk name.

Chapter 6. Implementing FlashCopy 125 The next window shows us the filtered VDisks, as shown in Figure 6-5.

Figure 6-5 Create a FlashCopy mapping - Select the Source VDisk

We select the vd-prod-0001 VDisk to be the source VDisk for our FlashCopy mapping. Clicking Next, we view a window where we can choose the target VDisk - this time without a filter and we display all VDisks in the system.

To filter the target VDisk candidates, we select Show Filter Row from the pull-down menu, then click filter above the Name column to view the filter properties, shown in Figure 6-6.

Figure 6-6 Create a FlashCopy mapping - Select the Target VDisk - Filter

126 SVC 4.2.1 Advanced Copy Services In the Select the Target VDisk window, we do not use the wildcard as we did within the filter window for the source VDisk candidates. After selecting the VDisk to be the target VDisk and clicking Next, we verify the settings of the FlashCopy mapping as shown in Figure 6-7.

Figure 6-7 Create a FlashCopy mapping - Verify FlashCopy Mapping

Clicking Finish creates the mapping and produces the Viewing FlashCopy window, shown in Figure 6-8.

Figure 6-8 One mapping created in Idle or Copied state

Chapter 6. Implementing FlashCopy 127 We have successfully created the first FlashCopy mapping whose properties we can view, clicking the FlashCopy mapping name, shown in Figure 6-9.

Figure 6-9 View FlashCopy Mapping Details for fcm-001

We see the state of the mapping is Idle_or_Copied; therefore the mapping is ready to be prepared and started. The consistency group-related fields hold no information; internally this mapping belongs to consistency group 0.

128 SVC 4.2.1 Advanced Copy Services 6.1.2 Starting a FlashCopy mapping

Given that the application accessing and altering the data on the source VDisk has been stopped from changing the data, we can now start the FlashCopy mapping. To start it, we must prepare and then start the mapping. We can do so either by selecting Prepare a Mapping from the pull-down menu or by selecting Start a Mapping, as shown in Figure 6-10.

Figure 6-10 Start a Mapping

We view the window that enables us to prepare the mapping, shown in Figure 6-11.

Figure 6-11 Starting FlashCopy mapping fcm-001

Clicking OK first prepares the mapping (that is, the mapping is in the Preparing state) and then starts the mapping. The state of the mapping is now Copying, as shown in Figure 6-12. The application altering the data residing on the source VDisk can resume, and the target VDisk is brought online and is ready.

Figure 6-12 One mapping started, Copying state

Chapter 6. Implementing FlashCopy 129 6.1.3 Modifying a mapping

At this point, we can modify the properties of the mapping we created and started. Selecting the mapping and choosing Modify a Mapping from the pull-down menu takes us to the window shown in Figure 6-13.

Figure 6-13 Modifying FlashCopy Mapping fcm-001

On the Modifying FlashCopy Mapping window, we can modify the name of the mapping, the Consistency group it belongs to, the background copy priority, the cleaning priority, and the autodelete option.

We cannot modify the grain size, the type of mapping (standard, incremental), or the I/O group the bitmap resides on.

130 SVC 4.2.1 Advanced Copy Services 6.1.4 Stopping a mapping

After we have backed up our data from the target VDisk, we stop the mapping, as shown in Figure 6-14. This is needed in the next step, when we use this mapping as a member in a new consistency group.

Figure 6-14 Stop a Mapping

At this point, we get a warning window, shown in Figure 6-15, telling us that the content of the target VDisk will be unusable after we stop the mapping.

Figure 6-15 FlashCopy mapping fcm-001 - Stop or Forced Stop

The content of the target VDisk is unusable because most of the data on the target VDisk is still located on the source VDisk; only data that could have been overwritten on the source VDisk has been copied to the target VDisk (we chose not to enable the background copy process). This means that the mapping still exists, but whatever data was written to the target VDisk is invalidated.

Chapter 6. Implementing FlashCopy 131 In addition, any mapping that depends on this mapping is affected. When we use Forced Stop, no further cleaning occurs, and dependent mappings are also stopped. We do not have any dependent mappings at that point in time, so we click the Stop button (we do not need to click the Forced Stop button). At this point, the status of the mapping is Stopping, as seen in Figure 6-16, and shortly thereafter in the state Stopped.

Figure 6-16 FlashCopy mapping, Stopping state

6.1.5 Creating a consistency group

Our application has related data spanning multiple VDisks. To be able to produce a consistent snapshot of the data, we put them in a consistency group to ensure consistency of not only the data within one VDisk but also across data spanning all the involved VDisks. First we choose FlashCopy Consistency Groups on the My Work pane as shown in Figure 6-17.

Figure 6-17 Viewing FlashCopy Consistency Groups - no groups exist

132 SVC 4.2.1 Advanced Copy Services No consistency groups have been set up yet. To create one, we choose Create a Consistency Group from the pull-down menu. The next window, shown in Figure 6-18, enables us to choose the name of the consistency group and the FlashCopy mappings we want to be members in it.

Figure 6-18 Creating FlashCopy Consistency Groups - one existing mapping

We want to include the mapping FlashCopy fcm-001 as the first member in the consistency group; therefore, we select it and press OK. If we had created more mappings, they would be displayed here and could be included in the consistency group at once.

The next window, shown in Figure 6-19, displays the result of the creation, which was a success in our case. If, for instance, the mapping put into the consistency group was in the Copying state, we would view an error message. Nonetheless, an empty consistency group would be created.

Figure 6-19 Creating FlashCopy Consistency Groups - Success

Chapter 6. Implementing FlashCopy 133 Closing this window brings us back to the Viewing FlashCopy Consistency Group window seen in Figure 6-20.

Figure 6-20 Viewing FlashCopy Consistency Groups - one group created

Clicking the consistency group Name shows us the properties of the consistency group. The first panel, General, shows the name, ID, and the state of the consistency group, as shown in Figure 6-21.

Figure 6-21 View FlashCopy Consistency Group General Details for fcg-001

The consistency group state is Stopped because its member, the FlashCopy mapping fcm-001 is in the Stopped state. The Relationships panel shows the details for the relationship as shown in Figure 6-22.

Figure 6-22 View FlashCopy Consistency Group Relationship Details for fcg-001

134 SVC 4.2.1 Advanced Copy Services 6.1.6 Creating a mapping and adding it to a consistency group

We can create a new mapping without choosing a consistency group, thus associating it with consistency group 0, as we did in 6.1.1, “Creating a FlashCopy mapping” on page 122. We can later place it in a consistency group from within the Viewing FlashCopy Mappings window by selecting Modify a Mapping from the pull-down menu.

If we already know which consistency group the mapping should be a member of, we can put the mapping in the consistency group from the Create a FlashCopy Mapping wizard in one step, shown in Figure 6-23.

Figure 6-23 Create a FlashCopy mapping - Set Properties

Chapter 6. Implementing FlashCopy 135 The next window enables us to filter the VDisks to be displayed as source VDisk candidates. This time we filter using the Managed Disk Group name field (the backend storage system that holds the production VDisks is the DS4500), shown in Figure 6-24.

Figure 6-24 Filter source VDisk candidates

At this point, we choose the source VDisk for the mapping from the filtered list, shown in Figure 6-25.

Figure 6-25 Select the Source VDisk

136 SVC 4.2.1 Advanced Copy Services At this point, we choose the target VDisk for the new mapping, as shown in Figure 6-26.

Figure 6-26 Select the Target VDisk

The last step before the mapping is created is to verify the settings, as shown in Figure 6-27.

Figure 6-27 Verify FlashCopy Mapping

Chapter 6. Implementing FlashCopy 137 At this point, we return to the consistency group window, shown in Figure 6-28, and we see that both mappings are now members of the consistency group fcg-001.

Figure 6-28 Two mappings are members of one consistency group

The state of both mappings is Idle_or_Copied, which therefore is also the state of the consistency group (see Figure 6-29).

Figure 6-29 Viewing FlashCopy Consistency Groups - fcg-001, Idle or Copied state

138 SVC 4.2.1 Advanced Copy Services When we click the name of the consistency group, we access the details of the consistency group. Clicking Relationships, as shown in Figure 6-30, we see both FlashCopy mappings as members of the consistency group.

Figure 6-30 View FlashCopy Consistency Group Relationship Details for fcg-001

6.1.7 Preparing a consistency group

We must prepare a consistency group (as well as a FlashCopy mapping) before we can start it (see 5.4.3, “FlashCopy consistency group states” on page 114). Although a consistency group is automatically prepared when we choose Start a consistency group, doing it manually has some advantages.

Depending on the amount of data associated with the VDisks to be copied still in the cache and the load on the system, it can take some time to prepare a mapping. One advantage to preparing a consistency group manually is that time does not run out for the completion of the start process.

For example, the prepare command is asynchronous, and if the preparation time for maps exceeds two minutes, it does not cause a problem. However, the start command is synchronous and must complete within two minutes.

In a case where the start command is prevented from executing within two minutes, the event flush failed takes the mapping into the Stopped state. While in the Stopped state, it can be prepared again. The advantage of preparing the mappings first is that in the case just described, the mappings are already prepared; thus the time does not run out for the completion of the start process.

Another advantage to preparing a consistency group manually is that we can specify the point in time to start the consistency group or the mapping, whereas with an unprepared consistency group, the (variable) time to prepare the consistency group decides the exact point in time it is started.

Chapter 6. Implementing FlashCopy 139 We begin the process of preparing a consistency group by selecting Prepare a Consistency Group, as shown in Figure 6-31.

Figure 6-31 Prepare a Consistency Group

A consistency group can be put into the Preparing state, shown in Figure 6-32, only when it is either in the Idle_or_Copied or Stopped state.

Figure 6-32 Preparing state

140 SVC 4.2.1 Advanced Copy Services After a refresh, the state changes to Prepared, as shown in Figure 6-33.

Figure 6-33 Prepared state

6.1.8 Starting a consistency group

To start our consistency group without preparing it first, we use the pull-down menu from the Viewing FlashCopy Consistency Group window, as shown in Figure 6-34.

Figure 6-34 Start a Consistency Group

Chapter 6. Implementing FlashCopy 141 As with starting the standalone mapping, when starting the consistency group, the next window prompts us to prepare the consistency group, as shown in Figure 6-35. This applies only if we did not prepare the consistency group previously. When we prepare a consistency group ahead of time, the check box is greyed out. The check box is checked by default but can be unchecked.

Figure 6-35 Prepare mappings

As stated earlier, a consistency group or a FlashCopy mapping always must be prepared before being started. The check box serves no other purpose than informing us that the consistency group will be prepared before it is started. If we uncheck it, the start of the consistency group fails, as shown in Figure 6-36.

Figure 6-36 Starting consistency group fails

142 SVC 4.2.1 Advanced Copy Services Click the OK button with the check box left checked, The consistency group first enters the Preparing state, then the Prepared state, and finally the Copying state, in one uninterrupted sequence, as shown in Figure 6-37.

Figure 6-37 Viewing FlashCopy consistency groups, fcg-001 in Copying state

Looking at the details of the consistency group fcg-001, we see that both mappings, which are members of the consistency group, are in the Copying state, shown in Figure 6-38.

Figure 6-38 Copying state - fcg-01

Chapter 6. Implementing FlashCopy 143 Because we have chosen the NOCOPY option, the target VDisks are a valid snapshot of the source VDisks as long as the mappings exist. All we can do to manipulate the consistency group at this point is to stop it, which ends the relationship and renders all data on the target VDisk invalid. To stop the consistency group, we choose Stop a Consistency Group from the pull-down menu (see Figure 6-39).

Figure 6-39 Stop a consistency group

Figure 6-40 shows the confirmation window where we can stop the consistency group or chose to force the consistency group to be stopped in case mappings have dependencies to other mappings outside the consistency group to be stopped.

Figure 6-40 Stopping FlashCopy Consistency Group fcg-001

144 SVC 4.2.1 Advanced Copy Services Before reaching the Stopped state, the consistency group enters the Stopping state, as shown in Figure 6-41.

Figure 6-41 Stopping state

6.1.9 Renaming a consistency group

The only property of a consistency group that we can change from the Viewing FlashCopy Consistency Groups window is to rename the consistency group. Again, we use the pull-down menu to chose the action as shown in Figure 6-42.

Figure 6-42 Rename a Consistency Group

Chapter 6. Implementing FlashCopy 145 The Renaming FlashCopy Consistency Group window enables us to change the name of the consistency group, which can be done in the Copying state, as shown in Figure 6-43.

Figure 6-43 Renaming FlashCopy Consistency Group fcg-001

6.1.10 Issuing a command to a mapping within a consistency group

From the Viewing FlashCopy Mappings window, we can stop a FlashCopy mapping. If the mapping is a member of a consistency group, we stop the entire consistency group while stopping the mapping.

When we stop the mapping FlashCopy fcm-001, the other mappings, which are members of that consistency group, in the Stopping state, shown in Figure 6-44, and then in the Stopped state.

Figure 6-44 One in Stopped mapping state, one in Stopping state

146 SVC 4.2.1 Advanced Copy Services When we prepare or start a mapping that is a member of a consistency group, we are presented with a confirmation window that informs us that because the FlashCopy mapping is a member of a consistency group, the command will be issued for all mappings that are members of the consistency group, as shown in Figure 6-45.

Figure 6-45 Starting FlashCopy Mappings Confirmation window

6.2 Example 2: Multiple Target FlashCopy using the GUI

In this section, one VDisk acts as a source VDisk and can be in multiple mappings with different VDisks, acting as the target VDisk.

Our goal is to create multiple consistent copies of the same data. If we need copies of VDisks for different purposes (and with different properties), we can create these copies of the source at the same time.

Our solution is to use MTFC to create multiple snapshots of one source. For instance, with MTFC we can create FlashCopies for our daily backup and others for application testing using the same source VDisks.

The snapshots for backup can be set up as described in 6.1, “Example 1: Basic FlashCopy using the GUI” on page 122. We set up the snapshots to be used for application testing with different properties. For example, we want the VDisks copied fully using the background copy process to establish real independent clones.

Chapter 6. Implementing FlashCopy 147 6.2.1 Creating mappings and a consistency group

The FlashCopy mappings are already established for our backup purposes, as shown in Figure 6-46.

Figure 6-46 Two mappings for backup in consistency group fcg-001

148 SVC 4.2.1 Advanced Copy Services Clicking the name of the mapping accesses its properties, as shown in Figure 6-47.

Figure 6-47 View FlashCopy details for fcm-001

Chapter 6. Implementing FlashCopy 149 At this point, we establish two new mappings that use the same source VDisk but have a different target VDisk (see Figure 6-48).

Figure 6-48 Two mappings using the same source

150 SVC 4.2.1 Advanced Copy Services The properties of the mapping FlashCopy fcm-test-001 are shown in Figure 6-49.

Figure 6-49 View FlashCopy Mapping Details for fcm-test-001

The background copy priority is 50% for these mappings. This tells the SVC to perform a background copy of the data of the source VDisk to the target VDisk with 2 MBps.

Because we do not need the mapping after the clone of the source VDisks is established, we enable the option to delete the mapping when the background copy has finished.

Chapter 6. Implementing FlashCopy 151 The mappings are not yet in a consistency group. Because we want the timing of the FlashCopy to be independent from the consistency group we use for backup, we put the mappings in a new consistency group. At this point, we create the consistency group and while doing so, put both new FlashCopy mappings into the consistency group, as shown in Figure 6-50.

Figure 6-50 FlashCopy consistency groups include both mappings for test data

To check the settings, we look up the FlashCopy relationship details of the consistency group, as shown in Figure 6-51.

Figure 6-51 FlashCopy Consistency Group Relationship Details for fcg-test-001

152 SVC 4.2.1 Advanced Copy Services 6.2.2 Starting a consistency group in an MTFC tree

After we set up the consistency group, we can prepare and start it. As we can see in Figure 6-52, the mappings in consistency group fcg-test-001 are in the Copying state, while the mappings in fcg-001 are in the Idle_Copyied state.

Figure 6-52 Viewing FlashCopy Mappings - fcg-test-001 mapping in Copying state

At this point, we start (and prepare in one step) the FlashCopy mapping fcg-001 again for our backup purposes.

Both of our consistency groups, with mappings that share the same source VDisks, are in the Copying state. We can select a mapping in fcg-001 and choose View FC Mappings in Dependency Order from the pull-down menu, shown in Figure 6-53, to access information about the mapping dependencies.

Figure 6-53 View FC Mappings in Dependency Order

Chapter 6. Implementing FlashCopy 153 The list of mappings that are dependent on the mapping fcm-001 are shown in Figure 6-54.

Figure 6-54 View mappings in dependency order for mapping fcm-001

The only mapping that depends on the chosen mapping is fcm-test-001. The mapping fcm-001 itself is not shown in the list. The mapping fcm-test-001 depends on the mapping fcm-001 because fcm-001 is newer (started later) than fcm-test-001.

After the SVC finished copying the data of the source VDisks to the target VDisks for the mapping used to establish a clone of the production VDisks, it then deleted the related mapping as we had specified.

At this point, we have only two mappings, which are the ones we use for our backups, as shown in Figure 6-55.

Figure 6-55 Mappings in consistency group fcg-001

154 SVC 4.2.1 Advanced Copy Services 6.3 Example 3: Cascaded and Incremental FC using the GUI

In this section, we implement FlashCopy where a VDisk can be a target VDisk in one mapping and a source VDisk in another mapping. In addition, we implement a FlashCopy that can be incrementally refreshed. We do so using consistency groups.

Our first goal is to take a snapshot of a VDisk, which is already the target VDisk in a mapping. We create and start a FlashCopy mapping, and we, want to establish a FlashCopy where the target VDisk of the first FlashCopy mapping is the source of the second FlashCopy mapping.

The first copy we set up with background copy to obtain a fully copied clone of a production VDisk (this first copy is to be used for application testing). However, the application that is using the first copy might change the data residing on the target VDisk of the first FlashCopy mapping. At this point we want to copy the target VDisk of the first FlashCopy mapping. With a FlashCopy function where a VDisk can be a member of only one FlashCopy mapping, we would have to wait until the first copy was finished to fully copy the data to the target VDisk. Then we would have to delete the mapping (or enable the autodelete option), and this procedure can be time consuming.

Our solution is to use CFC to create a FlashCopy where a source VDisk of the second mapping is still the target VDisk in the first FlashCopy mapping. This enables us to create the second copy before the first FlashCopy mapping has finished copying all the data.

Our second goal is to refresh a FlashCopy target. We want the first copy (the copy of the production VDisk) to be reestablished within a short period of time and without much impact on the production VDisk.

Our solution is to use IFC for the first FlashCopy mapping, which reestablishes the first copy in a short period of time with little impact on the production volume.

6.3.1 Creating consistency groups

First we create two FlashCopy consistency groups, named so that we know the second consistency group is supposed to hold mappings for the “copy of the copy” (see Figure 6-56).

Figure 6-56 Create two empty consistency groups

Chapter 6. Implementing FlashCopy 155 6.3.2 Creating Incremental FlashCopy mappings

We put the mapping to be created into the first consistency group. We also set it up to be incremental, which is shown in Figure 6-57.

Figure 6-57 Create Incremental FlashCopy mapping

156 SVC 4.2.1 Advanced Copy Services At this point, we have associated the source VDisks and the target VDisk with the mapping, and we finish with the wizard. The properties of the mapping we created are shown in Figure 6-58.

Figure 6-58 View FlashCopy Mapping Details for FlashCopy fcm-test-001

Chapter 6. Implementing FlashCopy 157 We set up the second mapping for the second production VDisk the same way we did with the first. Both mappings are in the Idle_or_Copied state and are shown in Figure 6-59.

Figure 6-59 Viewing FlashCopy Mappings - two mappings in first consistency group

At this point, we can switch to the Viewing FlashCopy Consistency Group window and start the first consistency group, as shown in Figure 6-60.

Figure 6-60 Start a Consistency Group

158 SVC 4.2.1 Advanced Copy Services This puts the consistency group in the Copying state, as shown in Figure 6-61.

Figure 6-61 View FlashCopy Consistency Groups

At this point, the target VDisks of the first mapping are ready to be used for our application testing. We have to temporarily stop application testing (and flush any host caches) before we can start the cascaded mappings. Meanwhile, we create the next two mappings, where the target VDisks of the now Copying first mapping are the source VDisks.

We put the mappings in another consistency group and specify them as incremental, as shown in the Verify FlashCopy Mapping window at the final stage of creating the mapping (see Figure 6-62).

Figure 6-62 Verify FlashCopy Mapping

Chapter 6. Implementing FlashCopy 159 At this point, we have set up four mappings in two consistency groups, as shown in Figure 6-63.

Figure 6-63 One new consistency group in Idle_or_Copied state

While the first consistency group and the mappings associated with it are still in the Copying state, we start the second consistency group, as shown in Figure 6-64.

Figure 6-64 Start a consistency group

160 SVC 4.2.1 Advanced Copy Services At this point, we have four mappings in two consistency groups in a Copying state, where VDisks act as both source VDisks and target VDisks (see Figure 6-65).

Figure 6-65 VDisks in the role of source VDisk and target VDisk

The mapping FlashCopy fcm-test-003 depends on FlashCopy fcm-test-001, as shown in Figure 6-66. The same applies to the other two mappings.

Figure 6-66 View FlashCopy Mappings in Dependency Order for mapping fcm-test-001

Chapter 6. Implementing FlashCopy 161 After some time, the Viewing FlashCopy Mappings window indicates the copy process for the first FlashCopy mapping is almost complete (see Figure 6-67).

Figure 6-67 First mapping copy process almost complete

We can see the progress using the View Progress window, shown in Figure 6-68, where we see that the first mapping successfully finished copying all the data to the target VDisk.

Figure 6-68 View FlashCopy progress

162 SVC 4.2.1 Advanced Copy Services After both copies are fully copied, the state changes back to Idle_or_Copied for both consistency groups, as shown in Figure 6-69.

Figure 6-69 Viewing FlashCopy Consistency Groups - both in Idle_or_Copied state

At this point, we restart the first consistency group, where the production VDisks are source VDisks (see Figure 6-70).

Figure 6-70 (Restart a consistency group

Chapter 6. Implementing FlashCopy 163 Soon after we start the FlashCopy mapping, we access the Viewing FlashCopy Mappings window shown in Figure 6-71. Because the mapping is incremental, FlashCopy just copied the grains that were changed, which was about 200 MB, instead of the entire 10 GB VDisk.

Figure 6-71 Mappings in fcg-test-001 refreshed

6.4 Example 4: FlashCopy of Metro Mirror VDisks using the CLI

In this section, we implement FlashCopy where the source VDisks are the secondary VDisks in a remote copy relationship.

Our goal is to create a full consistent copy of production data for a given point in time at a remote location for disaster tolerance. We set up a Remote Copy Relationship to copy VDisks to a second site using intercluster Metro Mirror. We want to take a consistent backup of our production data at the remote site. Also, we want a full copy (a clone) of our production VDisks taken on a daily basis and residing at the second location. This copy serves the purpose of providing a consistent point-in-time copy of the production data in case the data of both the primary and secondary of the Metro Mirror relationship become invalidated (for example, due to a logical error that affects both the primary and the secondary because they are in a synchronized copy relationship). This clone is a building block for our business continuity strategy.

Note: Even though Global Mirror is “asynchronous,” this case applies to Global Mirror as well, because the data is copied over to the secondary VDisk as soon as possible.

Our solution is to set up FlashCopy where the source VDisks for the mappings are the secondary VDisks of the one Metro Mirror relationship. A backup is taken from the target VDisks. We establish the clone of the Metro Mirror secondary using the background copy process, and we use IFC to refresh this clone to minimize the load on the system and to quickly establish the clone. Establishing the clone in a short time decreases the interval when no full clone of the VDisks is available, thus enhancing our recovery time objective.

164 SVC 4.2.1 Advanced Copy Services 6.4.1 Checking VDisks and Metro Mirror relationship

To check the VDisks to be used for our FlashCopy mapping, we use the command svcinfo lsvdisk -delim :, as shown in Example 6-1. To eliminate the white space between the output columns we use the -delim switch.

Example 6-1 svcinfo lsvdisk -delim : IBM_2145:ITSOCL3:admin>svcinfo lsvdisk -delim : id:name:IO_group_id:IO_group_name:status:mdisk_grp_id:mdisk_grp_name:capacity:type :FC_id:FC_name:RC_id:RC_name:vdisk_UID:fc_map_count 0:vd-prod-l2-0002:0:io_grp0:online:1:DS4700:25.0GB:striped:::0:MetroMirror Relationship-002:6005076801A100E90800000000000003:0 1:vd-prod-l2-0001:0:io_grp0:online:1:DS4700:25.0GB:striped:::1:MetroMirror Relationship-001:6005076801A100E90800000000000002:0 2:vd-back-l2-0001:0:io_grp0:online:1:DS4700:25.0GB:striped:::::6005076801A100E9080 0000000000004:0 3:vd-back-l2-0002:0:io_grp0:online:1:DS4700:25.0GB:striped:::::6005076801A100E9080 0000000000005:0 IBM_2145:ITSOCL3:admin>

We use the VDisks whose names contain the characters prod as the source VDisk for the FlashCopy mapping and the VDisks whose name contains the characters back as the target VDisks.

For example, we display the properties of the VDisk with the ID 2, using the command svcinfo lsvdisk -delim : 2, as shown in Example 6-2.

Example 6-2 Details of VDisk 2 IBM_2145:ITSOCL3:admin>svcinfo lsvdisk -delim : 2 id:2 name:vd-back-l2-0001 IO_group_id:0 IO_group_name:io_grp0 status:online mdisk_grp_id:1 mdisk_grp_name:DS4700 capacity:25.0GB type:striped formatted:no mdisk_id: mdisk_name: FC_id: FC_name: RC_id: RC_name: vdisk_UID:6005076801A100E90800000000000004 throttling:0 preferred_node_id:2 fast_write_state:empty cache:readwrite udid:0 fc_map_count:0 IBM_2145:ITSOCL3:admin>

Chapter 6. Implementing FlashCopy 165 To display the one Metro Mirror relationship, we issue the command svcinfo lsrcconsistgrp, as shown in Example 6-3.

Example 6-3 Existing remote copy consistency groups IBM_2145:ITSOCL3:admin>svcinfo lsrcconsistgrp -delim : id:name:master_cluster_id:master_cluster_name:aux_cluster_id:aux_cluster_name:prim ary:state:relationship_count:copy_type 255:mmg-001:000002006B603A38:ITSOCL2:0000020068403A42:ITSOCL3:master:consistent_sy nchronized:2:metro IBM_2145:ITSOCL3:admin>

As we can see, only one relationship exists. To show the properties of only this relationship, we specify the ID with the command, as shown in Example 6-4.

Example 6-4 Properties of the consistency group with ID 255 IBM_2145:ITSOCL3:admin>svcinfo lsrcconsistgrp -delim : 255 id:255 name:mmg-001 master_cluster_id:000002006B603A38 master_cluster_name:ITSOCL2 aux_cluster_id:0000020068403A42 aux_cluster_name:ITSOCL3 primary:master state:consistent_synchronized relationship_count:2 freeze_time: status:online sync: copy_type:metro RC_rel_id:0 RC_rel_name:MetroMirror Relationship-002 RC_rel_id:1 RC_rel_name:MetroMirror Relationship-001 IBM_2145:ITSOCL3:admin>

The displayed information verifies that we have successfully set up the remote copy consistency group, which is associated with two Metro Mirror relationships.

We can also verify the properties of the two Metro Mirror relationships by issuing the command svcinfo lsrcrelationship -delim : 0. One of the two relationships is shown in Example 6-5.

Example 6-5 .Properties of the relationship with ID 0 IBM_2145:ITSOCL3:admin>svcinfo lsrcrelationship -delim : 0 id:0 name:MetroMirror Relationship-002 master_cluster_id:000002006B603A38 master_cluster_name:ITSOCL2 master_vdisk_id:1 master_vdisk_name:vd-prod-l1-0002 aux_cluster_id:0000020068403A42 aux_cluster_name:ITSOCL3

166 SVC 4.2.1 Advanced Copy Services aux_vdisk_id:0 aux_vdisk_name:vd-prod-l2-0002 primary:master consistency_group_id:255 consistency_group_name:mmg-001 state:consistent_synchronized bg_copy_priority:50 progress: freeze_time: status:online sync: copy_type:metro IBM_2145:ITSOCL3:admin>

The other relationship is shown in Example 6-6.

Example 6-6 Properties of the relationship with ID 1 IBM_2145:ITSOCL3:admin>svcinfo lsrcrelationship -delim : 1 id:1 name:MetroMirror Relationship-001 master_cluster_id:000002006B603A38 master_cluster_name:ITSOCL2 master_vdisk_id:0 master_vdisk_name:vd-prod-l1-0001 aux_cluster_id:0000020068403A42 aux_cluster_name:ITSOCL3 aux_vdisk_id:1 aux_vdisk_name:vd-prod-l2-0001 primary:master consistency_group_id:255 consistency_group_name:mmg-001 state:consistent_synchronized bg_copy_priority:50 progress: freeze_time: status:online sync: copy_type:metro IBM_2145:ITSOCL3:admin>

The VDisks of our Metro Mirror relationship in location 1 (production location) are named: vd-prod-l1-0001 vd-prod-l1-0002

In location 2 (business continuity location) the VDisks are named: vd-prod-l2-0001 vd-prod-l2-0002

We use the two VDisks that serve as the secondary VDisks for the Metro Mirror relationship in location 2 as the source for the FlashCopy mapping.

Chapter 6. Implementing FlashCopy 167 6.4.2 Creating mappings

We set up the FlashCopy mappings and consistency group in the following order: 1. Create FlashCopy mappings. 2. Create a FlashCopy consistency group. 3. Add FlashCopy mappings to the consistency group.

Note: Another method is to create the consistency group first. Then create the mappings and add the mappings to the consistency group in the same step.

We create the mappings using the command svctask mkfcmap. The syntax is shown in Example 6-7.

Example 6-7 svctask mkfcmap - syntax >>- svctask -- -- mkfcmap ------>

>-- -source --+- src_vdisk_id ---+------> '- src_vdisk_name -'

>-- -target --+- target_vdisk_id ---+------> '- target_vdisk_name -'

>--+------+------> '- -name -- new_name_arg -'

>--+------+------> '- -consistgrp --+- consist_group_id ---+-' '- consist_group_name -'

>--+------+-- --+------+------> '- -copyrate -- percent -' '- -autodelete -'

>--+------+--+------+------> '- -grainsize --+- 64 --+-' '- -incremental -' '- 256 -'

>--+------+------> '- -cleanrate ---- percent ---'

>--+------+------>< '- -iogrp --+- iogroup_name -+-' '- iogroup_id --'

We do not use every optional parameter; for instance, we use the default grain size at 256 KB. In our example, we create the first mapping as shown in Example 6-8.

Example 6-8 Create first mapping IBM_2145:ITSOCL3:admin>svctask mkfcmap -source vd-prod-l2-0001 -target vd-back-l2-0001 -name fcm-back-001 -copyrate 70 -incremental FlashCopy Mapping, id [0], successfully created IBM_2145:ITSOCL3:admin>

168 SVC 4.2.1 Advanced Copy Services We then create the second mapping, as shown in Example 6-9.

Example 6-9 Create second mapping IBM_2145:ITSOCL3:admin>svctask mkfcmap -source vd-prod-l2-0002 -target vd-back-l2-0002 -name fcm-back-002 -copyrate 70 FlashCopy Mapping, id [1], successfully created IBM_2145:ITSOCL3:admin>

We review the creation of both mappings using the command svcinfo lsfcmap -delim :, shown in Example 6-10.

Example 6-10 Review settings for both mappings IBM_2145:ITSOCL3:admin>svcinfo lsfcmap -delim : id:name:source_vdisk_id:source_vdisk_name:target_vdisk_id:target_vdisk_name:group_ id:group_name:status:progress:copy_rate:clean_progress:incremental 0:fcm-back-001:1:vd-prod-l2-0001:2:vd-back-l2-0001:::idle_or_copied:0:70:100:on 1:fcm-back-002:0:vd-prod-l2-0002:3:vd-back-l2-0002:::idle_or_copied:0:70:100:off IBM_2145:ITSOCL3:admin>

When reviewing the mappings, we realize that we forgot the -incremental parameter in our second mapping. We cannot modify this mapping to be incremental, so we have to delete the mapping and recreate it. (We can modify a mapping using the command svctask chfcmap.) We delete the mapping using the command svctask rmfcmap, shown in Example 6-11.

Example 6-11 svctask refcmap - syntax >>- svctask -- -- rmfcmap -- --+------+------> '- -force -'

>--+- fc_map_id ---+------>< '- fc_map_name -'

After we recreated the second mapping, this time using the flag -incremental, we view the properties of that mapping using the command svcinfo lsfcmap 1, as shown in Example 6-12.

Example 6-12 Properties of the mapping with ID 1 IBM_2145:ITSOCL3:admin>svcinfo lsfcmap 1 id 1 name fcm-back-002 source_vdisk_id 0 source_vdisk_name vd-prod-l2-0002 target_vdisk_id 3 target_vdisk_name vd-back-l2-0002 group_id group_name status idle_or_copied progress 0 copy_rate 70 start_time dependent_mappings 0 autodelete off clean_progress 100 clean_rate 50

Chapter 6. Implementing FlashCopy 169 incremental on difference 100 grain_size 256 IO_group_id 0 IO_group_name io_grp0 IBM_2145:ITSOCL3:admin>

Using svcinfo lsvdisk again, as shown in Example 6-13, we see that all four VDisks are part of FlashCopy mappings.

Example 6-13 All four VDisks are part of FlashCopy mappings IBM_2145:ITSOCL3:admin>svcinfo lsvdisk -delim : id:name:IO_group_id:IO_group_name:status:mdisk_grp_id:mdisk_grp_name:capacity:type :FC_id:FC_name:RC_id:RC_name:vdisk_UID:fc_map_count 0:vd-prod-l2-0002:0:io_grp0:online:1:DS4700:25.0GB:striped:1:fcm-back-002:0:MetroM irror Relationship-002:6005076801A100E90800000000000003:1 1:vd-prod-l2-0001:0:io_grp0:online:1:DS4700:25.0GB:striped:0:fcm-back-001:1:MetroM irror Relationship-001:6005076801A100E90800000000000002:1 2:vd-back-l2-0001:0:io_grp0:online:1:DS4700:25.0GB:striped:0:fcm-back-001:::600507 6801A100E90800000000000004:1 3:vd-back-l2-0002:0:io_grp0:online:1:DS4700:25.0GB:striped:1:fcm-back-002:::600507 6801A100E90800000000000005:1 IBM_2145:ITSOCL3:admin>

6.4.3 Creating a consistency group

At this point, we create a FlashCopy consistency group in which to put the mappings. The command to be used is svctask mkfcconsistgrp, and the syntax is shown in Example 6-14.

Example 6-14 svctask mkfcconsistgrp - syntax >>- svctask -- -- mkfcconsistgrp ------>

>--+------+------>< '- -name -- consist_group_name -'

The actual command we used to create our consistency group is shown in Example 6-15.

Example 6-15 Consistency group successfully created IBM_2145:ITSOCL3:admin>svctask mkfcconsistgrp -name fcg-back-001 FlashCopy consistency group, id [1], successfully created IBM_2145:ITSOCL3:admin>

Example 6-16 shows the syntax of the command svcinfo lsfcconsistgrp, which displays the created consistency groups.

Example 6-16 svcinfo lsfcconsistgrp - syntax >>- svcinfo -- -- lsfcconsistgrp ------>

>--+------+-- --+------+------> '- -filtervalue -- attribute=value -' '- -nohdr -'

>--+------+-- --+------+------>

170 SVC 4.2.1 Advanced Copy Services '- -delim -- delimiter -' +- object_id ---+ '- object_name -'

>--+------+------>< '- -filtervalue? -'

We use the svcinfo lsfcconsistgrp command to verify the creation of the consistency group we created, as shown in Example 6-17.

Example 6-17 Empty consistency group IBM_2145:ITSOCL3:admin>svcinfo lsfcconsistgrp id name status 1 fcg-back-001 empty IBM_2145:ITSOCL3:admin>

At this point, we have two FlashCopy mappings and one empty FlashCopy consistency group.

6.4.4 Adding mappings to consistency group

Before we can start the FlashCopy, we include the two mappings in the consistency group we just created. We do so by issuing the command svctask chfcmap. The syntax is shown in Example 6-18.

Example 6-18 svctask chfcmap - syntax >>- svctask -- -- chfcmap -- --+------+-- ---> '- -name -- new_name_arg -'

>--+------+------> '- -force -'

>--+------+------> '- -consistgrp --+- consist_group_id ---+-' '- consist_group_name -'

>--+------+-- --+------+---> '- -copyrate -- percent-' '- -autodelete --+-on--+-' '-off-'

>--+------+--+- fc_map_id ---+------>< '- -cleanrate ---- percent ---' '- fc_map_name -'

We put the mappings into the consistency group using the command shown in Example 6-19.

Example 6-19 Associating both mappings with the same consistency group IBM_2145:ITSOCL3:admin>svctask chfcmap -consistgrp fcg-back-001 fcm-back-001 IBM_2145:ITSOCL3:admin>svctask chfcmap -consistgrp fcg-back-001 fcm-back-002 IBM_2145:ITSOCL3:admin>

Chapter 6. Implementing FlashCopy 171 To view the consistency group, we issue the command svcinfo lsfcconsistgrp again, shown in Example 6-20.

Example 6-20 Consistency group in Idle_or_Copied state IBM_2145:ITSOCL3:admin>svcinfo lsfcconsistgrp id name status 1 fcg-back-001 idle_or_copied IBM_2145:ITSOCL3:admin>

The consistency group state changed from Empty to Idle_or_Copied. The properties, including the FlashCopy mappings that are now associated with the consistency group, we get when specifying the ID of the group with the command shown in Example 6-21.

Example 6-21 Properties of consistency group with ID 1 IBM_2145:ITSOCL3:admin>svcinfo lsfcconsistgrp 1 id 1 name fcg-back-001 status idle_or_copied FC_mapping_id 0 FC_mapping_name fcm-back-001 FC_mapping_id 1 FC_mapping_name fcm-back-002 IBM_2145:ITSOCL3:admin>

Example 6-22 shows the two FlashCopy mappings, using the command svcinfo lsfcmap. We see that both mappings are members of the same consistency group.

Example 6-22 Both mappings associated with same consistency group IBM_2145:ITSOCL3:admin>svcinfo lsfcmap -delim : id:name:source_vdisk_id:source_vdisk_name:target_vdisk_id:target_vdisk_name:group_ id:group_name:status:progress:copy_rate:clean_progress:incremental 0:fcm-back-001:1:vd-prod-l2-0001:2:vd-back-l2-0001:1:fcg-back-001:idle_or_copied:0 :70:100:on 1:fcm-back-002:0:vd-prod-l2-0002:3:vd-back-l2-0002:1:fcg-back-001:idle_or_copied:0 :70:100:on IBM_2145:ITSOCL3:admin>

6.4.5 Preparing a mapping

We want to prepare the FlashCopy consistency group (and thus the FlashCopy mappings associated with it) prior to issuing the start of the group. We issue the command svctask prestartfcconsistgrp, and the syntax is shown in Example 6-23.

Example 6-23 .svctask prestartfcconsistgrp - syntax >>- svctask -- -- prestartfcconsistgrp ------>

>--+- fc_consist_group_id ---+------>< '- fc_consist_group_name -'

172 SVC 4.2.1 Advanced Copy Services We prepare the group, and then we immediately issue the command to verify that the state of the consistency group is Prepared, which is shown in Example 6-24.

Example 6-24 Prepare and view group IBM_2145:ITSOCL3:admin>svctask prestartfcconsistgrp fcg-back-001 IBM_2145:ITSOCL3:admin>svcinfo lsfcconsistgrp id name status 1 fcg-back-001 prepared IBM_2145:ITSOCL3:admin>

This time we use the consistency group name (not the ID) to specify which group to start. We can also issue the command again to see the state of the mappings, as shown in Example 6-25.

Example 6-25 Both mappings in Prepared state IBM_2145:ITSOCL3:admin>svcinfo lsfcmap -delim : id:name:source_vdisk_id:source_vdisk_name:target_vdisk_id:target_vdisk_name:group_ id:group_name:status:progress:copy_rate:clean_progress:incremental 0:fcm-back-001:1:vd-prod-l2-0001:2:vd-back-l2-0001:1:fcg-back-001:prepared:0:70:10 0:on 1:fcm-back-002:0:vd-prod-l2-0002:3:vd-back-l2-0002:1:fcg-back-001:prepared:0:70:10 0:on IBM_2145:ITSOCL3:admin>

6.4.6 Starting a consistency group

Example 6-26 shows the syntax of the command svctask startfcconsistgrp.

Example 6-26 svctask startfcconsistgrp - syntax >>- svctask -- -- startfcconsistgrp -- --+------+------> '- -prep -'

>--+- fc_consist_group_id ---+------>< '- fc_consist_group_name -'

Note: We could prepare the consistency group first using the -prep option, but because we have already prepared the group we do not need to do it now.

Example 6-27 shows how we start the consistency group (using the ID to specify it), and immediately after starting the consistency group, we view its status.

Example 6-27 Start and view consistency group IBM_2145:ITSOCL3:admin>svctask startfcconsistgrp 1 IBM_2145:ITSOCL3:admin>svcinfo lsfcconsistgrp id name status 1 fcg-back-001 copying IBM_2145:ITSOCL3:admin>

Chapter 6. Implementing FlashCopy 173 At this point, the state of the consistency group is Copying, as expected. In Example 6-28, we see the state of both of the mappings is Copying as well.

Example 6-28 Both mappings in Copying state IBM_2145:ITSOCL3:admin>svcinfo lsfcmap -delim : id:name:source_vdisk_id:source_vdisk_name:target_vdisk_id:target_vdisk_name:group_ id:group_name:status:progress:copy_rate:clean_progress:incremental 0:fcm-back-001:1:vd-prod-l2-0001:2:vd-back-l2-0001:1:fcg-back-001:copying:34:70:10 0:on 1:fcm-back-002:0:vd-prod-l2-0002:3:vd-back-l2-0002:1:fcg-back-001:copying:34:70:10 0:on IBM_2145:ITSOCL3:admin>

Example 6-29 shows the details of the FlashCopy mapping with the ID 0.

Example 6-29 svcinfo lsfcmap -delim : 0 IBM_2145:ITSOCL3:admin>svcinfo lsfcmap -delim : 0 id:0 name:fcm-back-001 source_vdisk_id:1 source_vdisk_name:vd-prod-l2-0001 target_vdisk_id:2 target_vdisk_name:vd-back-l2-0001 group_id:1 group_name:fcg-back-001 status:copying progress:51 copy_rate:70 start_time:071107011320 dependent_mappings:0 autodelete:off clean_progress:100 clean_rate:50 incremental:on difference:48 grain_size:256 IO_group_id:0 IO_group_name:io_grp0 IBM_2145:ITSOCL3:admin>

We see that the state of the mapping is Copying and that the progress is 51%, using a background copy priority of 70%.

The last output we view is the details of one of the VDisks, which is shown in Example 6-30.

Example 6-30 svcinfo lsvdisk -delim : 1 IBM_2145:ITSOCL3:admin>svcinfo lsvdisk -delim : 1 id:1 name:vd-prod-l2-0001 IO_group_id:0 IO_group_name:io_grp0 status:online mdisk_grp_id:1 mdisk_grp_name:DS4700

174 SVC 4.2.1 Advanced Copy Services capacity:25.0GB type:striped formatted:no mdisk_id: mdisk_name: FC_id:0 FC_name:fcm-back-001 RC_id:1 RC_name:MetroMirror Relationship-001 vdisk_UID:6005076801A100E90800000000000002 throttling:0 preferred_node_id:2 fast_write_state:empty cache:readwrite udid: fc_map_count:1 IBM_2145:ITSOCL3:admin>

VDisk vd-prod-l2-0001 is a member of both a Remote Copy Relationship and a FlashCopy mapping.

After the background copy process is finished copying the data to the target VDisk, the state changes to Idle_or_Copied, as shown in Example 6-31.

Example 6-31 Idle_or_Copied state IBM_2145:ITSOCL3:admin>svcinfo lsfcconsistgrp 1 id 1 name fcg-back-001 status idle_or_copied FC_mapping_id 0 FC_mapping_name fcm-back-001 FC_mapping_id 1 FC_mapping_name fcm-back-002 IBM_2145:ITSOCL3:admin>

We can also review the details of one of the mappings, as shown in Example 6-32.

Example 6-32 svcinfo lsfcmap -delim : 0 IBM_2145:ITSOCL3:admin>svcinfo lsfcmap -delim : 0 id:0 name:fcm-back-001 source_vdisk_id:1 source_vdisk_name:vd-prod-l2-0001 target_vdisk_id:2 target_vdisk_name:vd-back-l2-0001 group_id:1 group_name:fcg-back-001 status:idle_or_copied progress:100 copy_rate:70 start_time:071107011320 dependent_mappings:0 autodelete:off clean_progress:100

Chapter 6. Implementing FlashCopy 175 clean_rate:50 incremental:on difference:0 grain_size:256 IO_group_id:0 IO_group_name:io_grp0 IBM_2145:ITSOCL3:admin>

Because we set up the mappings as incremental, we can prepare and start the consistency group again to refresh it, and only the grains that have changed since the last FlashCopy process are copied. This puts less load on the SVC and makes the full clone of the VDisks available faster.

6.4.7 Modifying a mapping

To modify a mapping, we issue the command svctask chfcmap, which uses the syntax shown in Example 6-33.

Example 6-33 svctask chfcmap - syntax >>- svctask -- -- chfcmap -- --+------+-- ---> '- -name -- new_name_arg -'

>--+------+------> '- -force -'

>--+------+------> '- -consistgrp --+- consist_group_id ---+-' '- consist_group_name -'

>--+------+-- --+------+---> '- -copyrate -- percent-' '- -autodelete --+-on--+-' '-off-'

>--+------+--+- fc_map_id ---+------>< '- -cleanrate ---- percent ---' '- fc_map_name -'

We want to change both mappings to a lower background copy priority. Example 6-34 shows how we change the background copy priority to 50% and verify the settings for the mapping with the ID 0. We do the same with the second mapping.

Example 6-34 Change the copy priority to 50% IBM_2145:ITSOCL3:admin>svctask chfcmap -copyrate 50 0 IBM_2145:ITSOCL3:admin>svcinfo lsfcmap 0 id 0 name fcm-back-001 source_vdisk_id 1 source_vdisk_name vd-prod-l2-0001 target_vdisk_id 2 target_vdisk_name vd-back-l2-0001 group_id 1 group_name fcg-back-001 status idle_or_copied progress 100

176 SVC 4.2.1 Advanced Copy Services copy_rate 50 start_time 071107021634 dependent_mappings 0 autodelete off clean_progress 100 clean_rate 50 incremental on difference 0 grain_size 256 IO_group_id 0 IO_group_name io_grp0 IBM_2145:ITSOCL3:admin>

6.4.8 Stopping a mapping and a consistency group

Example 6-35 shows the syntax of the command svctask stopfcmap, which is used to stop a FlashCopy mapping.

Example 6-35 svctask stopfcmap - syntax >>- svctask -- -- stopfcmap -- --+------+------> '- -force-'

>--+- fc_map_id ---+------>< '- fc_map_name -'

We can stop the consistency group by issuing the command svctask stopfcconsistgrp, for which the syntax is shown in Example 6-36.

Example 6-36 svctask stopfcconsistgrp - syntax >>- svctask -- -- stopfcconsistgrp -- --+------+------> '- -force-'

>--+- fc_consist_group_id --+------>< '- fc_consist_group_name -'

6.4.9 Deleting a mapping and a consistency group

Example 6-37 shows the syntax of the command svctask rmfcmap, which is used to delete a FlashCopy mapping.

Example 6-37 svctask rmfcmap - syntax >>- svctask -- -- rmfcmap -- --+------+------> '- -force -'

>--+- fc_map_id ---+------>< '- fc_map_name -'

Chapter 6. Implementing FlashCopy 177 Example 6-38 shows the syntax of the command svctask rmfcconsistgrp, used to delete a FlashCopy consistency group.

Example 6-38 svctask rmfcconsistgrp - syntax >>- svctask -- -- rmfcconsistgrp -- --+------+------> '- -force -'

>--+- fc_consist_group_id ---+------>< '- fc_consist_group_name -'

178 SVC 4.2.1 Advanced Copy Services 7

Chapter 7. Server integration

In this chapter, we discuss the steps that you take to access copied VDisks from servers running various operating systems. In addition, we discuss the steps you must take to ensure that the copies that are generated can be easily accessed when needed.

As far as servers are concerned, there is no difference between a VDisk that was copied using FlashCopy and one that was copied using Metro/Global Mirror. As a result, this chapter focuses on accessing the VDisks regardless of how they were created.

When a VDisk is copied using SVC’s copy services, a full block-by-block replica is created. This means that everything on the source or primary VDisk is copied to the target or secondary VDisk. On many systems, not only application data is copied. Many systems write configuration data to the disks to enable volume managers to uniquely identify them and determine their place in volume groups.

This is can be an advantage or a disadvantage. When presenting the copied VDisks to a brand new server, it is much simpler to bring the volumes back online. However, when presenting these VDisks to the same server that accesses the original VDisks, there are challenges involved, because the metadata on the VDisks is expected to be unique and presenting the copied VDisk breaches that requirement.

You cannot selectively copy data from one VDisk to another; all data is copied. This chapter discusses techniques for handling this requirement.

7.2 Data on copied VDisks

As discussed in Chapter 2, “Planning for Advanced Copy Services implementation” on page 5, the data on the original VDisk may not match what your applications and file systems expect. This is because of the caches between that layer and the SVC layer.

It is important to ensure one of the following: Data in the caches above the SVC are fully flushed before initiating the copy. The systems that access the copied VDisks can recover from the fact that the data is not an identical copy of the system’s view.

The second option can be achieved by using journaled file systems and having applications that protect their actions with a transaction log.

A critical moment in a copy services mapping relationship’s life cycle affects the contents of the target or secondary VDisk. After this critical moment, the readable contents of the VDisk are no longer updated. This critical moment depends on which copy service is in use: For FlashCopy, the critical moment is when the mapping is started. For Metro/Global Mirror, the critical moment is when the relationship is stopped (either manually or due to errors).

In this chapter, the action is referred to as freezing, and the critical moment is referred to as the freeze point.

At the freeze point, the target or secondary VDisk is an image of the source or primary VDisk at that moment in time (plus or minus a few seconds for Global Mirror). Any data in the caches above SVC will not be on the target or secondary VDisk.

In the case of manual actions, it is relatively straightforward to ensure the target or secondary VDisk is a useful copy. For Metro/Global Mirror relationships that are stopped due to SAN errors, it is not possible to ensure this, so it is important that the systems accessing these VDisks are configured to allow recovery.

Some data is not copied. Persistent reserves are made against a specific logical unit and are not copied with FlashCopy or Metro/Global Mirror.

180 SVC 4.2.1 Advanced Copy Services 7.3 AIX specifics

In this section, we describe the steps for creating and accessing copied VDisks on AIX® hosts.

SVC supports the use of SDD and MPIO+SDDPCM to provide multipathing. When SDD is in use, AIX takes hdisks and creates vpaths. When SDDPCM is in use, AIX creates hdisks, and each one has multiple paths.

For the sake of clarity, the following section assumes the use of SDD and thus discusses vpaths. The methods here apply just as well with SDDPCM; however, the term hdisk should replace vpath in our discussion.

7.3.1 Creating useful copies

Prior to the freeze point, steps must be taken to ensure that any information in the file system cache is written to all of the relevant VDisks. The simplest way to do so is to unmount the file system, as in Example 7-1.

Example 7-1 Unmount a file system prior to making the break #umount

If unmounting file systems is not an option, an alternative is to force an update of the node table and a flush of buffered files. The AIX freeze and thaw functionality does precisely this. It is available only on JFS2 (Journaled File System 2) file systems. Freezing a file system writes all dirty file system metadata and user data to the disk. While frozen, the file system is read-only, and anything that attempts to modify the file system or its contents must wait for the freeze to end. At this point, the break can be made, and the resulting copy can be easily picked up by another server. Example 7-2 shows how this is done.

Example 7-2 A simple freeze and thaw process #chfs -a freeze=60

#chfs -a freeze=0

Of course, sometimes the freeze point occurs at an unplanned time, such as during a data center outage. So long as the mappings are consistent, your file systems should be recoverable, and in the case of a journaled file system, this recovery should not result in any data loss.

Example 7-3 shows the initial configuration of the SVC cluster presenting VDisks to this AIX server. The two VDisks represent a data drive and a log drive.

Example 7-3 Cluster configuration prior to FlashCopying two VDisks IBM_2145:ITSOCL2:admin>svcinfo lsvdisk -delim : id:name:IO_group_id:IO_group_name:status:mdisk_grp_id:mdisk_grp_name:capacity:type :FC_id:FC_name:RC_id:RC_name:vdisk_UID:fc_map_count 5:kanagaFS:0:io_grp0:online:1:DS4700:36.0GB:striped:::::6005076801AD80E8E000000000 000005:0

Chapter 7. Server integration 181 6:kanagaLog:0:io_grp0:online:0:DS4500:512.0MB:striped:::::6005076801AD80E8E0000000 00000006:0

IBM_2145:ITSOCL2:admin>svcinfo lshost KANAGA id 2 name KANAGA port_count 2 type generic mask 1111 iogrp_count 4 WWPN 10000000C932A7FB node_logged_in_count 2 state active WWPN 10000000C932A800 node_logged_in_count 0 state active

Example 7-4 shows how these VDisks have been configured on an AIX server to hold a JFS2 file system. The vdisk_UID property of a VDisk matches the SERIAL property of a vpath. Thus you can use these two properties to determine which vpath represents which VDisk.

Example 7-4 AIX configuration prior to FlashCopying two VDisks #datapath query device

Total Devices : 2

DEV#: 0 DEVICE NAME: vpath0 TYPE: 2145 POLICY: Optimized SERIAL: 6005076801AD80E8E000000000000005 ======Path# Adapter/Hard Disk State Mode Select Errors 0 fscsi0/hdisk3 OPEN NORMAL 209 0 1 fscsi0/hdisk5 OPEN NORMAL 220 0 2 fscsi0/hdisk7 OPEN NORMAL 0 0 3 fscsi0/hdisk9 OPEN NORMAL 0 0

DEV#: 1 DEVICE NAME: vpath1 TYPE: 2145 POLICY: Optimized SERIAL: 6005076801AD80E8E000000000000006 ======Path# Adapter/Hard Disk State Mode Select Errors 0 fscsi0/hdisk4 OPEN NORMAL 0 0 1 fscsi0/hdisk6 OPEN NORMAL 0 0 2 fscsi0/hdisk8 OPEN NORMAL 74 0 3 fscsi0/hdisk10 OPEN NORMAL 79 0

182 SVC 4.2.1 Advanced Copy Services hdisk9 none None hdisk10 none None vpath0 0009cdda1b5058e4 st7s99 active vpath1 0009cdda1b55c48c st7s99 active

#lsvg -l st7s99 st7s99: LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT itsoLV jfs2 32 32 1 open/syncd /itsoFS itsoLog jfs2log 4 4 1 open/syncd N/A

#lsfs Name Nodename Mount Pt VFS Size Options Auto Accounting /dev/hd4 -- / jfs2 196608 -- yes no /dev/hd1 -- /home jfs2 65536 -- yes no /dev/hd2 -- /usr jfs2 19005440 -- yes no /dev/hd9var -- /var jfs2 65536 -- yes no /dev/hd3 -- /tmp jfs2 393216 -- yes no /proc -- /proc procfs -- -- yes no /dev/hd10opt -- /opt jfs2 2031616 -- yes no /dev/itsoLV -- /itsoFS jfs2 4194304 rw no no

#mount node mounted mounted over vfs date options ------/dev/hd4 / jfs2 Nov 07 09:55 rw,log=/dev/hd8 /dev/hd2 /usr jfs2 Nov 07 09:55 rw,log=/dev/hd8 /dev/hd9var /var jfs2 Nov 07 09:55 rw,log=/dev/hd8 /dev/hd3 /tmp jfs2 Nov 07 09:55 rw,log=/dev/hd8 /dev/hd1 /home jfs2 Nov 07 09:56 rw,log=/dev/hd8 /proc /proc procfs Nov 07 09:56 rw /dev/hd10opt /opt jfs2 Nov 07 09:56 rw,log=/dev/hd8 /dev/itsoLV /itsoFS jfs2 Nov 07 11:17 rw,log=/dev/itsoLog

The Volume Group (VG) that is being copied is st7s99, containing vpath0 and vpath1.

Is this is a JFS2 file system, we can freeze the file system prior to starting the FlashCopy. Write requests on the applications that access this file system are delayed until the file system is thawed. With appropriate planning and scripting, this freeze can be kept to a few seconds at most. 1. Create the FlashCopy target VDisks. IBM_2145:ITSOCL2:admin>svctask mkvdisk -vtype striped -size 36 -unit gb -mdiskgrp 1 -iogrp 0 -name kanagaFS-FC Virtual Disk, id [4], successfully created

IBM_2145:ITSOCL2:admin>svctask mkvdisk -vtype striped -size 512 -unit mb -mdiskgrp 0 -iogrp 0 -name kanagaLog-FC Virtual Disk, id [3], successfully created 2. Create a FlashCopy consistency group. IBM_2145:ITSOCL2:admin>svctask mkfcconsistgrp -name kanagaFC FlashCopy Consistency Group, id [2], successfully created

Chapter 7. Server integration 183 3. Create FlashCopy mappings for the VDisks and add them to the new FlashCopy consistency group. IBM_2145:ITSOCL2:admin>svctask mkfcmap -source kanagaFS -target kanagaFS-FC -consistgrp kanagaFC FlashCopy Mapping, id [0], successfully created

IBM_2145:ITSOCL2:admin>svctask mkfcmap -source kanagaLog -target kanagaLog-FC -consistgrp kanagaFC FlashCopy Mapping, id [1], successfully created 4. Prepare the FlashCopy consistency group. IBM_2145:ITSOCL2:admin>svctask prestartfcconsistgrp kanagaFC 5. Wait until the consistency group is in the Prepared state. IBM_2145:ITSOCL2:admin>svcinfo lsfcconsistgrp id name status 2 kanagaFC preparing . . . IBM_2145:ITSOCL2:admin>svcinfo lsfcconsistgrp id name status 2 kanagaFC prepared 6. Freeze the file system on the AIX server. #chfs -a freeze=60 /itsoFS 7. Start the FlashCopy consistency group. This is the freeze point. IBM_2145:ITSOCL2:admin>svctask startfcconsistgrp kanagaFC 8. Thaw the file system on the AIX server. #chfs -a freeze=0 /itsoFS

At this point, the FlashCopy operation has been started, and VDisks 3 and 4 are exact copies of VDisks 5 and 6 at the time of the file system freeze.

7.3.2 Accessing useful copies

If the source or primary VDisk is defined to the AIX logical volume manager (LVM), all of its data structures and identifiers are copied to the target or secondary VDisk as well. This includes the Volume Group Descriptor Area (VGDA), which contains the physical volume identifier (PVID) and volume group identifier (VGID).

For AIX LVM, it is currently not possible to activate a volume group with a physical volume (vpath) containing a VGID and a PVID that are already used in a volume group existing on the same server. The restriction still applies even if the vpath PVID is cleared and reassigned with the two commands listed in Example 7-5.

Example 7-5 Clearing PVIDs chdev -l -a pv=clear chdev -l -a pv=yes

Therefore, it is necessary to redefine the volume group information about the FlashCopy target VDisk using special procedures or the recreatevg command. This alters the PVIDs and VGIDs in all the VGDAs of the FlashCopy target VDisks, so that no conflicts occur with

184 SVC 4.2.1 Advanced Copy Services existing PVIDs and VGIDs on existing volume groups residing on the source VDisks. If you do not redefine the volume group information prior to importing the volume group, the importvg command fails.

Accessing target or secondary VDisks from the same AIX host In this section, we describe a method of accessing the FlashCopy target VDisks on a single AIX host while the source VDisks are still active on the same server. The procedure is intended to be used as a guide and may not cover all scenarios.

AIX recreatevg command Copying source volume content using FlashCopy duplicates all of the data structures and identifiers used by AIX LVM to the target volume. The duplicate definitions (PVID and VGID) cause conflicts within LVM, but this problem is solved by issuing the AIX command recreatevg. The recreatevg command is officially available in all levels of AIX that SVC supports

The recreatevg command overcomes the problem of duplicated LVM data structures and identifiers caused by a disk duplication process such as FlashCopy. It is used to recreate an AIX volume group (VG) on a set of target volumes that are copied from a set of source volumes belonging to a specific VG. The command allocates new physical volume identifiers (PVIDs) for the member disks and a new VGID to the volume group. The command also provides options for renaming the logical volumes with a prefix you specify, and options for renaming labels to specify different mount points for file systems. To use this command, you must have root user authority.

Attention: Running recreatevg changes the PVID on VDisks. In the case of a Metro/Global Mirror relationship, if you decide to reverse the direction of the relationship, you will change the PVID of the original primary VDisk during the synchronization process.

The following steps assume that the AIX server and SVC cluster are still in the same state as at the end of the process described previously.

Perform these tasks to access the FlashCopy target VDisks: 1. Map the FlashCopy target VDisks to the host. IBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host KANAGA kanagaFS-FC Virtual Disk to Host map, id [2], successfully created IBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host KANAGA kanagaLog-FC Virtual Disk to Host map, id [3], successfully created 2. Run Configuration Manager to pick up the new devices on the AIX server #cfgmgr -vl fscsi0 (We skipped the debug output from this command.) 3. Run Configuration Manager to create vpaths with the new hdisks. #cfgmgr -vl dpo

Chapter 7. Server integration 185 At this point, the configuration of the AIX server has been changed. Example 7-6 shows the new status.

Example 7-6 AIX configuration prior to FlashCopying two VDisks #datapath query device

Total Devices : 4

DEV#: 0 DEVICE NAME: vpath0 TYPE: 2145 POLICY: Optimized SERIAL: 6005076801AD80E8E000000000000005 ======Path# Adapter/Hard Disk State Mode Select Errors 0 fscsi0/hdisk3 OPEN NORMAL 214 0 1 fscsi0/hdisk5 OPEN NORMAL 232 0 2 fscsi0/hdisk7 OPEN NORMAL 0 0 3 fscsi0/hdisk9 OPEN NORMAL 0 0

DEV#: 2 DEVICE NAME: vpath2 TYPE: 2145 POLICY: Optimized SERIAL: 6005076801AD80E8E000000000000007 ======Path# Adapter/Hard Disk State Mode Select Errors 0 fscsi0/hdisk11 CLOSE NORMAL 0 0 1 fscsi0/hdisk13 CLOSE NORMAL 0 0 2 fscsi0/hdisk15 CLOSE NORMAL 0 0 3 fscsi0/hdisk17 CLOSE NORMAL 0 0

DEV#: 3 DEVICE NAME: vpath3 TYPE: 2145 POLICY: Optimized SERIAL: 6005076801AD80E8E000000000000009 ======Path# Adapter/Hard Disk State Mode Select Errors 0 fscsi0/hdisk12 CLOSE NORMAL 0 0 1 fscsi0/hdisk14 CLOSE NORMAL 0 0 2 fscsi0/hdisk16 CLOSE NORMAL 0 0 3 fscsi0/hdisk18 CLOSE NORMAL 0 0

186 SVC 4.2.1 Advanced Copy Services hdisk9 none None hdisk10 none None vpath0 0009cdda1b5058e4 st7s99 active vpath1 0009cdda1b55c48c st7s99 active hdisk11 0009cdda1b5058e4 st7s99 active hdisk12 0009cdda1b55c48c st7s99 active hdisk13 0009cdda1b5058e4 st7s99 active hdisk14 0009cdda1b55c48c st7s99 active hdisk15 0009cdda1b5058e4 st7s99 active hdisk16 0009cdda1b55c48c st7s99 active hdisk17 0009cdda1b5058e4 st7s99 active hdisk18 0009cdda1b55c48c st7s99 active vpath2 none None vpath3 none None

The new vpaths, vpath2 and vpath3, appear with no PVID, whereas the hdisks that make up those vpaths do. This is as expected. Note also that vpath2 and vpath3 are in a Close state because nothing is currently accessing them. The following steps recreate the VG with a new name and recover the logical volumes (LV). 1. Create the new VG and prefix all file system path names with /fc and prefix all AIX LVs with fc: #recreatevg -y st7s99fc -L /fc -Y FC vpath2 vpath3 st7s99fc

Restriction: The LV prefix cannot be “fc”. The reason is because “fc” is already defined as a prefix in the PdDv class of the device configuration database and so is not a valid LV prefix.

2. Run fsck on the new file system. Strictly speaking, it is not necessary to run fsck because the steps taken while creating the copy ensure that the file system is client, but it is a prudent defensive step. #fsck -y /fc/itsoFS

The current volume is: /dev/FCitsoLV Primary superblock is valid. J2_LOGREDO:log redo processing for /dev/FCitsoLV Primary superblock is valid. *** Phase 1 - Initial inode scan *** Phase 2 - Process remaining directories *** Phase 3 - Process remaining files *** Phase 4 - Check and repair inode allocation map *** Phase 5 - Check and repair block allocation map File system is clean. 3. Mount the new file system. #mount /fc/itsoFS Replaying log for /dev/FCitsoLV.

Chapter 7. Server integration 187 At this point, the FlashCopy target VDisks are ready for access. Example 7-7 shows the final server state. This new file system contains all of the files that were in the original file system at the time of the freeze.

Example 7-7 AIX configuration after recreating the VG #datapath query device

Total Devices : 4

DEV#: 0 DEVICE NAME: vpath0 TYPE: 2145 POLICY: Optimized SERIAL: 6005076801AD80E8E000000000000005 ======Path# Adapter/Hard Disk State Mode Select Errors 0 fscsi0/hdisk3 OPEN NORMAL 217 0 1 fscsi0/hdisk5 OPEN NORMAL 237 0 2 fscsi0/hdisk7 OPEN NORMAL 0 0 3 fscsi0/hdisk9 OPEN NORMAL 0 0

DEV#: 2 DEVICE NAME: vpath2 TYPE: 2145 POLICY: Optimized SERIAL: 6005076801AD80E8E000000000000007 ======Path# Adapter/Hard Disk State Mode Select Errors 0 fscsi0/hdisk11 OPEN NORMAL 202 0 1 fscsi0/hdisk13 OPEN NORMAL 206 0 2 fscsi0/hdisk15 OPEN NORMAL 0 0 3 fscsi0/hdisk17 OPEN NORMAL 0 0

DEV#: 3 DEVICE NAME: vpath3 TYPE: 2145 POLICY: Optimized SERIAL: 6005076801AD80E8E000000000000009 ======Path# Adapter/Hard Disk State Mode Select Errors 0 fscsi0/hdisk12 OPEN NORMAL 0 0 1 fscsi0/hdisk14 OPEN NORMAL 0 0 2 fscsi0/hdisk16 OPEN NORMAL 61 0 3 fscsi0/hdisk18 OPEN NORMAL 57 0

188 SVC 4.2.1 Advanced Copy Services hdisk8 none None hdisk9 none None hdisk10 none None vpath0 0009cdda1b5058e4 st7s99 active vpath1 0009cdda1b55c48c st7s99 active hdisk11 none None hdisk12 none None hdisk13 none None hdisk14 none None hdisk15 none None hdisk16 none None hdisk17 none None hdisk18 none None vpath2 0009cdda1bf46d57 st7s99fc active vpath3 0009cdda1bf46f3b st7s99fc active

#lsvg rootvg st7s99 st7s99fc

#lsvg -l st7s99 st7s99: LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT itsoLV jfs2 32 32 1 open/syncd /itsoFS itsoLog jfs2log 4 4 1 open/syncd N/A

#lsvg -l st7s99fc st7299fc: LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT FCitsoLV jfs2 32 32 1 open/syncd /fc/itsoFS FCitsoLog jfs2log 4 4 1 open/syncd N/A

Chapter 7. Server integration 189 /dev/hd10opt /opt jfs2 Nov 07 09:56 rw,log=/dev/hd8 /dev/itsoLV /itsoFS jfs2 Nov 07 11:17 rw,log=/dev/itsoLog /dev/FCitsoLV /fc/itsoFS jfs2 Nov 07 13:15 rw,log=/dev/FCitsoLog

Accessing target or secondary VDisks from a different AIX host Accessing target or secondary VDisks from a different server from the one accessing the source or primary VDisks is much simpler.

The following procedure makes the data of the FlashCopy target VDisk available to another AIX host that has no prior definitions of the target VDisk in its configuration database (ODM). Example 7-8 shows the state of the AIX server prior to presenting the target VDisks.

Example 7-8 AIX configuration prior to presenting FlashCopy target VDisks #datapath query device

No device file found

#lsvg rootvg

#lspv hdisk0 0009cddaea97bf61 rootvg active hdisk1 0009cdda43c9dfd5 rootvg active hdisk2 0009cddabaef1d99 rootvg active

The initial process for accessing the VDisks is as before. 1. Map the FlashCopy target VDisks to the host. IBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host UMNAK kanagaFS-FC Virtual Disk to Host map, id [2], successfully created IBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host UMNAK kanagaLog-FC Virtual Disk to Host map, id [3], successfully created 2. Run Configuration Manager to pick up the new devices on the AIX server. #cfgmgr -vl fscsi0 (We skipped the debug output from this command.) 3. Run Configuration Manager to create vpaths with the new hdisks. #cfgmgr -vl dpo

190 SVC 4.2.1 Advanced Copy Services At this point, the configuration of the AIX server has been changed. Example 7-9 shows the new status.

Example 7-9 AIX configuration after presenting FlashCopy target VDisks datapath query device

Total Devices : 2

DEV#: 0 DEVICE NAME: vpath2 TYPE: 2145 POLICY: Optimized SERIAL: 6005076801AD80E8E000000000000007 ======Path# Adapter/Hard Disk State Mode Select Errors 0 fscsi0/hdisk3 CLOSE NORMAL 0 0 1 fscsi0/hdisk5 CLOSE NORMAL 0 0 2 fscsi0/hdisk7 CLOSE NORMAL 0 0 3 fscsi0/hdisk9 CLOSE NORMAL 0 0

DEV#: 1 DEVICE NAME: vpath3 TYPE: 2145 POLICY: Optimized SERIAL: 6005076801AD80E8E000000000000009 ======Path# Adapter/Hard Disk State Mode Select Errors 0 fscsi0/hdisk4 CLOSE NORMAL 0 0 1 fscsi0/hdisk6 CLOSE NORMAL 0 0 2 fscsi0/hdisk8 CLOSE NORMAL 0 0 3 fscsi0/hdisk10 CLOSE NORMAL 0 0

#lspv hdisk0 0009cddaea97bf61 rootvg active hdisk1 0009cdda43c9dfd5 rootvg active hdisk2 0009cddabaef1d99 rootvg active hdisk3 none None hdisk4 none None hdisk5 none None hdisk6 none None hdisk7 none None hdisk8 none None hdisk9 none None hdisk10 none None vpath2 0009cdda1b5058e4 st7s99 vpath3 0009cdda1b55c48c st7s99

At this time, note that the new vpaths have PVIDs, and the hdisks have none. We use the importvg command to import the information about the VG into this server’s ODM: 1. Import the target volume group: #importvg -y st7s99 vpath2 st7s99 2. Vary on the Volume Group (the importvg command should vary on the volume group): #varyonvg st7s99

Chapter 7. Server integration 191 3. Verify consistency of all file systems on the FlashCopy target VDisk: #fsck -y /itsoFS

The current volume is: /dev/itsoLV Primary superblock is valid. J2_LOGREDO:log redo processing for /dev/itsoLV Primary superblock is valid. *** Phase 1 - Initial inode scan *** Phase 2 - Process remaining directories *** Phase 3 - Process remaining files *** Phase 4 - Check and repair inode allocation map *** Phase 5 - Check and repair block allocation map File system is clean. 4. Mount all the target file systems: #mount /itsoFS

At this point, the FlashCopy target VDisks are ready for access. Example 7-10 shows the final server state. This new file system contains all of the files that were in the original file system at the time of the freeze.

Example 7-10 AIX configuration after importing the VG #datapath query device

Total Devices : 2

DEV#: 0 DEVICE NAME: vpath2 TYPE: 2145 POLICY: Optimized SERIAL: 6005076801AD80E8E000000000000007 ======Path# Adapter/Hard Disk State Mode Select Errors 0 fscsi0/hdisk3 OPEN NORMAL 150 0 1 fscsi0/hdisk5 OPEN NORMAL 161 0 2 fscsi0/hdisk7 OPEN NORMAL 0 0 3 fscsi0/hdisk9 OPEN NORMAL 0 0

DEV#: 1 DEVICE NAME: vpath3 TYPE: 2145 POLICY: Optimized SERIAL: 6005076801AD80E8E000000000000009 ======Path# Adapter/Hard Disk State Mode Select Errors 0 fscsi0/hdisk4 OPEN NORMAL 0 0 1 fscsi0/hdisk6 OPEN NORMAL 0 0 2 fscsi0/hdisk8 OPEN NORMAL 40 0 3 fscsi0/hdisk10 OPEN NORMAL 44 0

192 SVC 4.2.1 Advanced Copy Services hdisk8 none None hdisk9 none None hdisk10 none None vpath2 0009cdda1b5058e4 st7s99 active vpath3 0009cdda1b55c48c st7s99 active

#lsvg rootvg st7s99

#lsvg -l st7s99 st7s99: LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT itsoLV jfs2 32 32 1 open/syncd /itsoFS itsoLog jfs2log 4 4 1 open/syncd N/A

#mount node mounted mounted over vfs date options ------/dev/hd4 / jfs2 Nov 07 14:04 rw,log=/dev/hd8 /dev/hd2 /usr jfs2 Nov 07 14:04 rw,log=/dev/hd8 /dev/hd9var /var jfs2 Nov 07 14:04 rw,log=/dev/hd8 /dev/hd3 /tmp jfs2 Nov 07 14:04 rw,log=/dev/hd8 /dev/hd1 /home jfs2 Nov 07 14:05 rw,log=/dev/hd8 /proc /proc procfs Nov 07 14:05 rw /dev/hd10opt /opt jfs2 Nov 07 14:05 rw,log=/dev/hd8 /dev/itsoLV /itsoFS jfs2 Nov 07 14:36 rw,log=/dev/itsoLog

Chapter 7. Server integration 193 7.4 Windows 2000 and Windows 2003 and copy services

Windows® 2000 and Windows 2003 incorporate a stripped-down version of the VERITAS Volume Manager, called the Logical Disk Manager (LDM) for handling their disks.

With the LDM, you can create logical partitions, perform disk mounts, and create dynamic volumes. The five types of dynamic volumes are simple, spanned, mirrored, striped, and RAID-5.

On earlier versions of Windows, the information relating to the disks was stored in the Windows registry. With Windows 2000 and Windows 2003, this information is stored on the disk drive itself in a partition called the LDM database, which is kept on the last few tracks of the disk. Each volume has its own 128-bit GUID and belongs to a disk group. This is similar to the concept of physical volume identifier (PVID) and volume group in AIX. Because the LDM is stored on the physical drive itself, with Windows 2000 and Windows 2003, it is possible to move disk drives between different computers.

Copy services limitations with Windows 2000 and Windows 2003 FlashCopy source and target VDisks cannot be mapped to the same Windows host when they are created as dynamic volumes. The reason is that each dynamic volume has to have its own 128-bit GUID. As its name implies, the GUID must be unique on one system. When you perform FlashCopy, the GUID is copied as well, w means that if you tried to mount the source and target volume on the same host system, you would have two volumes with exactly the same GUID. This is not allowed, and you will not be able to mount the target volume.

7.4.1 Creating useful copies As with AIX, the way to create useful copies is to ensure that no data is in the system write cache when you freeze the VDisks. The methods for doing so are conceptually similar. Both methods require access to the Computer Management console on your Windows host.

194 SVC 4.2.1 Advanced Copy Services One option is to remove the drive letter associated with the disk. Figure 7-1 shows a VDisk that is currently being used by a Windows server. Selecting Change Drive Letter and Paths opens the dialog box shown in Figure 7-2. Removing a drive letter from a Windows disk results in completing all outstanding writes to the disk. Like unmounting a file system in AIX, no applications can access data on this drive without the drive letter.

Figure 7-1 Change the drive letter of a Windows disk

Figure 7-2 Remove a drive letter from a Windows disk

Chapter 7. Server integration 195 An alternative method is to disable the write cache for the disk in question. In Figure 7-3 the Computer Management console is used to access the properties of a Windows disk.

Figure 7-3 Set the properties of a Windows disk

196 SVC 4.2.1 Advanced Copy Services Choosing Properties brings up the dialog box shown in Figure 7-4.

Figure 7-4 Disable the write cache of a Windows disk

By selecting Optimize for quick removal, the write cache is disabled, which enables you to make the break and be sure that the VDisk contents matches the image on the host.

Example 7-11 shows the initial configuration of the SVC cluster presenting VDisks to this Windows server. The two VDisks represent a data drive and a log drive.

Example 7-11 Cluster configuration prior to FlashCopying two VDisks IBM_2145:ITSOCL2:admin>svcinfo lsvdisk -delim : id:name:IO_group_id:IO_group_name:status:mdisk_grp_id:mdisk_grp_name:capacity:type :FC_id:FC_name:RC_id:RC_name:vdisk_UID:fc_map_count 4:senegalDynamic2:0:io_grp0:online:1:DS4700:36.0GB:striped:2:fcmap2:::6005076801AD 80E8E00000000000000C:1 5:senegalDynamic1:0:io_grp0:online:1:DS4700:36.0GB:striped:1:fcmap1:::6005076801AD 80E8E00000000000000B:1 6:senegalBasic:0:io_grp0:online:1:DS4700:36.0GB:striped:0:fcmap0:::6005076801AD80E 8E00000000000000A:1

IBM_2145:ITSOCL2:admin>svcinfo lshost SENEGAL id 1 name SENEGAL port_count 2 type generic mask 1111 iogrp_count 4 WWPN 210000E08B89B9C0 node_logged_in_count 2 state active WWPN 210000E08B89CCC2

Chapter 7. Server integration 197 node_logged_in_count 0 state active

Example 7-12 shows the SDD configuration of the Windows 2003 server to which these VDisks are mapped. Figure 7-5 on page 199 shows how these VDisks have been configured on a Windows 2003 server.

Example 7-12 SDD configuration of Windows 2003 server C:\Program Files\IBM\Subsystem Device Driver>datapath query device

Total Devices : 3

DEV#: 0 DEVICE NAME: Disk1 Part0 TYPE: 2145 POLICY: OPTIMIZED SERIAL: 6005076801AD80E8E00000000000000C ======Path# Adapter/Hard Disk State Mode Select Errors 0 Scsi Port3 Bus0/Disk1 Part0 OPEN NORMAL 8 0 1 Scsi Port3 Bus0/Disk1 Part0 OPEN NORMAL 10 0 2 Scsi Port3 Bus0/Disk1 Part0 OPEN NORMAL 0 0 3 Scsi Port3 Bus0/Disk1 Part0 OPEN NORMAL 0 0 4 Scsi Port3 Bus0/Disk1 Part0 OPEN NORMAL 0 0 5 Scsi Port3 Bus0/Disk1 Part0 OPEN NORMAL 0 0

DEV#: 1 DEVICE NAME: Disk2 Part0 TYPE: 2145 POLICY: OPTIMIZED SERIAL: 6005076801AD80E8E00000000000000B ======Path# Adapter/Hard Disk State Mode Select Errors 0 Scsi Port3 Bus0/Disk2 Part0 OPEN NORMAL 0 0 1 Scsi Port3 Bus0/Disk2 Part0 OPEN NORMAL 0 0 2 Scsi Port3 Bus0/Disk2 Part0 OPEN NORMAL 3 0 3 Scsi Port3 Bus0/Disk2 Part0 OPEN NORMAL 5 0 4 Scsi Port3 Bus0/Disk2 Part0 OPEN NORMAL 5 0 5 Scsi Port3 Bus0/Disk2 Part0 OPEN NORMAL 5 0

DEV#: 2 DEVICE NAME: Disk3 Part0 TYPE: 2145 POLICY: OPTIMIZED SERIAL: 6005076801AD80E8E00000000000000A ======Path# Adapter/Hard Disk State Mode Select Errors 0 Scsi Port3 Bus0/Disk3 Part0 OPEN NORMAL 11 0 1 Scsi Port3 Bus0/Disk3 Part0 OPEN NORMAL 9 0 2 Scsi Port3 Bus0/Disk3 Part0 OPEN NORMAL 0 0 3 Scsi Port3 Bus0/Disk3 Part0 OPEN NORMAL 0 0 4 Scsi Port3 Bus0/Disk3 Part0 OPEN NORMAL 0 0 5 Scsi Port3 Bus0/Disk3 Part0 OPEN NORMAL 0 0

198 SVC 4.2.1 Advanced Copy Services Figure 7-5 VDisks appearing on a Windows 2003 host prior to mapping target or secondary VDisks

Chapter 7. Server integration 199 The vdisk_UID property of a VDisk matches the SERIAL property of a vpath.The device name of the vpath matches the disk property shown under the Volumes tab (see Figure 7-6). Thus you can use these two properties to determine which Windows disk represents which vpath. Subsequently, you can match a Windows disk to a VDisk.

Figure 7-6 Identifying a Windows disk as a VDisk

The following process shows how you can make sure that the FlashCopy target VDisks contain useful copies: 1. Create the FlashCopy target VDisks. IBM_2145:ITSOCL2:admin>svctask mkvdisk -vtype striped -unit gb -size 36 -iogrp 0 -mdiskgrp DS4700 -name senegalBasic-FC Virtual Disk, id [3], successfully created IBM_2145:ITSOCL2:admin>svctask mkvdisk -vtype striped -unit gb -size 36 -iogrp 0 -mdiskgrp DS4700 -name senegalDyn1-FC Virtual Disk, id [8], successfully created IBM_2145:ITSOCL2:admin>svctask mkvdisk -vtype striped -unit gb -size 36 -iogrp 0 -mdiskgrp DS4700 -name senegalDyn2-FC Virtual Disk, id [7], successfully created 2. Create a FlashCopy consistency group. IBM_2145:ITSOCL2:admin>svctask mkfcconsistgrp -name senegalFC FlashCopy Consistency Group, id [1], successfully created 3. Create FlashCopy mappings for the VDisks and add them to the new FlashCopy consistency group. IBM_2145:ITSOCL2:admin>svctask mkfcmap -source senegalBasic -target senegalBasic-FC -consistgrp senegalFC FlashCopy Mapping, id [0], successfully created IBM_2145:ITSOCL2:admin>svctask mkfcmap -source senegalDynamic1 -target senegalDyn1-FC -consistgrp senegalFC FlashCopy Mapping, id [1], successfully created

200 SVC 4.2.1 Advanced Copy Services IBM_2145:ITSOCL2:admin>svctask mkfcmap -source senegalDynamic2 -target senegalDyn2-FC -consistgrp senegalFC FlashCopy Mapping, id [2], successfully created 4. Prepare the FlashCopy consistency group. IBM_2145:ITSOCL2:admin>svctask prestartfcconsistgrp senegalFC 5. Wait until the consistency group is in the Prepared state. IBM_2145:ITSOCL2:admin>svcinfo lsfcconsistgrp id name status 1 senegalFC preparing . . . IBM_2145:ITSOCL2:admin>svcinfo lsfcconsistgrp id name status 1 senegalFC prepared 6. Disable the write cache for the Windows disks, shown previously in Figure 7-4 on page 197. 7. Start the FlashCopy consistency group. This is the freeze point. IBM_2145:ITSOCL2:admin>svctask startfcconsistgrp senegalFC 8. Reenable the write cache for the Windows disks.

At this point, the FlashCopy operation has been started, and VDisks 3, 8, and 7 are exact copies of VDisks 6, 5, and 4 when the FlashCopy consistency group is started.

7.4.2 Accessing useful copies Basic disks are relatively straightforward to access from a Windows 2000 and Windows 2003 host. However, dynamic disks are a little more complicated because of the restrictions imposed on duplicate GUIDs.

Accessing target or secondary VDisks from the same Windows host In this section, we describe a method of accessing the FlashCopy target VDisks on a single Windows host while the source VDisks are still active on the same server.

The following steps assume that the Windows server and SVC cluster are in the state following the process in the previous section.

Perform these tasks to access the FlashCopy target VDisks: 1. Map the FlashCopy target VDisks to the host. IBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host SENEGAL senegalBasic-FC Virtual Disk to Host map, id [3], successfully created IBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host SENEGAL senegalDyn1-FC Virtual Disk to Host map, id [4], successfully created IBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host SENEGAL senegalDyn2-FC Virtual Disk to Host map, id [5], successfully created

Chapter 7. Server integration 201 2. Scan for new hardware changes, as shown in Figure 7-7, to pick up the newly presented VDisks.

Figure 7-7 Scan for hardware changes to pick up target or secondary VDisks

202 SVC 4.2.1 Advanced Copy Services At this point, the configuration of the Windows host has been changed. Figure 7-8 shows the new status. The new disks, Disk 4 and Disk 6, which are the dynamic disks (as is reported by the LDM), are unallocated. Disk 5, the basic disk, has already been recognized as one. All you have to do is to assign it a drive letter as shown previously in Figure 7-1 on page 195.

Figure 7-8 Windows configuration after discovering target or secondary VDisks

Chapter 7. Server integration 203 After clicking Add (see Figure 7-2 on page 195), we are presented with the dialog box shown in Figure 7-9.

Figure 7-9 Add a drive letter to a basic drive

When you have assigned drive letters to your basic drives, you should run chkdsk to ensure file system consistency as shown in Example 7-13. This is particularly important if the break is due to an unplanned Metro/Global Mirror stoppage.

Example 7-13 Run chkdsk on a newly mapped target or secondary VDisk C:\Program Files\IBM\Subsystem Device Driver>chkdsk /f e: The type of the file system is NTFS. Volume label is basicVol.

CHKDSK is verifying files (stage 1 of 3)... File verification completed. CHKDSK is verifying indexes (stage 2 of 3)... Index verification completed. CHKDSK is verifying security descriptors (stage 3 of 3)... Security descriptor verification completed. Windows has checked the file system and found no problems.

37744685 KB total disk space. 20 KB in 2 files. 4 KB in 9 indexes. 0 KB in bad sectors. 67137 KB in use by the system. 65536 KB occupied by the log file. 37677524 KB available on disk.

4096 bytes in each allocation unit. 9436171 total allocation units on disk. 9419381 allocation units available on disk.

204 SVC 4.2.1 Advanced Copy Services Target or secondary VDisks that are dynamic disks should never be presented to a host that can see the source or primary VDisk because you risk losing data.

Accessing target or secondary VDisks from the same Windows host

Accessing target or secondary VDisks from a different Windows server to the one accessing the source or primary VDisks is much simpler. In Example 7-14, the server originally had no VDisks mapped to it but now we can see that the VDisks are mapped to the server.

Example 7-14 Map FlashCopy targets to a new Windows server® IBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host SIAM senegalBasic-FC Virtual Disk to Host map, id [2], successfully created IBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host SIAM senegalDyn1-FC Virtual Disk to Host map, id [3], successfully created IBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host SIAM senegalDyn2-FC Virtual Disk to Host map, id [4], successfully created

Chapter 7. Server integration 205 As shown in Figure 7-7 on page 202, the next step is to scan for hardware changes. The result of this action is shown in Figure 7-10.

Figure 7-10 Results of hardware scan on a Windows 2003 server

As shown, the basic disk is picked up normally and needs a drive letter assigned to it. Figure 7-10 also shows the state of the dynamic disks; they register as foreign.

206 SVC 4.2.1 Advanced Copy Services By selecting Import Foreign Disks, you access the dialog box shown in Figure 7-11.

Figure 7-11 Import Foreign Disks

After you click OK, you access Figure 7-12, which lists the drives you will import.

Figure 7-12 Foreign Disk Volumes

At this point, all of the drive letters have been assigned to your VDisks; you should run chkdsk to ensure file system consistency as shown previously in Example 7-13 on page 204. This is particularly important if the freezing was due to an unplanned Metro/Global Mirror stoppage.

Chapter 7. Server integration 207 7.5 Linux specifics

In this section, we describe the steps needed to create and access copied VDisks on Linux® hosts. SVC currently supports Red Hat and Suse Linux, but the steps are distribution independent.

SVC supports the use of multiple multipathing solutions when accessed with a Linux server. The examples in the sections that follow focus on the use of SDD.

7.5.1 Creating useful copies

As with AIX and Windows, it is necessary to ensure that any information in the file system cache is written to the VDisks prior to freezing the VDisks. The simplest way to do this is to unmount the file system, as was shown in Example 7-1 on page 181.

The XFS file system includes support for freezing file systems in a similar manner to AIX. Details on this functionality is beyond the scope of this Redbooks publication, but you can find information in an IBM developerWorks article at: http://www.ibm.com/developerworks/linux/library/l-fs9.html.

Example 7-15 shows the initial configuration of the SVC cluster, presenting VDisks to a Red Hat Linux server.

Example 7-15 Configuration of clusters prior to copying a VDisk with Metro Mirror IBM_2145:ITSOCL2:admin>svcinfo lsvdisk -delim : id:name:IO_group_id:IO_group_name:status:mdisk_grp_id:mdisk_grp_name:capacity:type :FC_id:FC_name:RC_id:RC_name:vdisk_UID:fc_map_count 8:diomede-1ary:0:io_grp0:online:0:DS4500:36.0GB:striped:::::6005076801AD80E8E00000 0000000014:0

IBM_2145:ITSOCL2:admin>svcinfo lshost DIOMEDE id 0 name DIOMEDE port_count 2 type generic mask 1111 iogrp_count 4 WWPN 210000E08B0548BC node_logged_in_count 2 state active WWPN 210000E08B0541BC node_logged_in_count 2 state active

IBM_2145:ITSOCL3:admin>svcinfo lsvdisk -delim : id:name:IO_group_id:IO_group_name:status:mdisk_grp_id:mdisk_grp_name:capacity:type :FC_id:FC_name:RC_id:RC_name:vdisk_UID:fc_map_count 6:diomede-2dary:0:io_grp0:online:0:DS4500:36.0GB:striped:::::6005076801A100E908000 00000000008:0

208 SVC 4.2.1 Advanced Copy Services Example 7-16 shows the configuration of an Linux server prior to copying a VDisk with Metro Mirror.

Example 7-16 Linux host configuration prior to copying VDisks with Metro Mirror [root@diomede ~]# datapath query device

Total Devices : 1

DEV#: 0 DEVICE NAME: vpatha TYPE: 2145 POLICY: Optimized Sequential SERIAL: 6005076801ad80e8e000000000000014 ======Path# Adapter/Hard Disk State Mode Select Errors 0 Host0Channel0/sda OPEN NORMAL 1 0 1 Host0Channel0/sdb OPEN NORMAL 41815 0 2 Host1Channel0/sdc OPEN NORMAL 0 0 3 Host1Channel0/sdd OPEN NORMAL 41679 0

[root@diomede ~]# pvdisplay --- Physical volume --- PV Name /dev/hda2 VG Name VolGroup00 PV Size 74.41 GB / not usable 0 Allocatable yes PE Size (KByte) 32768 Total PE 2381 Free PE 3 Allocated PE 2378 PV UUID Xq0UQs-94ID-tCq7-4jK2-KRt7-4iYF-lCf9wf

--- Physical volume --- PV Name /dev/vpatha VG Name itso PV Size 36.00 GB / not usable 0 Allocatable yes PE Size (KByte) 4096 Total PE 9215 Free PE 4607 Allocated PE 4608 PV UUID dAZQYq-VsI1-n2xp-vp0c-hfSE-glCx-MjnPV3

[root@diomede ~]# vgdisplay itso --- Volume group --- VG Name itso System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1

Chapter 7. Server integration 209 Max PV 0 Cur PV 1 Act PV 1 VG Size 36.00 GB PE Size 4.00 MB Total PE 9215 Alloc PE / Size 4608 / 18.00 GB Free PE / Size 4607 / 18.00 GB VG UUID 0PsMcJ-nDr8-b3ds-YyCM-Mlde-OegZ-LZFddT

[root@diomede /]# df -T File System Type 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 ext3 74699952 11571300 59334120 17% / /dev/hda1 ext3 101086 22817 73050 24% /boot none tmpfs 1033144 0 1033144 0% /dev/shm /dev/mapper/itso-mmPrimary ext3 18578172 77888 17556568 1% /mmSource

[root@diomede /]# mount /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw) none on /proc type proc (rw) none on /sys type sysfs (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) usbfs on /proc/bus/usb type usbfs (rw) /dev/hda1 on /boot type ext3 (rw) none on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) /dev/mapper/itso-mmPrimary on /mmSource type ext3 (rw)

The VG that is being copied is itso, which contains a single vpath, vpatha.

There are no user commands for freezing this ext3 file system, so the only way to freeze the VDisk cleanly is to unmount the file system and deactivate the LV. However, with appropriate planning and scripting, the disruption can be minimized. 1. Create the Metro Mirror consistency group. IBM_2145:ITSOCL2:admin>svctask mkrcconsistgrp -name diomedeMM -cluster ITSOCL3 RC Consistency Group, id [253], successfully created 2. Create the Metro Mirror relationship connecting the two VDisks and add it to the consistency group. IBM_2145:ITSOCL2:admin>svctask mkrcrelationship -master diomede-1ary -aux diomede-2dary -cluster ITSOCL3 -name diomedeMM1 -consistgrp diomedeMM RC Relationship, id [8], successfully created 3. Start the consistency group. IBM_2145:ITSOCL2:admin>svctask startrcconsistgrp diomedeMM 4. Monitor the consistency group and wait until it attains consistency and synchronization. IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp diomedeMM id 253 name diomedeMM master_cluster_id 000002006B603A38

210 SVC 4.2.1 Advanced Copy Services master_cluster_name ITSOCL2 aux_cluster_id 0000020068403A42 aux_cluster_name ITSOCL3 primary master state consistent_synchronized relationship_count 1 freeze_time status online sync copy_type metro RC_rel_id 8 RC_rel_name diomedeMM1 5. Unmount the file system to flush all the data to the primary VDisk. [root@diomede ~]# umount /mmSource 6. Stop the Metro Mirror consistency group. This is the freeze point. IBM_2145:ITSOCL3:admin>svctask stoprcconsistgrp -access diomedeMM

IBM_2145:ITSOCL3:admin>svcinfo lsrcconsistgrp diomedeMM id 253 name diomedeMM master_cluster_id 000002006B603A38 master_cluster_name ITSOCL2 aux_cluster_id 0000020068403A42 aux_cluster_name ITSOCL3 primary state idling relationship_count 1 freeze_time status sync in_sync copy_type metro RC_rel_id 6 RC_rel_name diomedeMM1 7. Remount the file system. [root@diomede ~]# mount /dev/itso/mmPrimary /mmSource

At this point, the Metro Mirror relationship has been stopped and diomede-2ary is an exact copy of diomede-1ary.

7.5.2 Accessing useful copies

As with AIX, if the source or primary VDisk is defined to the AIX logical volume manager (LVM), all of its data structures and identifiers are copied to the target or secondary VDisk as well. This includes the Universal Unique Identifiers (UUIDs) that are used to identify PVs, VGs, and LVs.

It is currently not possible to activate a volume group with a physical volume (vpath) containing a VG UUID and a PV UUID that is already used in a volume group existing on the same server.

Chapter 7. Server integration 211 Accessing target or secondary VDisks from the same Linux host Our research into methods for bringing copied volume groups online on a Linux server indicate that users have published their own solutions on various newsgroups. No clear method for doing this in a nondisruptive manner has been found. Most strategies involve manually altering backups of the LVM configuration. As a result, we recommend that you do not present copied VDisks back to their original servers. If you determine that this process is necessary, you should consult with LVM experts to determine a suitable method.

Accessing target or secondary VDisks from a different Linux host

In this section, we describe how to recover a VDisk that was not frozen cleanly. This can happen when a Metro Mirror relationship stops while a file system is in use. 1. Map the target VDisk to the new server. IBM_2145:ITSOCL3:admin>svctask mkvdiskhostmap -host PALAU diomede-2dary Virtual Disk to Host map, id [0], successfully created 2. Discover the LU on the Linux server. QLogic® provides a tool to perform this step. More information can be found at: http://download.qlogic.com/ms/53595/readme_dynamic_lun_util_18.html 3. Run cfgvpath to create the relevant vpath. [root@palau ~]# cfgvpath c------1 root root 254, 0 Nov 14 22:32 /dev/IBMsdd Added vpatha 252 0 path sda 8 0.... Added vpatha 252 0 path sdb 8 16.... Added vpatha 252 0 path sdc 8 32.... Added vpatha 252 0 path sdd 8 48.... Writing out new configuration to file /etc/vpath.conf Waiting for hotplug/udev system to configure all vpath device nodes... 4. Run pvscan to pick up the copied PV. [root@palau ~]# pvscan PV /dev/hda2 VG VolGroup00 lvm2 [74.41 GB / 96.00 MB free] PV /dev/vpatha VG itso lvm2 [36.00 GB / 18.00 GB free] Total: 2 [110.40 GB] / in use: 2 [110.40 GB] / in no VG: 0 [0 ] 5. Run vscan to make sure that you have your VG. [root@palau ~]# vgscan Reading all physical volumes. This may take a while... Found volume group "VolGroup00" using metadata type lvm2 Found volume group "itso" using metadata type lvm2 6. Run lvscan to make sure that you have your expected LVs. [root@palau ~]# lvscan ACTIVE '/dev/VolGroup00/LogVol00' [72.38 GB] inherit ACTIVE '/dev/VolGroup00/LogVol01' [1.94 GB] inherit inactive '/dev/itso/mmPrimary' [18.00 GB] inherit 7. You must activate your newly discovery volume group. [root@palau ~]# vgchange -ay itso 1 logical volume(s) in volume group "itso" now active 8. Run fsck on the LV and then mount it.

212 SVC 4.2.1 Advanced Copy Services 7.6 Other operating systems

The principles outlined in previous sections are equally relevant to other operating systems. You must be aware of the following when determining operating procedures. What metadata is written to the disk (and thus copied by copy services)? How does the operating system handle duplicated metadata? How does the operating system handle “foreign” metadata - that is, disks that were originally mapped to a different server? How does the operating system handle VDisks that were not frozen cleanly?

Chapter 7. Server integration 213 214 SVC 4.2.1 Advanced Copy Services 8

Chapter 8. Automation

In this chapter, we discuss methods for automating your copy services solution. A number of methods are open to you, including SVC command-line scripting Server-side scripting SVC CIMOM programming

Automation is the removal of human decision making from system operating processes. As a general rule, system designs require a human operator to be involved at some stage. For example, humans are necessary for deciding when to: Stop and start FlashCopy mappings or Remote Copy relationships Failover Remote Copy relationships (or fail back)

This chapter also discusses the importance of logging as part of automation.

To automate copy services processes, you need to connect to the cluster. In normal operations, you do so by using the GUI or command line. The GUI is not an appropriate interface for automating processes, so we do not be discuss that alternative. All automation techniques are achieved through the SVC command line or the Common Information Model Object Manager (CIMOM), which, currently, acts as a proxy to the command line.

In this section, we refer to the user agent. This may be the CIMOM (which connects to the cluster using SSH) or a user connecting directly with an SSH client, either in an interactive mode or by means of a script.

Running commands to the cluster follow these steps: 1. Connection 2. Authentication 3. Submission 4. Authorization 5. Execution

8.1.1 Connection

Commands are submitted to the cluster during a connection session to the cluster. User agents make connections via the SSH protocol. SVC has a number of security features that affect how often you can attempt connections. These security features are in place to prevent attacks (malicious or accidental) that can bring down an SVC node. These features may initially seem restrictive, but they are relatively simple to work around to maintain a valid connection.

Any automation system should ensure that it behaves responsibly and does not attempt to breach the connection rules. At the very least, any automation system should ensure that it can gracefully handle rejected connection attempts.

216 SVC 4.2.1 Advanced Copy Services Figure 8-1 shows how SVC connection restrictions work.

Figure 8-1 SVC SSH restrictions

Two “queues” are in action: Pending Connections and Active Connections. The connection process follows this sequence: 1. A connection request comes into the SVC. If the Pending Connections queue has a free position, the request is added to it. Otherwise, there are two possibilities: Prior to version 4.2.1.0, the connection request times out; as of version 4.2.1.0, the connection is explicitly rejected. 2. Pending Connections are handled in one of two ways. a. If any of the following are true, the connection request is rejected: • No key is provided or the provided key is incorrect. • The provided user name is not admin or service. • The Active Connections queue is full. In this case, a warning is returned to the SSH client as shown in Example 8-1 on page 218. b. If none of the scenarios previously listed in item a are true, the connection request is accepted and moved from the Pending Connections queue to the Active Connections queue.

Chapter 8. Automation 217 3. Active Connections are ended after any of the following events: – User logs off manually. – SVC’s SSH daemon recognizes that the connection has grown idle. – Network connectivity fails. – The configuration node fails over. In this case, both queues are cleared as the SHH daemon stops and restarts on a different node.

Example 8-1 SVC command-line warning - too many logins Using username "admin". Authenticating with public key "rsa-key-20071010" Too many logins for 'admin'. Last login: Thu Oct 25 14:15:30 2007 from 9.67.163.247

8.1.2 Authentication

This stage is represented by step 2 on page 217 in the connection process (described in 8.1.1, “Connection” on page 216). The only authentication process available is public/private key authentication. IBM System Storage SAN Volume Controller, SG24-6423 provides detailed instructions on creating keys and uploading them to the SVC. We emphasize a few points here to make automation simpler to implement.

User names and keys The SVC command line only has two users defined: admin service

Any attempt to log on as a different user results in rejection, regardless of whether a valid key is provided. In this way, keys and user names are wholly independent.

Multiple user names can be defined in the SVC GUI: the default user name, superuser, and any others that are created. User names have no connection to the SVC cluster. They are, in fact, user names for logging on to the CIMOM, which runs on the Master console. When the CIMOM logs on to the SVC cluster, it still uses the admin user name and the icat.ppk private key.

The SVC cluster enables you to store up to 100 keys. Any of these keys can be used to authenticate with the cluster, regardless of which user name you enter on the command line.

Once the user agent provides a valid user name, the cluster receives the private key. If it matches one of the cluster’s public keys, that key’s label is remembered and used for auditing purposes.

If the connection is made through that CIMOM, the key label will always be the same (being the public key to match the icat.ppk file). However, the user name that was used to log on to the CIMOM is sent to the SVC cluster and is also remember for auditing purposes.

8.7, “Auditing” on page 243 discusses the auditing of commands on the SVC cluster.

218 SVC 4.2.1 Advanced Copy Services 8.1.3 Submission

Once connected to a cluster, the user agent may start submitting commands. First, the syntax is checked. If this fails, an appropriate error message is returned. Any automation implementation must ensure that all submitted commands have the correct syntax. If they do not, they must be designed to handle syntax errors. It is much easier to design a solution that does not generate invalid syntax than one to handle all potential syntax errors.

8.1.4 Authorization

Next, commands with valid syntax are checked to determine whether the user agent has the authority to submit the command. Section 2.8.3, “Authorization roles” on page 21 outlines the authorization roles that SVC supports. A role is associated with the key that was used to authenticate the connection. SVC checks the submitted command against the authorization role. If the user agent is not authorized to execute this command, the following error is returned: CMMVC6253E The command failed authorization because the session ssh key does not have the requisite role. If the user agent is authorized, the command is sent for execution.

8.1.5 Execution

When a command is executed, one of the following occurs: The command fails, and an error message is written to STDERR. The command succeeds, and a warning is written to STDERR. The command succeeds, a warning is written to STDERR, and information is sent to STDOUT. The command succeeds, and information is written to STDOUT. The command succeeds, and nothing is written to STDOUT.

Data written to STDOUT and STDERR by the cluster should be written to STDOUT and STDERR by your SSH client, but you should check this yourself.

8.2 Creating connections

Connecting to the cluster is the first step in running commands. Any automation solution requires a connection component. This component should be as robust as possible because it forms the foundation of your solution.

There are two forms of connection solutions: Transient One command is submitted per connection, and the connection is closed once the command is completed. Persistent The connection is made and stays open. Multiple commands are submitted through this single connection, including interactive sessions and the CIMOM.

Chapter 8. Automation 219 8.2.1 Transient connections

Transient connections are simple to create. The most common SSH clients enable the user to submit a single command as part of their invocation. Example 8-2 and Example 8-3 show this using ssh on a AIX server and plink on a Windows server.

Example 8-2 Transient connection to SVC cluster from AIX server [root@heket]/ >ssh -i privateKey -l admin rc-cluster-13 svcinfo lscluster -delim : id:name:location:partnership:bandwidth:cluster_IP_address:cluster_service_IP_addre ss:id_alias 00000200694052D0:rc-cluster-13:local:::9.71.42.19:9.71.42.75:00000200694052D0 [root@heket]/ >

Example 8-3 Transient connection to SVC cluster from Windows server C:\Documents and Settings\Administrator>plink -i "C:\Documents and Settings\Administrator\My Documents\ITSO\private.ppk" -l admin 9.43.86.117 svcinfo lscluster -delim : id:name:location:partnership:bandwidth:cluster_IP_address:cluster_service_IP_addre ss:id_alias 0000020060406FCA:ITSOCL1:local:::9.43.86.117:9.43.86.118:0000020060406FCA

C:\Documents and Settings\Administrator>

These transient connections go through all five stages of running a command and return to the command line. You can redirect the two output streams (STDOUT and STDERR) using the operating system’s standard redirection operators to capture the responses.

These lengthy invocations can be shorted in client-specific ways. User configuration files can be used with the AIX SSH client. The configuration file in Example 8-4 enables you to create a transient connection, which is shown in Example 8-5.

Example 8-4 Sample SSH configuration file (saved as “sampleCfg”) Host rc13 HostName rc-cluster-13 IdentityFile /privateKey User admin

Host someOtherCluster IdentityFile /someOtherKey User admin

Example 8-5 Transient connection to SVC cluster using ssh and configuration file [root@heket]/ >ssh -F sampleCfg rc13 svcinfo lscluster -delim : id:name:location:partnership:bandwidth:cluster_IP_address:cluster_service_IP_addre ss:id_alias 00000200694052D0:rc-cluster-13:local:::9.71.42.19:9.71.42.75:00000200694052D0 [root@heket]/ >

220 SVC 4.2.1 Advanced Copy Services Shortening the plink invocation requires the creation of a PuTTY session. To do this, open the PuTTY application and enter admin@ in the Host Name textbox as shown in Figure 8-2

Figure 8-2 Add user name and IP address to a PuTTY session

As shown in Figure 8-3, the next step is to configure the private key for this session. Select Connection →SSH →Auth.

Figure 8-3 Set private key for a PuTTY SSH session

Chapter 8. Automation 221 Click the Browse button and select the appropriate private key, locating it in your file system.

The final step is to save the session for future use. We recommend you use a descriptive session name as shown in Figure 8-4.

Figure 8-4 Save PuTTY session for use with plink

After a session has been saved, it can be used for making transient connections from the command line as shown in Example 8-6.

Example 8-6 Transient connection to an SVC cluster using plink with a PuTTY session C:\Documents and Settings\Administrator>plink -load ITSOCL1 svcinfo lscluster -delim : id:name:location:partnership:bandwidth:cluster_IP_address:cluster_service_IP_addre ss:id_alias 0000020060406FCA:ITSOCL1:local:::9.43.86.117:9.43.86.118:0000020060406FCA

C:\Documents and Settings\Administrator>

8.2.2 Persistent connections

A persistent connection is a connection that exists beyond the submission and execution of a single command. As outlined previously, the CIMOM provides a persistent connection, but it does not provide direct access to the command line. To provide a persistent connection to the command line, you have to use multiple processes. There are as many ways to do this as there are programming languages. Example 8-12 on page 226 shows a method of creating a persistent connection using the Korn shell. Most methods involve creating a process that connects to the cluster, writing to its STDIN stream, and reading from its STDOUT and STDERR streams. The difficulty is in detecting when the output from a command ends. This is handled in Example 8-12 on page 226.

222 SVC 4.2.1 Advanced Copy Services Persistent connections can be used in a number of ways: On a per-script basis A script opens up a connection that exists for the life of the script, allowing multiple commands to be submitted, but ends when the script ends. As a standalone script A connection is opened, and other scripts communicate with this one script to submit commands to the cluster. This allows the connection to be shared by multiple scripts, which, in turn, allows a greater number of independent scripts to access the cluster without using up all of the connection slots.

8.3 SVC command-line scripting

When connected to the cluster command line, it is possible to do small amounts of automation, for instance: Repeatedly submitting a single command to a set of SVC objects Searching the configuration for objects conforming to certain criteria

The SVC command line is actually a highly restricted Bash shell. You cannot access such UNIX commands as cd or ls. The only commands available are those that are built in, such as echo or read. In addition, redirecting inputs and outputs is not supported, but you can pipe commands together.

Example 8-7 shows a sample script lists all of the VDisks that are not online. This complements the filtervalue parameter of the lsvdisk command. The filtervalue parameter allows matches only when a property matches a value. The command-line script in Example 8-7 allows matches according to other criteria.

Example 8-7 SVC command-line script listing VDisks that are not online 001. svcinfo lsvdisk -nohdr | while read id name IOGid IOGname status rest 002. do 003. if [ "$status" != "online" ] 004. then 005. echo "VDisk '$name' $$id$ is $status" 006. fi 007. done

Line 1 is of particular note. This line submits the svcinfo lsvdisk command and pipes the output to the read command, combined with a while command. This creates a loop that executes once per line of output from the svcinfo command.

The read command is followed by a list of variables. A line is read from the svcinfo command, and the first word in that line is assigned to the first variable, the second word is assigned to the second variable, and so on, with any leftover words assigned to the final variable (with intervening spaces included).

We use the -nohdr parameter because we are not interested in this information.

Lines 3 to 6 simply check the status variable. If it is not equal to online, the information is printed to STDOUT.

Chapter 8. Automation 223 8.3.1 Submitting command-line scripts

You can submit command-line scripts from an interactive prompt, if required. However, you can also submit the scripts as batch files. Example 8-8 and Example 8-9 show how this can be done with ssh and plink, respectively. Both commands submit a simple batch file, which is shown in Example 8-10. This command lists the quorum MDisks; in a large configuration, this script can be useful.

Example 8-8 Submission of batch file to SVC using ssh [root@heket]/ >ssh -F sampleCfg rc13 -T < batchFile.sh MDisk 0 (mdisk0) is quorum disk 0 MDisk 1 (mdisk1) is quorum disk 1 MDisk 2 (mdisk2) is quorum disk 2

Example 8-9 Submission of batch file to SVC using plink C:\Documents and Settings\Administrator\My Documents\SVC>plink -load ITSOCL1 -m batchFile.sh MDisk 0 (mdisk0) is quorum disk 0 MDisk 1 (mdisk1) is quorum disk 1 MDisk 2 (mdisk2) is quorum disk 2

Example 8-10 Command-line batch file batchFile.sh svcinfo lsmdisk -nohdr | while read id name rest do svcinfo lsmdisk $id | while read key value do if [ "$key" == "quorum_index" ] then if [ "$value" != "" ] then echo "MDisk $id ($name) is quorum disk $value" fi fi done done

8.3.2 Example SVC command-line script

Example 8-11 shows a nontrivial script that can be executed directly from the SVC command line using batch submission. The script generates a CSV table showing the number of extents each VDisk gets from each MDisk.

Example 8-11 Generating CSV representation of MDisk/VDisk extent usage 001. echo vDisk,mDisk,Controller,mDisk Group,Extents; 002. 003. vdiskIds=(`svcinfo lsvdisk -nohdr | while read id rest; do echo -n "$id "; done`) 004. vdiskNames=(`svcinfo lsvdisk -nohdr | while read id name rest; do echo -n "$name "; done`) 005. vdiskNameMap=() 006. for (( i = 0 ; i < ${#vdiskNames[@]} ; i++ ))

224 SVC 4.2.1 Advanced Copy Services 007. do 008. vdiskNameMap[${vdiskIds[$i]}]=${vdiskNames[$i]} 009. done 010. 011. svcinfo lsmdisk -nohdr | while read mdiskId mDiskName status mode mdgId mdgName capacity LUN controllerName UniqueID; 012. do 013. svcinfo lsmdiskextent -nohdr $mdiskId | while read vdiskId extents; 014. do 015. echo ${vdiskNameMap[$vdiskId]},$mDiskName,$controllerName,$mdgName, $extents; 016. done 017. done

Line 1 simply generates the table header row.

Lines 3 and 4 work around a limitation of the bash shell. If you assign a value to a variable while inside a loop that is running from a pipe, that value is not available to you when outside the loop. A way to get around this is command substitution. Line 3 creates an array of VDisk IDs by looping through all of the VDisks and printing the ID. This output is captured using backticks. The parentheses turn this list of IDs into an array. Line 4 does the same with VDisk names.

Lines 5 to 9 create an associative array that maps a VDisk ID to a VDisk name. The loop this time does not involve a pipe, so the changes to the vdiskNameMap array are available after line 9.

Line 11 begins a loop over all MDisks.

Line 13 loops over all of the MDisk Extent records for the current MDisk. A line is printed for each VDisk that uses extents from the current MDisk.

8.4 Server-side scripting

Server-side scripting involves scripting where the majority of the programming logic is executed on a server.

Part of server-side scripting is the generation and management of connections to the SVC cluster. Section 8.4.1, “Creating persistent SVC connections” on page 226 shows how to create and manage a persistent connection to a cluster and how to manage requests coming from multiple scripts.

In addition to this example, an SVC Perl module is available at: http://alphaworks.ibm.com/tech/svctools

The Perl module handles the connection aspect of any script. Because connection management is often the most complex part of any script, we recommend that you investigate this module. Currently, this module uses transient connections to submit commands to a cluster, and it may not be the best approach if you plan to use multiple scripts submitting commands independently.

Chapter 8. Automation 225 8.4.1 Creating persistent SVC connections

Example 8-12 is an example of a script that opens a persistent connection to a cluster and handles commands from other, external scripts. In this section, we describe the various features of the script to give you an idea of some of the issues that must be resolved.

Example 8-12 Creating persistent connection to SVC cluster

001. #!/bin/ksh 002. 003. # 004. # SVC Command Queue 005. # 006. # This script implements a queue that any number of clients can write to. 007. # The clients provide SVC commands and the name of a file to where the output 008. # should be written 009. # Commands should be written to the queue thus: 010. # 011. # 012. # 013. # command: SVC command to be submitted to the cluster 014. # delimiter: User definable character (defined in this script) 015. # outFile: File to which the output of the SVC command should be written 016. # 017. # e.g.: echo svcinfo lscluster -delim :+/tmp/output 018. # 019. # This would send the 'svcinfo lscluster -delim :' command to the cluster and 020. # place the output in the file '/tmp/output' 021. # 022. # The '+' character delimits the command and the outpue file 023. 024. ########################################################### 025. # Functions start 026. ########################################################### 027. # 028. # Opens an SSH pipe and purges any extraneous data from the buffer 029. # SSHPID gets set to the PID of the SSH process 030. # 031. # This function assumes that the connection will work 032. # 033. open_SSH_pipe() 034. { 035. echo "Opening SSH pipe to $CL..." 036. ssh -p 22 -q admin@$CL 2>&1 |& 037. SSHPID=$! 038. 039. echo "Started SSH process: $SSHPID" 040. 041. # 042. # Purge any data from the connection 043. # 044. echo "Purging connection buffer..." 045. svc_transaction "svcinfo lscluster" "/dev/null" 046. echo "Connection buffer purged" 047. }

226 SVC 4.2.1 Advanced Copy Services 048. 049. # 050. # Opens the command FIFO and connects it to file descriptor 4 051. # 052. open_command_FIFO() 053. { 054. # if the FIFO doesn't exists, create it 055. if [ ! -a $FIFO ] 056. then 057. echo "Cannot find $FIFO so creating it" 058. mkfifo $FIFO 059. fi 060. exec 4<$FIFO 061. } 062. 063. # 064. # svc_transaction(command,outputfile) 065. # 066. # Submits the supplied command to the SVC cluster 067. # Send the output to the outputfile 068. # 069. svc_transaction() 070. { 071. # 072. # First, submit the command to the cluster 073. # followed by a command to print the return code 074. # and then a command to print the End Of Output string 075. # 076. print -p "$1" 077. print -p "echo \$?" 078. print -p "echo $COMMAND_ENDER" 079. 080. # 081. # Now, read back from the SVC cluster 082. # 083. read -p prevLine 084. while true 085. do 086. read -p currLine 087. if [ "${currLine}" == "$COMMAND_ENDER" ] 088. then 089. # 090. # Append the SVC Return Code to the output file 091. # 092. echo $RC_PREPEND$prevLine$RC_APPEND >> $2 093. echo $COMMAND_ENDER >> $2 094. return 095. else 096. echo "${prevLine}" >> $2 097. prevLine=$currLine 098. fi 099. done 100. } 101. 102. ###########################################################

Chapter 8. Automation 227 103. # Functions end 104. ########################################################### 105. 106. 107. # 108. # CL: cluster name - to be provided on the command line 109. # FIFO: FIFO name 110. # DELIM: Delimiter separating commands from their output files 111. # COMMAND_ENDER: String that signifies that the SVC command output has ended 112. # RC_PREPEND: 113. # : Strings that wrap round the return code from the SVC cluster 114. # RC_APPEND : 115. # 116. if [ "$1" == "" ] 117. then 118. echo "Please provide a cluster name" 119. return 0 120. fi 121. CL=$1 122. FIFO=/tmp/FIFO.$CL 123. DELIM=+ 124. COMMAND_ENDER="+++COMPLETE+++" 125. RC_PREPEND="+++RC:" 126. RC_APPEND="+++" 127. 128. 129. open_SSH_pipe 130. open_command_FIFO 131. 132. echo "Queue is ready..." 133. while true 134. do 135. while read CMD <&4 136. do 137. OUTFILE=`echo $CMD | cut -d $DELIM -f 2` 138. echo "Outfile: $OUTFILE" 139. SVCCMD=`echo $CMD | cut -d $DELIM -f 1` 140. echo "SVC Command: $SVCCMD" 141. # 142. # Probably a good idea to check that the SSH process is still running and 143. # handle it if it is not. 144. # 145. ps -p $SSHPID | grep -q ssh 146. if [ $? = 0 ] 147. then 148. svc_transaction "$SVCCMD" $OUTFILE 149. else 150. # 151. # Handle the fact that SSH is no longer running 152. # 153. echo "SSH pipe seems to have died" 154. return 0 155. fi 156. done 157. done

228 SVC 4.2.1 Advanced Copy Services 158. 159. # 160. # Closes FIFO, although we will never get here. 161. # 162. exec 4<&-

open_SSH_pipe This function runs from lines 33 to 47 (the numbers correspond to the line numbers in Example 8-12 on page 226) and opens the connection to the SVC cluster. The only input is a global variable $CL, which holds the name of the cluster.

The function is quite simple, consisting of only three functional lines (36, 37, and 45). The first line opens the connection, redirects STDERR to STDOUT, and makes this connection a coprocess. A coprocess is a Korn shell term that means a process running in the background of the main process. STDIN and STDOUT of a coprocess are available to the main process for reading and writing.

Line 37 saves the process ID into $SSHPID for future reference.

Line 45 runs a subroutine called svc_transaction to clear the I/O buffer between the main process and the coprocess. This subroutine is described in “svc_transaction” on page 229.

open_command_FIFO This function runs from lines 52 to 61 and opens a first-in-first-out (FIFO) file. (A FIFO file is a UNIX construct. It is essentially a named pipe.) The FIFO file acts as a queue of commands that this script must handle. External scripts add commands to this queue, and this script takes them off the queue in the order that they came in and submits them to the cluster.

Lines 55 to 59 determine whether the FIFO file has been created (and create it if it has not been).

Line 60 opens up the FIFO file with file descriptor 4 for reading. A file descriptor is simply a handle to this file; UNIX shells can read and write from these file descriptors.

svc_transaction This function runs from lines 69 to 100 and submits a command to the SVC cluster, writing the output to a file. It takes two parameters - the first is the full command to submit to the cluster and the second is a local file, to which the output of the command should be written.

Lines 76 to 78 send information to the coprocess - that is, the persistent connection. This is done with the print -p command. First, the command is sent. Then, a request to echo the return code of the command is sent. Finally, a request to echo a known string is sent. The request to echo a known string is used to detect the end of output from the submitted command.

Lines 83 to 99 handle the gathering of the cluster’s response. The command read -p reads from the coprocess and puts the information into the supplied variable. The loop is fairly simple: 1. Read one line from the coprocess and save as prevLine. 2. Start loop. 3. Read next line from coprocess as currLine.

Chapter 8. Automation 229 4. If currLine matches the end of command string: a. Write prevLine to output file with a wrapper indicating that it is the return code. b. Write the end of command string to the output file. c. End subroutine. 5. If not: a. Write prevLine to output file. b. Save currLine as prevLine. 6. Go to step 2.

When this subroutine is complete, the I/O buffer to the coprocess is clear.

Main script The main script runs from line 116 to 162 and sets up the FIFO and persistent connection. The script then sits waiting for commands to be added to the queue, before submitting them to the cluster.

Lines 116 to 126 set up various variables required for execution. A more mature implementation would allow these variables to be set with a configuration file, but this method suffices for a demonstration. The script requires that a cluster name be provided as a command-line parameter.

Lines 129 and 130 open the connection to the SVC cluster and a connection to the command queue (the FIFO file).

Lines 133 to 157 are the main loop. This loop monitors the command queue and handles incoming commands. This process is followed: 1. Read a line from the FIFO. This is a blocking call that waits until a line is available. 2. Split this line on a predefined delimiter. The first field is the command destined for the cluster. The second field is the file to which all output should be directed. 3. Check that the connection to the cluster is still running. In this example, the script ends if the connection ends. A more sophisticated response would be to re-attempt connection. 4. If the connection is still running, the command is passed to the svc_transaction subroutine.

The final line should never be reached due to the constant loop, but its action is to close the file descriptor to the FIFO queue.

Command submission If multiple scripts are trying to write to the command queue that was created previously, steps must be taken to ensure that the writes are atomic. That means if two scripts attempt to write to the command queue at overlapping times, the commands written do not interleave.

Example 8-13 shows a simple C program that performs an atomic append to a file. This is a perfect utility for writing to the FIFO file when multiple scripts are accessing at the same time. It blocks on the write function on line 59 until it is free to write. All other scripts attempting to write are blocked at the same time.

Example 8-13 Utility that performs atomic appends to file 001. #include 002. #include 003. #include 004. #include 005. #include 006. 007. int main(int argc, char *argv[])

230 SVC 4.2.1 Advanced Copy Services 008. { 009. /* 010. * First parameter is the file 011. * Second parameter is the text 012. */ 013. 014. char *filename; // Name of file to append to 015. int fileD; // File descriptor (for file access) 016. char *text; // Text to append to the file in question 017. int textLen; // Length of the text that is appended to the file 018. 019. /* 020. * Ensure that we have the required parameters 021. */ 022. if (argc != 3) 023. { 024. printf("Usage: append \n"); 025. return -1; 026. } 027. 028. filename = argv[1]; 029. 030. /* 031. * The text appended to the file will have a "newline" appended to it 032. * We also need to include space in the buffer for the null-terminator 033. */ 034. textLen = strlen(argv[2]) + 2; 035. text = (char *) malloc((textLen)*sizeof(char)); 036. if(text == NULL) 037. { 038. printf("Could not allocate memory\n"); 039. return -2; 040. } 041. strcpy(text,argv[2]); 042. strcat(text,"\n"); 043. 044. /* 045. * Open the file that we are appending to 046. * NB: We do not attempt to create this file if it does not exist 047. */ 048. fileD = open(filename, O_WRONLY | O_APPEND, 0); 049. 050. if(fileD < 0) 051. { 052. perror("Cannot open append file for writing"); 053. return -2; 054. } 055. 056. /* 057. * Write to append file and then close the file 058. */ 059. write(fileD,text,textLen); 060. 061. close(fileD);

Chapter 8. Automation 231 062. 063. return 0; 064. }

8.4.2 Handling the SVC response

From this point on, we assume that you have an implementation that enables you to submit commands to the cluster and retrieve the response in an effective manner. The next step is to handle the responses usefully.

Handling errors The IBM TotalStorage SAN Volume Controller: Command-Line Interface User's Guide, SC26-7544, provides a list of the possible errors that can result from issuing an SVC command. It is important to recognize and handle these commands.

Every SVC cluster error has a unique code of the following format: CMMVCxxxxE for an error CMMVCxxxxW for a warning

These codes map to a single outcome, although different commands may result in identical errors. For example, the CMMVC5786E code maps to the The action failed because the cluster is not in a stable state error, which may be returned for any command that seeks to change the cluster configuration by adding or removing an object.

It is important that any automation solution at the very least reports these errors accurately because they are needed for troubleshooting. At the next level, an automation solution should be able to handle errors automatically. A large proportion of the errors have to do with invalid requests such as trying to use objects that do not exist, or trying to change the state of an object to an invalid state. These errors can be avoided by ensuring that your scripts check object states before changing them. They can also be used as sanity checks; if you encounter one of these errors, something is obviously wrong, and the user should be alerted.

svcinfo commands These commands are critical to automation because they indicate the current state of the cluster. It is important to ascertain object states before changing them if you intend to implement an enterprise-level automation solution.

If you use the SVCTools Perl module, you can access cluster object properties as a result of executing the functions that this module provides. In this section we discuss how to access these properties via the Korn shell.

We strongly recommend that commands submitted to the SVC cluster by an automation solution are submitted using the -delim parameter; in general, we use the colon character (:) as the delimiter. This is just one convention, and you can use any character that you would not otherwise expect in the output.

After the output has been captured, each line represents an object. This line can then be split into words, separated by the chosen delimiter. Each of these words represents a property.

At this point, you have access to the information you need; write your scripts to perform the processes according to your business logic.

232 SVC 4.2.1 Advanced Copy Services svctask commands When these commands succeed, the feedback is minimal. svctask commands that create objects return the ID of the created object. The format of the response is