US007290102B2 (12) United States Patent (10) Patent N0.: US 7,290,102 B2 Lubbers et a]. (45) Date of Patent: Oct. 30, 2007 (54) POINT IN TIME STORAGE COPY 5,184,281 A 2/1993 Samarov et al. 5,513,314 A 4/1996 Kandasamy et al. (75) Inventors: Clark E. Lubbers, Colorado Springs, 5,815,371 A 9/ 1998 Jeiffies et a1~ CO (US); James M. Reiser, Colorado 5,815,649 A 9/1998 Utter et al' Springs’ CO (Us); Anuja Korgaonkars 5,822,777 A 10/1998 Les-hem et al. Colorado Springs CO Randy L 5,832,222 A 11/1998 DZladosZ et al. Roberson New £30m Richeg/ FL (US)_' 5,835,700 A 11/1998 Carbonneau et al. ’ ’ ’ 5,923,876 A 7/1999 Teague Robert G- Bean, Monument, CO (Us) 5,987,622 A 11/1999 L0 Verso et al. 5,996,089 A 11/1999 Mann et a1. (73) Assignee: Hewlett-Packard Development 6,033,639 A 3/2000 Schmidt et a1, Company, LP, Houston, TX (US) (Continued) ( * ) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 OTHER PUBLICATIONS U-S-C- 154(1)) by 360 days- Smart Storage Inc., “SmartStor In?NetTM; Virtual Storage for Today’s E-economy,” Sep. 2000. (21) Appl. N0.: 11/081,061 _ (Contlnued) (22) Filed: Mar‘ 15’ 2005 Primary ExaminerAiary Portka (65) Prior Publication Data (57) ABSTRACT US 2005/0160243 A1 Jul. 21, 2005 A storage system permits Virtual storage of user data by Related US. Application Data implementing a logical disk mapping structure that provides (63) Continuation of application NO‘ 10/080 961 ?led on access to user data stored on physical storage media and Oct 22 2001 HOW Pat NO 6 915 392 Winch is a methods for generating point-in-time copies, or snapshots, con?nue’ltion_il’l_ an of 2'‘ li'cat’ion ’NO ’09/872 597 of logical disks. A snapshot logical disk is referred to as a ?led on Jun 1 5001 noglppat NO 6961838 ’ ’ predecessor logical disk and the original logical disk is ' ’ ’ ' ' ’ ’ ' referred to as a successor logical disk. Creating a snapshot (51) 1111.0. 1nvo' 1 ves creatin'gPd re ecessor lg'o 1ca ld'k1s maPP'gd 1n ata G06F 12/16 (200601) structures and populating the data structures With metadata G06F 1 7/30 (200601) that maps the predecessor logical disk to the user data stored (52) Us Cl 711/162 707/204 711/206 on physical media. Logical disks include metadata that 58 F,‘ I'd "" "" "s h ’ ’ N indicates Whether user information is shared between logical ( ) gm 01. $551 ?cla It?“ earcl t """"" one disks. Multiple generations of snapshots may be created, and ee app 1ca Ion e or Comp 6 e Seam 15 my‘ user data may be shared between these generations. Methods (56) References Cited are disclosed for maintaining data accuracy When Write l/O operations are directed to a logical disk. U.S. PATENT DOCUMENTS 5,109,318 A 4/1992 Funari et al. 17 Claims, 15 Drawing Sheets K- 501 | RSS ID I PHYSICAL MEMBER SELECTION RAIDINFO STATE LMAP POINTER ~._\ , STATE LMAPPOINTER (- 50a ,./" STATE LMAP POINTER RSSS STATE LMAP POINTER R5EG# STATE RSD POINTER STATE STATE LMAP POINTER ' RSEGTI STATE RSO POINTER P$EG# PSEG# STATE LMAF‘ POINTER RSEGII STATE RSO POINTER PSEG# PSEGit STATE LMAP POINTER : : PSEG‘t P5561‘ STATE LMAP POINTER I D I PSEG# PSEG# STATE LMAP POINTER wm :. :- STATE LMAP POINTER PsEOrr v.—l I I I ‘ ' ' - RSD STATE LMAP POINTER * ' STATE LMAPPOINTER K 505 STATE LMAF' POINTER US 7,290,102 B2 Page 2 US. PATENT DOCUMENTS 2002/0019923 A1 2/2002 Reuter 2002/0048284 A1 4/2002 Moulton 6,073,209 A 6/2000 Bergsten 2002/0188800 A1 12/2002 TomasZeWski 6,148,414 A 11/2000 Brown et 31- 2003/0051109 A1 3/2003 Cochran 6,161,192 A 12/2000 Lubbérs 2003/0056038 A1 3/2003 Cochran 6,170,063 B1 V2001 Goldlng 2003/0063134 A1 4/2003 Lord 6,188,973 B1 2/2001 Martinez et al. 2003/0074492 A1 4/2003 Cochran 6,237,112 B1 5/2001 YOO eta1~ 2003/0079014 A1 4/2003 Lubbers 6,243,790 B1 6/2001 Yorimitsu 2003/0079074 A1 4/2003 Sicola 6,260,120 B1 7/2001 Blumenau et al. 2003/00790g2 A1 4/2003 Sicola 6,266,721 B1 7/2001 Sheikh et 31 2003/0079083 A1 4/2003 Lubbers 6,278,609 B1 8/2001 Suzuki et 81- 2003/0079102 A1 4/2003 Lubbers 6,282,094 B1 8/2001 L0 eta1~ 2003/0079156 A1 4/2003 Sicola 6,282,096 B1 8/2001 Lo et 31- 2003/0084241 A1 5/2003 Lubbers 6,282,610 B1 8/2001 Bergsten 2003/0101318 A1 5/2003 Kaga 6,295,578 B1 9/2001 Dimitroff 2003/0110237 A1 6/2003 Kitamura 6,397,293 B2 5/2002 Shrader 2003/0126315 A1 7/2003 Tan 6,487,636 B1 11/2002 Dolphin 2003/0126347 A1 7/2003 Tan 6,490,122 B1 12/2002 Holmquist 2003/0140191 A1 7/2003 McGoWen 6,493,656 B1 12/2002 Houston 2003/0145045 A1 7/2003 Pellegrino 6,505,268 B1 V2003 Schultz 2003/0145130 A1 7/2003 Schultz 6,523,749 B2 2/2003 Reasoner 2003/0170012 A1 9/2003 Cochran 6,546,459 B2 4/2003 Rust 2003/0177323 A1 9/2003 Popp 6,560,673 B2 5/2003 Elliot 2003/0187847 A1 10/2003 Lubbers 6,587,962 B1 7/2003 Hepner 2003/0187947 A1 10/2003 Lubbers 6,594,745 B2 7/2003 Grover 2003/0188085 A1 10/2003 Arakawa 6,601,187 B1 7/2003 Sicola 2003/0188114 A1 10/2003 Lubbers 6,606,690 B2 8/2003 Padovano 2003/0188119 A1 10/2003 Lubbers 6,609,145 B1 8/2003 Thompson 2003/0188153 A1 10/2003 Dcinofr 6,629,108 B2 9/2003 Frey 2003/0188218 A1 10/2003 Lubbers 6,629,273 B1 9/2003 Patterson 2003/0188229 A1 10/2003 Lubbers 6,643,795 B1 11/2003 $100111 2003/0188233 A1 10/2003 Lubbers 6,647,514 B1 11/2003 Umberger 2003/0191909 A1 10/2003 Asano 6,658,590 B1 12/2003 Sicola 2003/0191919 A1 10/2003 Sato 6,663,003 B2 12/2003 Johnson 2003/0196023 A1 10/2003 Dickson 6,681,308 B1 V2004 Dallmann 2003/0212781 A1 11/2003 Kaneda 6,708,285 B2 3/2004 Old?eld 2003/0229651 A1 12/2003 MiZuno 6,715,101 B2 3/2004 Old?eld 2003/0236953 A1 12/2003 Giicfr 6,718,404 B2 4/2004 Reuter 2004/0019740 A1 1/2004 Nakayama 6,718,434 B2 4/2004 Veitch 2004/0022546 A1 2/2004 Cochran 6,721,902 B1 4/2004 Cochran 2004/0024838 A1 2/2004 Cochran 6,725,393 B1 4/2004 Pellegrlno 2004/0024961 A1 2/2004 Cochran 6,742,020 B1 5/2004 Dimitroff 2004/0030727 A1 2/2004 Armangau 6,745,207 B2 6/2004 Reuter 2004/0030846 A1 2/2004 Armangau 6,763,409 B1 7/2004 Elliot 2004/0049634 A1 3/2004 Cochran 6,772,231 B2 8/2004 Reuter 2004/0078638 A1 4/2004 Cochran 6,775,790 B2 8/2004 Reuter 2004/0078641 A1 4/2004 Fleischmann 6,795,904 B1 9/2004 Kamvysselis 2004/0128404 A1 7/2004 Cochran 6,802,023 B2 10/2004 Old?eld 2004/0168034 A1 8/2004 Sigeo 6,807,605 B2 10/2004 Umberger 2004/0215602 A1 10/2004 Cioccarelli 6,817,522 B2 11/2004 Brignone 2004/0230859 A1 11/2004 Cochran 2,253,322 5; 15588;‘ gaget?mn 2004/0267959 A1 12/2004 Cochran , , am e 6,842,833 B1 1/2005 Phillips OTHER PUBLICATIONS 6,845,403 B2 1/2005 Chadalapaka Compaq Computer Corporation, “The Compaq Enterprise Network 2002/0019863 A1 2/2002 Reuter Storage Architecture: An Overview,” May 2000. 2002/0019908 A1 2/2002 Reuter Compaq Computer Corporation, “Compaq Storage Works” Data 2002/0019920 A1 2/2002 Reuter Replication Manager HSG80 ACS V8.5P Operations Guide, Mar. 2002/0019922 A1 2/2002 Reuter 2000. U.S. Patent 0a. 30, 2007 Sheet 1 0f 15 US 7,290,102 B2 44069.. xwa1| $069 , xwE 130604 v56 h.QI U.S. Patent 0a. 30, 2007 Sheet 2 0f 15 US 7,290,102 B2 203[ ' 301 303 213.r\\ FIG.3 211 U.S. Patent 0a. 30, 2007 Sheet 3 0f 15 US 7,290,102 B2 Pom Q wow wow Now U.S. Patent 0a. 30, 2007 Sheet 4 0f 15 US 7,290,102 B2 wwwm IO OC _Emmi3w? E252”.05mmmmmsmg$0.9112955mm .Qwm .Emma :823mm".owm1526a55w#wwwm Emmanew?‘x5251Qwm35wEmma Emma“28mmun0\ x3260E55“6mm”.owmM55 F2555600mm“Emma. 8m\ ,. Pom\t U.S. Patent 0a. 30, 2007 Sheet 7 0f 15 US 7,290,102 B2 905 CREATE PREDECESSOR PLDMC 1 9O QUIESCE WRITE l/O OPERATIONS TO SUCCESSOR LOGICAL DISK I Y FLUSH SUCCESSOR LOGICAL DISK CACHE DATA L 920 SET SHARE BITS IN SUCCESSOR PLDMC _I__95 CREATE LZMAP. LMAP FOR PREDECESSOR LOGICAL DISK 1 930 PREDECESSORPOPULATE // LZMAP WITH PREDECESSOR LMAP POINTERS 1 93 5 POPULATE I PRED EC ESSOR LMAP WITH SUCCESSOR LMAP CONTENTS 1 940 SET SHARE BITS I IN SUCCESSOR / AND PREDECESOR LMAPS I 945 SUCCESSORUNQUIESCE / LOGICAL DISK Fig. 9 U.S. Patent 0a. 30, 2007 Sheet 8 0f 15 US 7,290,102 B2 _ Ss Sp S5 Sp Ss Sp Ss Sp __ FIG. 10 U.S. Patent 0a. 30, 2007 Sheet 9 0f 15 US 7,290,102 B2 WI ___Ss Sp Ss Sp Ss Sp Ss Sp _ FIG.