Designing Enterprise Solutions With Sun™ Cluster 3.0 Sun Microsystems, Inc. 901 San Antonio Road Palo Alto, CA 94303-4900 U.S.A. 650-960-1300 ISBN 0-13-008458-1 December 2001, Revision A Send comments about this document to: [email protected] Copyright 2001 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto, CA 94303-4900 U.S.A. All rights reserved. This product or document is distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any. Third-party software, including font technology, is copyrighted and licensed from Sun suppliers. Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and other countries, exclusively licensed through X/Open Company, Ltd. Sun, Sun Microsystems, Sun BluePrints, SunUP, the Sun logo, AnswerBook, AnswerBook2, DLT, docs.sun.com, IPC, Solaris, Solstice Backup, Trusted Solaris, SunDocs, Sun Quad FastEthernet, SunFastEthernet, Sun StorEdge, SunPlex, OpenBoot, Sun Enterprise, Sun Enterprise Network Array, Sun Enterprise SyMON , Sun Fire, Sun HighGround, Starfire, iPlanet, Netra, SunTone, JumpStart, Solstice, Solstice DiskSuite, Solstice Backup, Solstice SyMON, Ultra Enterprise, Java, Jiro, JavaServer Pages, JSP, J2EE, JDBC, Sun Ray, Sun RSM Array, SunSwift, Enterprise JavaBeans, EJB, Sun HPC ClusterTools, SPARCcenter, SPARCcluster, SPARCserver, SPARCstorage, Sun Professional Services, SunSolve, SPARCcluster, PDB, Prism, RSM, and Write Once, Run Anywhere are trademarks, registered trademarks, or service marks of Sun Microsystems, Inc. in the United States and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. CMS is a trademark or registered trademark of Eastman Kodak Company in the United States and other countries. ORACLE is a registered trademark of Oracle Corporation. Netscape is a trademark or registered trademark of Netscape Communications Corporation in the United States and other countries. Legato NetWorker is a registered trademark of Legato Systems, Inc. Adobe is a registered trademark of Adobe Systems, Incorporated. The OPEN LOOK and Sun Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun holds a non-exclusive license from Xerox to the Xerox Graphical User Interface, which license also covers Sun’s licensees who implement OPEN LOOK GUIs and otherwise comply with Sun’s written license agreements. Federal Acquisitions: Commercial Software—Government Users Subject to Standard License Terms and Conditions. DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID. Please Recycle Copyright 2001 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto, CA 94303-4900 Etats-Unis. Tous droits réservés. Ce produit ou document est distribué avec des licences qui en restreignent l’utilisation, la copie, la distribution, et la décompilation. Aucune partie de ce produit ou document ne peut être reproduite sous aucune forme, par quelque moyen que ce soit, sans l’autorisation préalable et écrite de Sun et de ses bailleurs de licence, s’il y en a. Le logiciel détenu par des tiers, et qui comprend la technologie relative aux polices de caractères, est protégé par un copyright et licencié par des fournisseurs de Sun. Des parties de ce produit pourront être dérivées des systèmes Berkeley BSD licenciés par l’Université de Californie. UNIX est une marque déposée aux Etats-Unis et dans d’autres pays et licenciée exclusivement par X/Open Company, Ltd. Sun, Sun Microsystems, Sun BluePrints, SunUP,the Sun logo, AnswerBook, AnswerBook2, docs.sun.com, Solaris, Solstice Backup, Trusted Solaris, SunDocs, Sun Quad FastEthernet, SunFastEthernet, Sun StorEdge, SunPlex, OpenBoot, Sun Enterprise, Sun Enterprise Network Array, Sun Enterprise SyMON , Sun Fire, Sun HighGround, Starfire, iPlanet, Netra, SunTone, JumpStart, Solstice, Solstice DiskSuite, Solstice Backup, Solstice SyMON, Ultra Enterprise, IPC, Java, Jiro, JavaServer Pages, JSP,J2EE, JDBC, Sun Ray, Sun RSM Array, Enterprise JavaBeans, EJB, Sun HPC ClusterTools, SPARCcenter, SPARCcluster, SPARCserver, SPARCstorage, Sun Professional Services, SunSolve, SPARCcluster, PDB, Prism, RSM, and Write Once, Run Anywhere so nt des marques de fabrique ou des marques déposées, ou marques de service, de Sun Microsystems, Inc. aux Etats-Unis et dans d’autres pays. Toutes les marques SPARC sont utilisées sous licence et sont des marques de fabrique ou des marques déposées de SPARC International, Inc. aux Etats-Unis et dans d’autres pays. Les produits portant les marques SPARC sont basés sur une architecture développée par Sun Microsystems, Inc. ORACLE est une marque déposée registre de Oracle Corporation. Netscape est une marque de Netscape Communications Corporation aux Etats-Unis et dans d’autres pays. Legato NetWorker est une marque de fabrique ou une marque déposée de Legato Systems, Inc. Adobe est une marque enregistree de Adobe Systems, Incorporated. L’interface d’utilisation graphique OPEN LOOK et Sun a été développée par Sun Microsystems, Inc. pour ses utilisateurs et licenciés. Sun reconnaît les efforts de pionniers de Xerox pour la recherche et le développement du concept des interfaces d’utilisation visuelle ou graphique pour l’industrie de l’informatique. Sun détient une licence non exclusive de Xerox sur l’interface d’utilisation graphique Xerox, cette licence couvrant également les licenciés de Sun qui mettent en place l’interface d’utilisation graphique OPEN LOOK et qui en outre se conforment aux licences écrites de Sun. LA DOCUMENTATION EST FOURNIE “EN L’ETAT” ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORIE EPAR LA LOI APPLICABLE, Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L’APTITUDE A UNE UTILISATION PARTICULIERE OU A L’ABSENCE DE CONTREFAÇON. Contents Preface xxi Acknowledgements xxvii 1. Cluster and Complex System Design Issues 1 Business Reasons for Clustered Systems 2 Risk Assessment 2 Cost Estimation 3 Failures in Complex Systems 5 Fault Detection 7 Probes 9 Latent Faults 10 Fault Isolation 10 Fault Reporting 10 Fault Containment 12 Reconfiguration Around Faults 13 Fault Prediction 14 Data Synchronization 14 Data Uniqueness 15 Complexity and Reliability 15 iii Synchronization Techniques 16 Microprocessor Cache Coherency 16 Kernel-Level Synchronization 17 Application-Level Synchronization 18 Synchronization Consistency Failures 18 Two-Phase Commit 18 Locks and Lock Management 19 Lock Performance 20 Arbitration Schemes 20 Asymmetric Arbitration 20 Symmetric Arbitration 21 Voting and Quorum 22 Data Caches 22 Cost and Latency Trade-Off 23 Cache Types 24 Cache Synchronization 25 Timeouts 26 Stable System 27 Unstable System 28 Stability Problems 28 Failures in Clustered Systems 29 Split Brain 29 Multiple Instances 29 Amnesia 30 Summary 30 2. Enterprise Cluster Computing Building Blocks 31 Data Repositories and Infrastructure Services 32 iv Designing Enterprise Solutions With Sun Cluster 3.0 File Services 32 NFS 32 SAMBA 33 Database Services 33 HA-Oracle 34 Oracle 8i OPS and Oracle 9i RAC 34 Messaging Services 36 Name Services 36 DNS 36 LDAP 37 NIS and NIS+ 37 Business Logic and Application Service 38 Packaged Business Solutions 38 Application Servers 39 User Access Services: Web Farms 42 Compute Clusters 45 Distributed Clusters 45 Parallel Processing 45 High-Performance Computing 46 Sun HPC Clusters 46 Sun Cluster Runtime Environment 47 Sun MPI Communications Library 47 Sun Parallel File System 47 Sun Scalable Scientific Subroutine Library 47 Prism Parallel Development Environment 48 Sun Grid Engine Software 49 Contents v Technologies for Building Distributed Applications 49 CORBA 49 JXTA 50 3. Sun Cluster 3.0 Architecture 51 System Architecture 52 Enterprise Infrastructure 52 Service Point Architecture 53 Fault Tolerant Systems 53 High Availability Versus Disaster Recovery 53 Data Deletion and Corruption Recovery 56 Kernel Infrastructure 57 Kernel Framework 58 Replica Management 59 Mini-Transactions 60 System Features 61 Storage Topologies 62 Clustered Pair Topology 62 N+1 Topology 63 Pair+M Topology 64 Cluster Device Connectivity 65 Global Devices 66 Primary and Secondary I/O Paths 66 Device ID 67 Namespace 68 Practical Uses 69 Global File Service 69 Application Access 70 vi Designing Enterprise Solutions With Sun Cluster 3.0 Client/Server Model 71 Read and Write Implementation 71 I/O Parallelism 73 File and Attribute Caches 73 CFS Mounting 75 Application Binaries, Data, and Logs 77 Application Performance 78 Node Separation Performance Impact 79 CFS Versus NFS 80 Global Networking Service 81 Packet Distribution Mechanisms 83 Advantages 84 Client Connection Recovery After a GIN Node Failure 84 Private Interconnects 85 Traffic