Distributed Shared Memory for Large-Scale Dynamic Systems Vincent Gramoli To cite this version: Vincent Gramoli. Distributed Shared Memory for Large-Scale Dynamic Systems. Networking and Internet Architecture [cs.NI]. Université Rennes 1, 2007. English. tel-00491439 HAL Id: tel-00491439 https://tel.archives-ouvertes.fr/tel-00491439 Submitted on 11 Jun 2010 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. INRIA Research Centre Rennes Universite´ de Rennes 1 INRIA Futurs Saclay Distributed Shared Memory for Large-Scale Dynamic Systems A dissertation presented to Universite´ de Rennes 1 in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the subject of Computer Science by Vincent Gramoli defended on Thursday, November 22nd 2007 in front of the jury composed of: Antonio Fernandez´ Professor Examiner Roy Friedman Professor Referee Anne-Marie Kermarrec Senior Researcher Examiner Michel Raynal Professor Advisor Pierre Sens Professor President Alex Shvartsman Professor Referee Year 2007 Ecole´ Doctorale Matisse Numero´ d’Ordre : 3637 ii This thesis focuses on newly arising challenges in the context of data sharing due to the recent scale shift of distributed systems. Distributed systems keep enlarging very rapidly. Not only more persons use their computer to communicate all over the world, but the amount of individual objects that get connected is increasing. Such large scale systems experience an inherent dynamism due to the unpredictability of the users behaviors. This drawback prevents traditional solutions from being adapted to this challenging context. More basically, this affects the communication among distinct computing entities. This thesis investigates the existing research work and proposes research suggestions to solve a fundamental issue, the distributed shared memory problem, in such a large scale and inherently dynamic environment. Keywords: Distributed shared memory, quorum, atomicity, consistency, reconfiguration, dynamic, large-scale. iii To my wife and my family vi Acknowledgments I wish to thank especially professor Michel Raynal who accepted to become my advisor while my subject was partially defined. I wish to thank Anne-Marie Kermarrec that encouraged my research at any time, I owe her a lot. Thanks to Alex Shvartsman for having accepted to review my thesis and with whom I lived a great research experience at very stimulating locations. Thanks to Roy Friedman to review my thesis and to Antonio Fernandez´ and Pierre Sens, who accepted to exam- ined my PhD work. I am grateful to Carole Delporte, Hugues Fauconnier, Matthieu Latapy, and Pierre Fraigniaud for having taught me very interesting fields of computer science and helping me to find my path into research. Also, I would like to thank Emmanuelle Anceaume, Maria Gradi- nariu, and Michel Hurfin with whom I discovered Peer-to-Peer systems, and those who enlightened me about the French doctoral status. Thanks to my friends Nic, Andri, Piotr, Monika, Alexis, and Kishori from Storrs; Achour, Aline, Corentin, Erwan, Etienne, Fabrice, Franc¸ois, Gilles, Marin, and Yann, the task force of the ASAP group; and Julien, Ludovic, Jean-Marie, Audrey, and Jean- Pierre from Brittany. Last but not least, thanks to all my co-authors who accepted working with me including Seth, Grisha, Toni, Achour, Ernesto, Bruno, Mark.´ vii viii Citations to Previously Published Work The essential of Chapter 2 appeared in the following papers: ”Reconfigurable Distributed Storage for Dynamic Networks”, G. Chockler, S. Gilbert, V. Gramoli, P. M. Musial, and A. A. Shvartsman, Proceedings of the OPODIS, 2005. ”Memoire´ partagees´ distribuees´ pour systemes` dynamiques a` grande echelle”,´ V.Gramoli, Proceedings of Journees´ Doctorales en Informatique et Reseaux,´ 2007. The most important part of Chapter 3 appeared in the following papers: ”Peer-to-Peer Architecture for Self-* Atomic Memory”, E. Anceaume, M. Gradinariu, V. Gramoli, and A. Virgillito, Proceedings of ISPAN, 2005. ”SQUARE: A Scalable Quorum-based Atomic Memory with Local Reconfiguration”, V. Gramoli, E. Anceaume, A. Virgillito, Proceedings of the ACM SAC, 2007 and IRISA technical report number 1805. Portions of Chapter 4 appeared or will appear in the following papers: ”Core Persistence in Peer-to-Peer Systems: Relating Size to Lifetime”, V. Gramoli, A.-M. Kermarrec, A. Mostefaoui,´ M. Raynal, and B. Sericola, Proceedings of the On The Move Workshops, 2006 and IRISA technical report number 1799. ”Timed Quorum Systems for Large-Scale and Dynamic Environments”, V. Gramoli and M. Raynal, Proceedings of the OPODIS, 2007. Portions of Chapter 4 and almost the whole Appendix A appeared in the following paper: ”Distributed Slicing in Dynamic Systems”, A. Fernandez,´ V. Gramoli, E. Jimenez,´ A.-M. Kermarrec, M. Raynal, Proceedings of the ICDCS, 2007 and INRIA technical report number 6051. Electronic preprints are available on the Internet at the following URL: http://www.irisa.fr/prive/vgramoli/php/pub irisa year.php ix x Contents Introduction 1 1 Preliminaries 5 1.1 Quorums at the Heart of Consistency ......................... 5 1.2 Shared Memory Models ............................... 6 1.3 General System Model ................................ 7 2 Facing Dynamism 9 2.1 All-to-all Operation and Reconfiguration ....................... 11 2.1.1 Quorum-based Read/Write Operations .................... 13 2.1.2 Quorum System Reconfiguration ....................... 16 2.1.3 Independence of Read/Write Operations ................... 18 2.2 Decentralized Reconfiguration ............................ 19 2.2.1 Coupling Installation and Upgrade of Configurations ............ 20 2.2.2 Deciding upon the Final Configuration using Paxos ............. 21 2.3 Dynamic Distributed Shared Memory (benefiting from Consensus) ......... 25 2.3.1 Read and Write Operations using Global Knowledge ............ 27 2.3.2 Reconfiguration by Replacement ....................... 28 2.3.3 Safety Proof: Implementing Atomicity ................... 31 2.3.4 Liveness Proof: Speeding up Reconfiguration to strengthen Fault Tolerance 33 2.3.5 Experimental Analysis of Reconfiguration .................. 37 2.4 Discussion and Conclusion .............................. 41 2.4.1 Communication Overhead .......................... 41 2.4.2 Conclusion .................................. 41 3 Facing Scalability 43 3.1 Structure of Quorum Systems ............................. 44 3.1.1 Single-Point of Failure ............................ 44 3.1.2 Replicating the Intersection ......................... 46 3.2 Communication Structure of Quorum Systems .................... 48 3.2.1 Non-Adaptive Quorum Probe vs. Adaptive Quorum Probe ......... 49 3.2.2 Reparation of Accumulating Failures .................... 50 3.3 Scalable Distributed Shared Memory (benefiting from Locality) .......... 51 xi 3.3.1 Trading Availability with Dynamism ..................... 51 3.3.2 Congestion Avoidance ............................ 53 3.3.3 Read and Write Operation using Local Knowledge ............. 59 3.3.4 Self-Adaptiveness to Unpredictability .................... 63 3.3.5 Correctness Proof of a Scalable DSM .................... 65 3.3.6 Peer-to-Peer Simulation Study ........................ 66 3.4 Discussion and Conclusion .............................. 71 3.4.1 Quorum Access in Ad Hoc Networks .................... 71 3.4.2 Limitations of Square ............................. 72 3.4.3 Conclusion .................................. 73 4 Facing Scalability and Dynamism 75 4.1 Probabilistic Guarantees ............................... 76 4.1.1 Probabilistic Consistency ........................... 76 4.1.2 Probabilistic Quorum System ........................ 79 4.1.3 Probabilistic Weak Quorum System ..................... 81 4.1.4 Probabilistic Quorum Systems for Dynamic Settings ............ 82 4.2 Avoiding Quorum System Reconfiguration ...................... 85 4.2.1 Structureless Quorum System ........................ 85 4.2.2 Timed Quorum System ............................ 87 4.3 Scalable Dynamic Distributed Shared Memory (benefiting from Prototypical Gossip) 90 4.3.1 Model and Definitions ............................ 91 4.3.2 Disseminating Memory using Underlying Gossip .............. 92 4.3.3 Correctness and Analysis of a Scalable and Dynamic DSM ......... 98 4.3.4 Exact Probability for Practical Measurements ................103 4.4 Discussion and Conclusion ..............................108 4.4.1 Approximating the System Size .......................108 4.4.2 Modeling the Churn in Dynamic Systems ..................109 4.4.3 Conclusion ..................................109 Conclusion and Open Issues 111 A Distributed Slicing 117 A.1 Introduction ......................................117 A.1.1 Context and Motivations ...........................117 A.1.2 Contributions .................................118 A.1.3 Related Work .................................119 A.1.4 Outline ....................................120 A.2 Model and Problem Statement ............................120 A.2.1 System Model ................................120