LIVE DISTRIBUTED OBJECTS A Dissertation Presented to the Faculty of the Graduate School of Cornell University in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy by Krzysztof Jan Ostrowski August 2008 c 2008 Krzysztof Jan Ostrowski ALL RIGHTS RESERVED LIVE DISTRIBUTED OBJECTS Krzysztof Jan Ostrowski, Ph.D. Cornell University 2008 Distributed multiparty protocols such as multicast, atomic commit, or gossip are currently underutilized, but we envision that they could be used pervasively, and that developers could work with such protocols similarly to how they work with CORBA/COM/.NET/Java objects. We have created a new programming model and a platform in which protocol instances are represented as objects of a new type called live distributed objects: strongly-typed building blocks that can be composed in a type-safe manner through a drag and drop interface. Unlike most prior object-oriented distributed protocol embeddings, our model appears to be flexible enough to accommodate most popular protocols, and to be applied uniformly to any part of a distributed system, to build not only front-end, but also back-end components, such as multicast channels, naming, or membership services. While the platform is not limited to applications based on multicast, it is replication- centric, and reliable multicast protocols are important building blocks that can be used to create a variety of scalable components, from shared documents to fault-tolerant storage or scalable role delegation. We propose a new multicast architecture compatible with the model and designed in accordance with object-oriented principles such as modu- larity and encapsulation. The architecture represents multicast objects as compositions of smaller objects, glued using a decentralized membership mechanism. The benefits include support for heterogeneous networks by allowing a single multicast protocol in- stance to simultaneously leverage different mechanisms in different parts of the Internet. Finally, we describe and evaluate a scalable reliable multicast prototype that incor- porates a novel approach to scaling with the number of protocol instances by leveraging regular patterns of overlap. The system achieves exceptional performance, scalability, and stability in the presence of perturbations, in part thanks to a novel application of techniques such as priority scheduling or pull-based processing. We describe a previ- ously unnoticed relationship between memory overheads and scheduling and the per- formance and scalability of a reliable multicast protocol. Our results demonstrate that to build a new global Web operating system that the live objects vision leads to, the distributed protocol and the local system architecture cannot be treated in isolation. BIOGRAPHICAL SKETCH Krzysztof Ostrowski is originally from Poland. He received his Bachelor of Science in Mathematics and Computer Science from the University of Warsaw in Poland in 1998, and his Master of Science in Computer Science from the same university in 2001. In August, 2008 he received his Ph.D. from Cornell University. iii To my beloved Yejin. iv ACKNOWLEDGEMENTS First and foremost, I am deeply grateful to Ken Birman for being an extremely support- ive advisor, open to my ideas and encouraging, and for enabling me to develop and to pursue my vision. Ken is a great teacher of assertiveness, optimism, and confidence in one’s work, and has motivated me to stay firmly on the course that I have taken, and to be determined in pursuing my goals despite the difficulties and obstacles I met along the way. Ken has created countless opportunities for my work to be noticed and recog- nized by people in the research community and in the industry. His critical feedback and guidance were invaluable in helping me learn how to efficiently communicate my ideas. Ken has also very much inspired me with his passion for building and measuring real systems and with his commitment to tackling concrete real-world problems, even if hard and intimidating. I especially appreciate having been able to benefit from Ken’s tremen- dous practical knowledge and experience while working together on understanding the performance of QSM and refining the design of the system. I am also profoundly grateful to Danny Dolev, whose deep insight and wisdom at both academic and personal level not only enriched my research, but also energized me with high morale. Danny has been a fantastic collaborator with an extraordinary ability to grasp the essence of one’s ideas and to quickly point out the subtleties involved in realizing them. While working together on the foundations of Properties Framework, he challenged and guided my thinking to help me steer around pitfalls; the architecture would not have been possible without him. I am tremendously lucky to have been able to work with Danny. I am also much indebted to Jong-Hoon Ahnn for all the great work he has done for the live objects platform. John has provided a great amount of feedback that has been extremely helpful in improving the stability and usability of the platform, and his help was essential to the success of our demos. He has also developed many components, without which the platform would be incomplete. I am sincerely grateful to Paul Francis, Andrew Myers, Anil Nerode, and Robbert van Renesse, who have been members of my special committee, for their many in- sightful comments and suggestions regarding my dissertation, research vision, style of presentation, and career in general. I would like to especially thank Andrew and Robbert for reading my dissertation so thoroughly; their detailed feedback has been very helpful in improving the structure and style of my writing. I would like to also thank Hussam Abu-Libdeh, Mahesh Balakrishnan, Lars Brenna, Daniel Freedman, Lakshmi Ganesh, Maya Haridasan, Chi Ho, Ingrid Jansch-Porto, Tu- dor Marian, Amar Phanishayee, Stefan Pleisch, Robbert van Renesse, Michael Siegen- thaler, Yee Jiun Song, Ymir Vifgusson, Werner Vogels, Einar Vollset, and Hakim Weath- erspoon, who currently are, or who have once been members of our group, collaborators, and visitors, for listening to my practice talks, reading drafts of my papers, and for their many helpful comments and suggestions. They have been the greatest colleagues I ever worked with, and I am proud to have been a part of the team. I would like to thank Lars Brenna, Dmitri Chmelev, Adam Davis, Richard Grandy, Ankur Khetrapal, Daniel Margo, Chuck Sakoda, Weijia Song, Zhen Xiao, as well as all v the students in CS514 in Spring 2007 and 2008, for being the users of QSM and live objects. I am especially indebted to Lars for pointing out the bugs in QSM’s support for unmanaged applications. I am grateful to Ernie Davis for believing in our technology and convincing Cornell to file patent applications for QSM and live objects, and to Kathleen Chapman for the hard work we went through together preparing these patent applications. I would like to also thank Brian Bershad, Marin Bertier, Jason Bryantt, Antonio Carzaniga, Michael Caspersen, Ranveer Chandra, Wei Chen, Gregory Chockler, Jon Currey, Alan Demers, Kathleen Fisher, Davide Frey, Johannes Gehrke, Rachid Guer- raoui, Chang Heng, Robert Hillman, Mingsheng Hong, Ray O. Johnson, Idith Kei- dar, Anne-Marie Kermarrec, Annie Liu, John Manferdelli, Milan Milenkovic, Benjamin Pierce, Josh Rattner, Fred Schneider, Gennady Staskevich, Daniel Sturman, Eric Suss, Marvin Theimer, Toste Wallmark, Calvin Wong, Zheng Zhang, and all the anonymous COMSWARE, DEPSA, ECOOP, HotOs, ICDCS, ICWS, Internet Computing, IPTPS, JWSR, NCA, NSDI, OSDI, SOSP, and STC reviewers for comments and suggestions. I would like to thank Ranveer Chandra for the opportunity to give a talk and to meet his colleagues researchers in Microsoft Research labs at Redmond. I would like to present my special gratitude to Bill Hogan for all his help and advice in administrative matters. I would like to also very much thank Ben Atkin for being my mentor at Cornell, and a great friend. Ben has been like an older brother, and made me feel at home in Ithaca. I am grateful to my parents Tadeusz and Maria for their love and sacrifice, and to my brother Piotr for being a wise, supportive and loving brother, and for being there for me whenever I needed him. Last, but not least, I would like to thank my wife Yejin for her support, for the time she spent reading early drafts of my papers and listening to my practice talks to help me improve my presentation and writing, for the many valuable technical comments she made, and most importantly, for being so wise and charming at the same time. This work was supported by DARPA/IPTO under the SRS program, by the Rome Air Force Research Laboratory, AFRL/IF, and by AFOSR (under AF TRUST). Additional support was provided by Intel and NSF. Any opinions, findings, or recommendations presented in the following pages, however, are my own, and do not necessarily reflect the views of DARPA/IPTO, AFRL, AFOSR, Intel, or NSF. vi TABLE OF CONTENTS BiographicalSketch............................... iii Dedication.................................... iv Acknowledgements ............................... v TableofContents ................................ vii ListofFigures.................................. x ListofCode................................... xiii ListofSymbols ................................. xiv 1 Introduction 1 1.1 TheLimitedAdoptionofDistributedComputing . .... 4 1.2 TheActiveWeb ............................. 5 1.3 TheEmergingParadigm ........................