Rediscovering Distributed Systems Steve Vinoski Basho Technologies Cambridge, MA USA http://basho.com @stevevinoski [email protected] http://steve.vinoski.net/ Thursday, October 17, 13 1 Distributed Systems are Everywhere Thursday, October 17, 13 2 Distributed Systems are Difficult Thursday, October 17, 13 3 On The Shoulders of Giants • "Distributed systems" describes an enormous history of research and practice • Dist Sys research/practice addresses many issues from many angles • Know the issues so you can choose the right trade-offs Thursday, October 17, 13 4 Scope • Way way too much to cover • This talk is based in part on my personal history and experiences • Others would give a completely different talk Thursday, October 17, 13 5 1960s Thursday, October 17, 13 6 1960s • Beginnings of concurrent systems • 1965: Dijkstra's semaphores • Beginnings of computer networking • J.C.R. Licklider's 1962 dream of an "Intergalactic Computer Network" would eventually lead to the Internet • Beginning of OO: Simula 67 Thursday, October 17, 13 7 Distributed Systems Failure • 1965 Northeast blackout affected 7 US states and Ontario • A single misconfigured relay caused massive cascading failures • Then and now: distributed systems failure is not uncommon Thursday, October 17, 13 8 Dijkstra and Multiprogramming • 1968: "The Structure of the 'THE' Multiprogramming System" • Describes a whole system designed as a set of hierarchical cooperating sequential processes • System resources shared via mutual synchronization via semaphores Thursday, October 17, 13 9 Dijkstra and Multiprogramming "At the time this was written the testing had not yet been completed, but the resulting system is guaranteed to be flawless." —E.W. Dijkstra "The Structure of the 'THE' Multiprogramming System" Thursday, October 17, 13 10 1970s Thursday, October 17, 13 11 1970s Issues • Interprocess Communication • Resource sharing • Programming languages and distributed computing • Application-to-application protocols Thursday, October 17, 13 12 Interprocess Communication (IPC) • "Interprocess Communication Facilities for Network Operating Systems", Akkoyunlu, Bernstein, Schantz, 1974 • Discusses IPC over different network topologies • Compares connection-oriented and message- oriented IPC facilities • Discusses concerns of how sender and receiver might find and identify each other Thursday, October 17, 13 13 IPC Thursday, October 17, 13 14 IPC Thursday, October 17, 13 15 Thursday, October 17, 13 16 Resource Sharing "Users and administrators of a small computer often desire more service than it can provide. In a network environment additional services can be provided to the small computer, and in turn to the users of the small computer, by one or more other computers." —Akkoyunlu, Bernstein, Schantz "Interprocess Communication Facilities for Network Operating Systems" Thursday, October 17, 13 17 Resource Sharing • "An Operational System for Computer Resource Sharing", Cosell et al., 1975 • Ideas like today's cloud computing Thursday, October 17, 13 18 Resource Sharing "Further , it was becoming clear that for many users, in particular those whose access to the network was via TIPs or other non- TENEX hosts, it should not actually matter which host provides the TENEX service so long as the users could do their computing in the manner to which they had become accustomed." —Cosell et al. "An Operational System for Computer Resource Sharing" Thursday, October 17, 13 19 Resource Sharing "A number of advantages would result from such resource sharing. The user would see TENEX as a much more accessible and reliable resource. Because he would no longer be dependent upon a single host for his computing, he would be able to access the TENEX virtual machine even when one or more of the TENEX hosts were unavailable." —Cosell et al. "An Operational System for Computer Resource Sharing" Thursday, October 17, 13 20 Distributed Processing, January 1978 Thursday, October 17, 13 21 Distributed Processing, January 1978 "A terminal with a resident text editor, whether it is provided by hardware or software, is not an example of a distributed data processing system." "If the terminal coordinates several concurrent and simultaneous remote jobs, giving each a different type of service at a different location, without human intervention, then it more closely resembles a distributed system." —P.H. Enslow, Jr. "What is a 'Distributed' Data Processing System?" Thursday, October 17, 13 22 1978: Issues in Distributed Processing "Participants generally agreed that distributed processing is made possible by the price-performance revolution in microelectronics." —Eckhouse and Stankovic "Issues in Distributed Processing - An Overview of Two Workshops" Thursday, October 17, 13 23 1978: Issues in Distributed Processing • IPC • Distributed operating systems • Distributed databases • Load balancing Thursday, October 17, 13 24 State Machines "A distributed system can be described as a particular sequential state machine that is implemented with a network of processors." Leslie Lamport http://research.microsoft.com/en-us/um/people/lamport/pubs/pubs.html#time-clocks Thursday, October 17, 13 25 Ordering of Events • "Time, Clocks, and the Ordering of Events in a Distributed System", L. Lamport, 1978 • Probably the most cited Dist Sys paper ever Thursday, October 17, 13 26 Ordering of Events • event A happens before event B if • A occurs before B in the same process • or A is a send of M from process P1, and B is the receive of M in process P2 • happens-before is written as A → B • if A → B and B → C, then A → C Thursday, October 17, 13 27 Ordering of Events • Clock Condition: • if A → B then LogicalClock(A) < LogicalClock(B) Thursday, October 17, 13 28 Ordering of Events e11 P1 1 t e21 P2 1 t e31 P3 1 t Thursday, October 17, 13 29 Ordering of Events e11 e12 P1 1 2 t e21 e22 P2 1 3 t e31 e32 P3 1 2 t Thursday, October 17, 13 30 Ordering of Events e11 e12 e13 e14 P1 1 2 3 5 t e21 e22 e23 e24 P2 1 3 4 5 t e31 e32 e33 e34 P3 1 2 3 4 t Only partial ordering, since e.g. e32 ↛ e14 Thursday, October 17, 13 31 Ordering of Events • Total ordering achieved by choosing an arbitrary ordering for processes • A ⇒ B iff • C(A) < C(B) • or, C(A) == C(B) and Pa < Pb • Remainder of paper shows how it all works with physical clocks Thursday, October 17, 13 32 Languages and Distributed Systems • Handling concurrency in programming languages • Handling distribution in programming languages • What are the best primitives in such languages? Thursday, October 17, 13 33 Communicating Sequential Processes • "Communicating Sequential Processes", Tony Hoare, 1978 • Defines "surprisingly versatile" primitives for structuring concurrent programs Thursday, October 17, 13 34 Communicating Sequential Processes • Dijkstra's guarded commands for sequential control structures • Dijkstra's parbegin to specify concurrent execution • Input and output commands to allow processes to communicate • Input commands in guards • Pattern matching on input messages Thursday, October 17, 13 35 Communicating Sequential Processes • The paper then uses these primitives to show solutions to a variety of programming problems Thursday, October 17, 13 36 Primitives for Distributed Computing • "Primitives for Distributed Computing", Barbara Liskov, 1979 • Describes distributed computing primitives, focusing on modularity and communication Thursday, October 17, 13 37 Primitives for Distributed Computing • Reasons for distributed computing: • To match distributed organizations • Reduced contention among organization divisions • Speed of access • Physical control • Better availability, reliability, extensibility • But at the time there was little experience in building distributed programs Thursday, October 17, 13 38 Primitives for Distributed Computing • Approach: extend the CLU language with distributed computing primitives • CLU supported data abstractions and objects Structured programming CLU ⇒ + Modularity Data abstraction Thursday, October 17, 13 39 Modularity • Guardian: a collection of processes and objects that provides a distributed service • Processes in same guardian share objects • Processes in different guardians communicate only via messages • Each guardian bound to a single node • Each node can host multiple guardians Thursday, October 17, 13 40 Communication • No-wait send • Receive with timeout • Messages are commands with zero or more arguments • Messages are sent to ports • Ports are named and typed, defining the messages they accept • Guardians have one or more ports Thursday, October 17, 13 41 Primitives for Distributed Computing • Paper finishes by showing an example of a transactional airline reservation system • One conclusion: more experience with distributed programming needed Thursday, October 17, 13 42 App-to-App Protocols • ARPANET, forerunner of the Internet, started operating in late 1969 • Early host-to-host protocols facilitated human-to-computer communications • Email in 1971 • FTP and interoperable Telnet in 1973 • Interest started growing in application-to- application protocols Thursday, October 17, 13 43 RFC 707 • J. E. White, RFC 707, “A High-Level Framework for Network-Based Resource Sharing, 1975 • Expressed concerns that programmers • didn't know how to write distributed applications • but did know how to write libraries • so, why not make distributed programming look just like library programming? Thursday, October 17, 13 44 RFC 674 and 707 • RFCs 674 and 707 basically define what would become the remote procedure call (RPC) • But