In Reference to RPC: It’s Time to Add Distributed Memory Stephanie Wang, Benjamin Hindman, Ion Stoica UC Berkeley ACM Reference Format: D (e.g., Apache Spark) F (e.g., Distributed TF) Stephanie Wang, Benjamin Hindman, Ion Stoica. 2021. In Refer- A B C i ii 1 2 3 ence to RPC: It’s Time to Add Distributed Memory. In Workshop RPC E on Hot Topics in Operating Systems (HotOS ’21), May 31-June 2, application 2021, Ann Arbor, MI, USA. ACM, New York, NY, USA, 8 pages. Figure 1: A single “application” actually consists of many https://doi.org/10.1145/3458336.3465302 components and distinct frameworks. With no shared address space, data (squares) must be copied between 1 Introduction different components. RPC has been remarkably successful. Most distributed ap- Load balancer Scheduler + Mem Mgt plications built today use an RPC runtime such as gRPC [3] Executors or Apache Thrift [2]. The key behind RPC’s success is the Servers request simple but powerful semantics of its programming model. client data request cache In particular, RPC has no shared state: arguments and return Distributed object store values are passed by value between processes, meaning that (a) (b) they must be copied into the request or reply. Thus, argu- ments and return values are inherently immutable. These Figure 2: Logical RPC architecture: (a) today, and (b) with a shared address space and automatic memory management. simple semantics facilitate highly efficient and reliable imple- mentations, as no distributed coordination is required, while then return a reference, i.e., some metadata that acts as a remaining useful for a general set of distributed applications. pointer to the stored data. The RPC caller can then use the The generality of RPC also enables interoperability: any ap- reference to retrieve the actual value when it needs it, or plication that speaks RPC can communicate with another pass the reference on in a subsequent RPC request. application that understands RPC. While a shared address space can eliminate inefficient or However, the lack of shared state, and in particular a shared unnecessary data copies, doing it at the application level address space, deters data-intensive applications from using places a significant burden on the application programmer RPC directly. Pass-by-value works well when data values to manage the data. For example, the application must decide are small. When data values are large, as is often the case when some stored data is no longer used and can be safely with data-intensive computations, it may cause inefficient deleted. This is a difficult problem, analogous to manual data transfers. For example, if a client invokes G = 5 ¹º then memory management in a non-distributed program. ~ =6¹Gº, it would have to receive G and copy G to 6’s executor, Thus, instead of using RPC directly, distributed data-intensive even if 5 executed on the same server as 6. applications tend to be built on top of specialized frameworks There has been more than one proposal to address this like Apache Spark [43] for big data processing or Distributed problem by introducing a shared address space at the ap- TensorFlow [6] for machine learning. These frameworks han- plication level [14]. The common approach is to enable an dle difficult systems problems such as memory management RPC procedure to store its results in a shared data store and on behalf of the application. However, with no common foun- dation like RPC, interoperability between these frameworks Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies is a problem. Resulting applications resemble Figure 1, where are not made or distributed for profit or commercial advantage and that some components communicate via RPC and others commu- copies bear this notice and the full citation on the first page. Copyrights nicate via a framework-specific protocol. This often results for third-party components of this work must be honored. For all other in redundant copies of the same data siloed in different parts uses, contact the owner/author(s). of an application. HotOS ’21, May 31-June 2, 2021, Ann Arbor, MI, USA © 2021 Copyright held by the owner/author(s). We argue that RPC itself should be extended with a shared ACM ISBN 978-1-4503-8438-4/21/05. address space and first-class references. This has two appli- https://doi.org/10.1145/3458336.3465302 cation benefits: 1) by allowing data-intensive applications to 191 HotOS ’21, May 31-June 2, 2021, Ann Arbor, MI, USA Stephanie Wang, Benjamin Hindman, Ion Stoica be built directly on RPC, we promote interoperability, and Ref r A first-class type. Points to a value which may not 2) by shifting automatic memory management to a common exist yet and may be located on a remote node. system, we can reduce duplicated work between specialized shared(Ref Returns a copy of r that can be shared with another frameworks. Indeed, we are already starting to see this re- r)!SharedRef client by passing to an RPC. alized by the latest generation of data systems, including f.remote(Ref Invoke f. Pass the argument by reference: the Ciel [27], Ray [26], and Distributed PyTorch [4]. All imple- r) ! Ref executor receives the dereferenced argument. ment an RPC-like interface with a shared address space. Our Returns a Ref pointing to the eventual reply. f.remote( f aim is to bring attention to the common threads and chal- Invoke . Pass the argument by shared reference: the SharedRef r) executor receives the corresponding Ref. Returns lenges of these and future systems. ! Ref a Ref pointing to the eventual reply. At first glance, introducing a shared address space toRPC get(Ref r) ! Dereference r. This blocks until the underlying seems to directly contradict the original authors, who argued Val value is computed and fetched to local memory. that doing so would make the semantics more complex and delete(Ref r) Called implicitly when r goes out scope. The client the implementation less efficient [12]. We believe, however, may not use r in subsequent API calls. that these concerns stem not from a shared address space Table 1: A language-agnostic pass-by-reference API. itself, but rather from supporting mutability. By adding an immutable shared address space, we can preserve RPC’s orig- A common asynchronous API has each RPC invocation inal semantics while enabling data-intensive applications. immediately return a future, or a pointer to the eventual re- ply [10, 25]. In Section 3, we explain how this in conjunction Immutability simplifies the system design, but what con- with a reference, i.e. a pointer to a possibly remote value, gives cretely would supporting a shared address space require? the system insight into the application, by giving a view of To answer this question, we first consider the design of RPC the client’s future requests. We will assume a futures API but systems today. While RPC is often assumed to be a point- focus primarily on the introduction of first-class references. to-point communication between the caller and a specific callee, today’s RPC systems look more like Figure 2a: a load First-class references. A first-class primitive is one that balancer schedules an RPC to one of several servers (which is part of the system API. For references, it means that the can themselves act as an RPC client). Even the original au- RPC system is involved in the creation and destruction of thors of RPC provided such an option, known as dynamic all clients’ references. Compared to the original RPC pro- binding [12]. Extending this architecture with an immutable posal [12], there are three key differences in the API (Table 1): shared address space would mean: (1) augmenting each “ex- First, all RPC invocations return a reference (of type Ref). ecutor” with a local data store or cache, and (2) enhancing The caller can dereference an RPC’s reply when it needs it the load balancer to be memory-aware (Figure 2b). by calling get. We choose to have all RPC invocations return a reference so that the application never needs to decide In the rest of this paper we explain what it means to add whether an executor should return by value or by reference. first-class support for immutable shared state and pass-by- Instead, this is decided transparently by the system. Refer- reference semantics to RPC. We give examples of applications ences are logical, so a system implementation can choose to that already rely on this interface today. Then, drawing from pass back all replies by value, by reference, or both, e.g., de- recent data systems, we discuss the challenges and design pending on the size of the reply. A future extension to the API options in implementing such an interface. could allow applications to control this decision, if needed. 2 API Second, the client can pass a reference as an RPC argument, in addition to normal values. There are two options for deref- There are two goals for the API: (1) It should preserve the erencing Ref arguments. If a function with the signature simple semantics of RPC, and (2) It should allow the system f(int x) is passed a Ref argument, the system implicitly to manage memory on behalf of the application. We use im- dereferences the Ref to its int value before dispatching f to mutability to achieve the former. It is the simplest approach its executor. Thus, the executor never sees the Ref.