DOCSLIB.ORG

Synapse: a Microservices Architecture for Heterogeneous-Database Web Applications

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Synapse: a Microservices Architecture for Heterogeneous-Database Web Applications Synapse: A Microservices Architecture for Heterogeneous-Database Web Applications Nicolas Viennot, Mathias Lecuyer,´ Jonathan Bell, Roxana Geambasu, and Jason Nieh fnviennot, mathias, jbell, roxana, [email protected] Columbia University Abstract added features, such as recommendations for what to search for, what to buy, or where to eat. Agility and ease of inte- The growing demand for data-driven features in today’s Web gration of new data types and data-driven services are key applications – such as targeting, recommendations, or pre- requirements in this emerging data-driven Web world. dictions – has transformed those applications into complex Unfortunately, creating and evolving data-driven Web ap- conglomerates of services operating on each others’ data plications is difficult due to the lack of a coherent data without a coherent, manageable architecture. We present architecture for Web applications. A typical Web applica- Synapse, an easy-to-use, strong-semantic system for large- tion starts out as some core application logic backed by a scale, data-driven Web service integration. Synapse lets in- single database (DB) backend. As data-driven features are dependent services cleanly share data with each other in added, the application’s architecture becomes increasingly an isolated and scalable way. The services run on top of complex. The user base grows to a point where performance their own databases, whose layouts and engines can be com- metrics drive developers to denormalize data. As features are pletely different, and incorporate read-only views of each removed and added, the DB schema becomes bloated, mak- others’ shared data. Synapse synchronizes these views in ing it increasingly difficult to manage and evolve. To make real-time using a new scalable, consistent replication mech- matters worse, different data-driven features often require anism that leverages the high-level data models in popular different data layouts on disk and even different DB engines. MVC-based Web applications to replicate data across het- For example, graph-oriented DBs, such as Neo4j and Titan, erogeneous databases. We have developed Synapse on top optimize for traversal of graph data and are often used to im- of the popular Web framework Ruby-on-Rails. It supports plement recommendation systems [7]; search-oriented DBs, data replication among a wide variety of SQL and NoSQL such as Elasticsearch and Solr, offer great performance for databases, including MySQL, Oracle, PostgreSQL, Mon- textual searches; and column-oriented DBs, such as Cassan- goDB, Cassandra, Neo4j, and Elasticsearch. We and others dra and HBase, are optimized for high write throughput. have built over a dozen microservices using Synapse with Because of the need to effectively manage, evolve, and great ease, some of which are running in production with specialize DBs to support new data-driven features, Web ap- over 450,000 users. plications are increasingly built using a service-oriented ar- 1. Introduction chitecture that integrates composable services using a va- riety and multiplicity of DBs, including SQL and NoSQL. We live in a data-driven world. Web applications today – These loosely coupled services with bounded contexts are from the simplest smartphone game to the Web’s most com- called microservices [29]. Often, the same data, needed by plex application – strive to acquire and integrate data into different services, must be replicated across their respective new, value-added services. Data – such as user actions, so- DBs. Replicating the data in real time with good semantics cial information, or locations – can improve business rev- across different DB engines is a difficult problem for which enues by enabling effective product placement and targeted no good, general approach currently exists. advertisements. It can enhance user experience by letting We present Synapse, the first easy-to-use and scalable applications predict and seamlessly adapt to changing user cross-DB replication system for simplifying the develop- needs and preferences. It can enable a host of useful value- ment and evolution of data-driven Web applications. With Synapse, different services that operate on the same data but Permission to make digital or hard copies of all or part of this work for personal or demand different structures can be developed independently classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation and with their own DBs. These DBs may differ in schema, on the first page. Copyrights for components of this work owned by others than ACM indexes, layouts, and engines, but each can seamlessly in- must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a tegrate subsets of their data from the others. Synapse trans- fee. Request permissions from [email protected]. parently synchronizes these data subsets in real-time with EuroSys’15, April 21–24, 2015, Bordeaux, France. Copyright c 2015 ACM 978-1-4503-3238-5/15/04. $15.00. little to no programmer effort. To use Synapse, developers http://dx.doi.org/10.1145/2741948.2741975 generally need only specify declaratively what data to share Type Supported Vendors Example use cases Synapse scales well up to 60,000 updates/second for various workloads. To achieve these goals, it lets subscribers paral- Relational PostgreSQL, MySQL, Oracle Highly structured content Document MongoDB,TokuMX, RethinkDB General purpose lelize their processing of updates as much as the workload Columnar Cassandra Write-intensive workloads and their semantic needs permit. Search Elasticsearch Aggregations and analytics Graph Neo4j Social network modeling 2. Background Table 1: DB types and vendors supported by Synapse. MVC (Model View Controller) is a widely-used Web appli- with or incorporate from other services in a simple publish/- cation architecture supported by many frameworks includ- subscribe model. The data is then delivered to the DBs in ing Struts (Java), Django (Python), Rails (Ruby), Symfony real-time, at scale, and with delivery semantic guarantees. (PHP), Enterprise Java Beans (Java), and ASP.NET MVC Synapse makes this possible by leveraging the same ab- (.NET). For example, GitHub, Twitter, and YellowPages are stractions that Web programmers already use in widely-used built with Rails, DailyMotion and Yahoo! Answers are built Model-View-Controller (MVC) Web frameworks, such as with Symfony, and Pinterest and Instragram are built with Ruby-on-Rails, Python Django, or PHP Symfony. Using an Django. In the MVC pattern, applications define data mod- MVC paradigm, programmers logically separate an appli- els that describe the data, which are typically persisted to cation into models, which describe the data persisted and DBs. Because the data is persisted to a DB, the model is ex- manipulated, and controllers, which are units of work that pressed in terms of constructs that can be manipulated by the implement business logic and act on the models. Develop- DBs. Since MVC applications interact with DBs via ORMs ers specify what data to share among services within model (Object Relational Mappers), ORMs provide the model def- declarations through Synapse’s intuitive API. Models are ex- inition constructs [5]. pressed in terms of high-level objects that are defined and ORMs abstract many DB-related details and let program- automatically mapped to a DB via Object/Relational Map- mers code in terms of high-level objects, which generally pers (ORMs) [5]. Although different DBs may need different correspond one-to-one to individual rows, or documents in ORMs, most ORMs expose a common high-level object API the underlying DB. Although ORMs were initially devel- to developers that includes create, read, update, and delete oped for relational DBs, the concept has recently been ap- operations. Synapse leverages this common object API and plied to many other types of NoSQL DBs. Although dif- lets ORMs do the heavy lifting to provide a cross-DB trans- ferent ORMs may offer different APIs, at a minimum they lation layer among Web services. must provide a way to create, update, and delete the ob- We have built Synapse on Ruby-on-Rails and released it jects in the DB. For example, an application would typically as open source software on GitHub [43]. We demonstrate instantiate an object, set its attributes in memory, and then three key benefits. First, Synapse supports the needs of mod- invoke the ORM’s save function to persist it. Many ORMs ern Web applications by enabling them to use many combi- and MVC frameworks also support a notion of active mod- nations of heterogeneous DBs, both SQL and NoSQL. Ta- els [18], which allow developers to specify callbacks that are ble 1 shows the DBs we support, many of which are very invoked before or after any ORM-based update operation. popular. We show that adding support for new DBs incurs MVC applications define controllers to implement busi- limited effort. The translation between different DBs is of- ness logic and act on the data. Controllers define basic units ten automatic through Synapse and its use of ORMs. of work in which data is read, manipulated, and then written Second, Synapse provides programmer simplicity through back to DBs. Applications are otherwise stateless outside of a simple programming abstraction based on a publish/sub- controllers. Since Web applications are typically designed to scribe data sharing model that allows programmers to choose respond to and interact with users, controllers typically op- their own data update semantics to match the needs of their erate within the context of a user session, which means their MVC Web applications. We have demonstrated Synapse’s logic is applied on a per user basis. ease of use in a case study that integrates several large and widely-used Web components such as the e-commerce plat- 3.
Load more

Recommended publications
  • Cloudikoulaone
    PRÉSENTE CLOUDIKOULAONE Le succès est votre prochaine destination MIAMI SINGAPOUR PARIS AMSTERDAM FRANCFORT ___ CLOUDIKOULAONE est une solution de Cloud public, privé et hybride qui vous permet de déployer en 1 clic et en moins de 30 secondes des machines virtuelles à travers le monde sur des infrastructures SSD haute performance. www.ikoula.com [email protected] 01 84 01 02 50 NOM DE DOMAINE | HÉBERGEMENT WEB | SERVEUR VPS | SERVEUR DÉDIÉ | CLOUD PUBLIC | MESSAGERIE | STOCKAGE | CERTIFICATS SSL LINUX PRATIQUE est édité par Les Éditions Diamond 10, Place de la Cathédrale - 68000 Colmar - France Tél. : 03 67 10 00 20 | Fax : 03 67 10 00 21 édito E-mail : [email protected] Linux Pratique n°102 [email protected] Service commercial : [email protected] Sites : http://www.linux-pratique.com http://www.ed-diamond.com Directeur de publication : Arnaud Metzler Chef des rédactions : Denis Bodor Rédactrice en chef : Aline Hof Responsable service infographie : Kathrin Scali Responsable publicité : Tél. : 03 67 10 00 27 Service abonnement : Tél. : 03 67 10 00 20 Photographie et images : http://www.fotolia.com Impression : pva, Landau, Allemagne Distribution France : (uniquement pour les dépositaires de presse) MLP Réassort : Plate-forme de Saint-Barthélemy-d’Anjou Au moment où je rédige ces lignes, la température extérieure affiche une tren- Tél. : 02 41 27 53 12 taine de degrés et une furieuse envie de troquer ma place au bureau devant Plate-forme de Saint-Quentin-Fallavier mon PC contre un transat au bord de la mer (à remplacer évidemment par ce Tél. : 04 74 82 63 04 qui vous fait plaisir lorsque la canicule pointe le bout de son nez) commence à Service des ventes : Distri-médias : Tél.
    [Show full text]
  • Nethserver-201 Cahier-11
    Micronator NethServer-201 Cahier-11 NethServer & diaspora* Version: RC-001 / vendredi 18 septembre 2020 - 13:14 © 2020 RF-232 6447, avenue Jalobert, Montréal Qc H1M 1L1 Tous droits réservés RF-232 AVIS DE NON-RESPONSABILITÉ Ce document est uniquement destiné à informer. Les informations, ainsi que les contenus et fonctionnalités de ce do- cument sont fournis sans engagement et peuvent être modifiés à tout moment. RF-232 n'offre aucune garantie quant à l'actualité, la conformité, l'exhaustivité, la qualité et la durabilité des informations, contenus et fonctionnalités de ce do- cument. L'accès et l'utilisation de ce document se font sous la seule responsabilité du lecteur ou de l'utilisateur. RF-232 ne peut être tenu pour responsable de dommages de quelque nature que ce soit, y compris des dommages di- rects ou indirects, ainsi que des dommages consécutifs résultant de l'accès ou de l'utilisation de ce document ou de son contenu. Chaque internaute doit prendre toutes les mesures appropriées (mettre à jour régulièrement son logiciel antivirus, ne pas ouvrir des documents suspects de source douteuse ou non connue) de façon à protéger le contenu de son ordinateur de la contamination d'éventuels virus circulant sur la Toile. Toute reproduction interdite Vous reconnaissez et acceptez que tout le contenu de ce document, incluant mais sans s’y limiter, le texte et les images, sont protégés par le droit d’auteur, les marques de commerce, les marques de service, les brevets, les secrets industriels et les autres droits de propriété intellectuelle. Sauf autorisation expresse de RF-232, vous acceptez de ne pas vendre, dé- livrer une licence, louer, modifier, distribuer, copier, reproduire, transmettre, afficher publiquement, exécuter en public, publier, adapter, éditer ou créer d’oeuvres dérivées de ce document et de son contenu.
    [Show full text]
  • Reference Coupling: an Exploration of Inter-Project Technical Dependencies and Their Characteristics Within Large Software Ecosystems
    Reference Coupling: An Exploration of Inter-project Technical Dependencies and their Characteristics within Large Software Ecosystems Kelly Blincoea,∗, Francis Harrisonb, Navpreet Kaurb, Daniela Damianb aUniversity of Auckland, New Zealand bUniversity of Victoria, BC, Canada Abstract Context: Software projects often depend on other projects or are developed in tandem with other projects. Within such software ecosystems, knowledge of cross-project technical dependencies is important for 1) practitioners under- standing of the impact of their code change and coordination needs within the ecosystem and 2) researchers in exploring properties of software ecosystems based on these technical dependencies. However, identifying technical depen- dencies at the ecosystem level can be challenging. Objective: In this paper, we describe Reference Coupling, a new method that uses solely the information in developers online interactions to detect technical dependencies between projects. The method establishes dependencies through user-specified cross-references between projects. We then use the output of this method to explore the properties of large software ecosystems. Method: We validate our method on two datasets | one from open-source projects hosted on GitHub and one commercial dataset of IBM projects. We manually analyze the identified dependencies, categorize them, and compare them to dependencies specified by the development team. We examine the types of projects involved in the identified ecosystems, the structure of the iden- ∗Corresponding author Email addresses: [email protected] (Kelly Blincoe), [email protected] (Francis Harrison), [email protected] (Navpreet Kaur), [email protected] (Daniela Damian) Preprint submitted to Elsevier February 2, 2019 tified ecosystems, and how the ecosystems structure compares with the social behaviour of project contributors and owners.
    [Show full text]
  • A Deep Dive Into Docker Hub's Security Landscape
    A Deep Dive into Docker Hub’s Security Landscape A story of inheritance? Emilien Socchi Jonathan Luu Thesis submitted for the degree of Master in Network and System Administration 30 credits Department of Informatics Faculty of Mathematics and Natural Sciences UNIVERSITY OF OSLO Spring 2019 A Deep Dive into Docker Hub’s Security Landscape A story of inheritance? Emilien Socchi Jonathan Luu © 2019 Emilien Socchi, Jonathan Luu A Deep Dive into Docker Hub’s Security Landscape http://www.duo.uio.no/ Printed: Reprosentralen, University of Oslo Abstract Docker containers have become a popular virtualization technology for running multiple isolated application services on a single host using minimal resources. That popularity has led to the cre- ation of an online sharing platform known as Docker Hub, hosting images that Docker containers instantiate. In this thesis, a deep dive into Docker Hub’s security landscape is undertaken. First, a Python based software used to conduct experiments and collect metadata, parental and vul- nerability information about any type of image available on Docker Hub is developed. Secondly, our tool allows analyzing the most recent image found in each Certified, Verified and Official repository, as well the most recent image found in 500 random Community repositories among the most popular ones. Using our software named Docker imAge analyZER (DAZER), the fol- lowing discoveries were made: (1) the Certified and Verified repositories introduced by Docker Inc. in December 2018 do not improve the overall Docker Hub’s security
    [Show full text]
  • Pythia: Grammar-Based Fuzzing of REST Apis with Coverage-Guided Feedback and Learning-Based Mutations
    Pythia: Grammar-Based Fuzzing of REST APIs with Coverage-guided Feedback and Learning-based Mutations Vaggelis Atlidakis Roxana Geambasu Patrice Godefroid Marina Polishchuk Baishakhi Ray Columbia University Columbia University Microsoft Research Microsoft Research Columbia University Abstract—This paper introduces Pythia, the first fuzzer that input format, and may also specify what input parts are to augments grammar-based fuzzing with coverage-guided feedback be fuzzed and how [41], [47], [7], [9]. From such an input and a learning-based mutation strategy for stateful REST API grammar, a grammar-based fuzzer then generates many new fuzzing. Pythia uses a statistical model to learn common usage patterns of a target REST API from structurally valid seed inputs, each satisfying the constraints encoded by the grammar. inputs. It then generates learning-based mutations by injecting Such new inputs can reach deeper application states and find a small amount of noise deviating from common usage patterns bugs beyond syntactic lexers and semantic checkers. while still maintaining syntactic validity. Pythia’s mutation strat- Grammar-based fuzzing has recently been automated in the egy helps generate grammatically valid test cases and coverage- domain of REST APIs by RESTler [3]. Most production-scale guided feedback helps prioritize the test cases that are more likely to find bugs. We present experimental evaluation on three cloud services are programmatically accessed through REST production-scale, open-source cloud services showing that Pythia APIs that are documented using API specifications, such as outperforms prior approaches both in code coverage and new OpenAPI [52]. Given such a REST API specification, RESTler bugs found.
    [Show full text]
  • Structure and Feedback in Cloud Service API Fuzzing
    Structure and Feedback in Cloud Service API Fuzzing Evangelos Atlidakis Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy under the Executive Committee of the Graduate School of Arts and Sciences COLUMBIA UNIVERSITY 2021 © 2021 Evangelos Atlidakis All Rights Reserved ABSTRACT Stucture and Feedback in Cloud Service API Fuzzing Evangelos Atlidakis Over the last decade, we have witnessed an explosion in cloud services for hosting software applications (Software-as-a-Service), for building distributed services (Platform- as-a-Service), and for providing general computing infrastructure (Infrastructure-as-a- Service). Today, most cloud services are programmatically accessed through Applica- tion Programming Interfaces (APIs) that follow the REpresentational State Trans- fer (REST) software architectural style and cloud service developers use interface- description languages to describe and document their services. My thesis is that we can leverage the structured usage of cloud services through REST APIs and feedback obtained during interaction with such services in order to build systems that test cloud services in an automatic, efficient, and learning-based way through their APIs. In this dissertation, I introduce stateful REST API fuzzing and describe its imple- mentation in RESTler: the first stateful REST API fuzzing system. Stateful means that RESTler attempts to explore latent service states that are reachable only with sequences of multiple interdependent API requests. I then describe how stateful REST API fuzzing can be extended with active property checkers that test for violations of desirable REST API security properties. Finally, I introduce Pythia, a new fuzzing system that augments stateful REST API fuzzing with coverage-guided feedback and learning-based mutations.
    [Show full text]
  • Deterministic, Mutable, and Distributed Record-Replay for Operating Systems and Database Systems
    Deterministic, Mutable, and Distributed Record-Replay for Operating Systems and Database Systems Nicolas Viennot Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Graduate School of Arts and Sciences COLUMBIA UNIVERSITY 2017 c 2017 Nicolas Viennot All Rights Reserved ABSTRACT Deterministic, Mutable, and Distributed Record-Replay for Operating Systems and Database Systems Nicolas Viennot Application record and replay is the ability to record application execution and replay it at a later time. Record-replay has many use cases including diagnosing and debugging applications by capturing and re- producing hard to find bugs, providing transparent application fault tolerance by maintaining a live replica of a running program, and offline instrumentation that would be too costly to run in a production environ- ment. Different record-replay systems may offer different levels of replay faithfulness, the strongest level being deterministic replay which guarantees an identical reenactment of the original execution. Such a guarantee requires capturing all sources of nondeterminism during the recording phase. In the general case, such record-replay systems can dramatically hinder application performance, rendering them unpractical in certain application domains. Furthermore, various use cases are incompatible with strictly replaying the original execution. For example, in a primary-secondary database scenario, the secondary database would be unable to serve additional traffic while being replicated. No record-replay system fit all use cases. This dissertation shows how to make deterministic record-replay fast and efficient, how broadening replay semantics can enable powerful new use cases, and how choosing the right level of abstraction for record-replay can support distributed and heterogeneous database replication with little effort.
    [Show full text]
  • Serendipity and Strategy in Rapid Innovation T
    Serendipity and strategy in rapid innovation T. M. A. Fink∗†, M. Reevesz, R. Palmaz and R. S. Farry yLondon Institute for Mathematical Sciences, Mayfair, London W1K 2XF, UK ∗Centre National de la Recherche Scientifique, Paris, France zBCG Henderson Institute, The Boston Consulting Group, New York, USA Innovation is to organizations what evolution is to organisms: it while, Bob chooses pieces such as axels, wheels, and small base is how organisations adapt to changes in the environment and plates that he noticed are common in more complex models, improve [1]. Governments, institutions and firms that innovate even though he is not able to use them straightaway to produce are more likely to prosper and stand the test of time; those new toys. We call this a far-sighted strategy. that fail to do so fall behind their competitors and succumb Who wins. At the end of the day, who will have innovated to market and environmental change [2, 3]. Yet despite steady the most? That is, who will have built the most new toys? We advances in our understanding of evolution, what drives inno- find that, in the beginning, Alice will lead the way, surging vation remains elusive [1, 4]. On the one hand, organizations ahead with her impatient strategy. But as the game progresses, invest heavily in systematic strategies to drive innovation [5{ fate will appear to shift. Bob's early moves will begin to look 8]. On the other, historical analysis and individual experience serendipitous when he is able to assemble a complex fire truck suggest that serendipity plays a significant role in the discovery from his choice of initially useless axels and wheels.
    [Show full text]