A Generic and Extensible Core and Prototype of Consistent, Distributed, and Resilient LIS
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Technical Project and Product Manager Solution Architect and Senior Full Stack Developer
Technical project and product manager Solution architect and senior full stack developer Profile Technical project and product manager, working with R&D software innovation processes, with a background as solution architect and senior full stack developer. Experienced manager of outsourced teams in many different countries, working with team efficiency based on pragmatic Scrum approach. Experience from scaled agile projects with hundreds of people involved. I am used to working with confidential information and have been security cleared several times. Knowledge I have been programming more than 20 different programming, script and database languages the last 36 years. My recent programming skills include NodeJS, JavaScript, Python, C#, Java, React for progressive web apps (PWA) and some courses in React-Native Expo. I have worked mostly with REST API architecture and a little GraphQL. I have specialist knowledge from leading CMS and DMS systems like Sitecore, DOCUMENTUM and Public 360, were I as product manager had to know Danish law on personal data and record management (ESDH). I worked many years on products for the educational sector and non-profit projects like Oligo Academy, that uses virtual worlds in primary schools for teaching environmental issues and other study subjects. Nikolaj Lisberg Hansen Born 1973 and started programming early in 1984. Got first programming job in 1995 and started working as freelance solution architect and technical project manager in 2006. I practice Tai Chi, love music and like to travel the world. Languages Very good Danish, English and German. Availability 10-25 hours per week normally remote or on-site near Copenhagen. Flexible salary between 96€ / hour (600 DKK) for remote work and 144€ / hour (900 DKK) for on-site work on product innovation or as technical project manager, solution architect or senior developer. -
Towards a Library for Deterministic Failure Testing of Distributed Systems
UC Irvine UC Irvine Electronic Theses and Dissertations Title Towards a Library for Deterministic Failure Testing of Distributed Systems Permalink https://escholarship.org/uc/item/6h90g8rz Author Balalaie, Armin Publication Date 2020 License https://creativecommons.org/licenses/by/4.0/ 4.0 Peer reviewed|Thesis/dissertation eScholarship.org Powered by the California Digital Library University of California UNIVERSITY OF CALIFORNIA, IRVINE Towards a Library for Deterministic Failure Testing of Distributed Systems THESIS submitted in partial satisfaction of the requirements for the degree of MASTER OF SCIENCE in Software Engineering by Armin Balalaie Thesis Committee: Associate Professor James A. Jones, Chair Professor Cristina V. Lopes Professor Michael J. Carey 2020 c 2020 Armin Balalaie DEDICATION To Sara, my love, without whom and her patience, this wasn't possible .. and My mom without her help, dedication and sacrifices, I wouldn't be where I am today .. ii TABLE OF CONTENTS Page LIST OF FIGURES v LIST OF LISTINGS vi LIST OF TABLES vii ACKNOWLEDGMENTS viii ABSTRACT OF THE THESIS ix 1 Introduction 1 2 Existing Work 5 2.1 Deployment-Focused Testing Frameworks . .5 2.2 Random Failure Injection . .6 2.3 Systematic Failure Injection . .8 2.4 Failure-specific Frameworks . .9 2.5 Model-based Approaches . .9 3 Example Failify Test Case for HDFS 11 4 Design Goals 14 4.1 Minimum Learning Curve . 14 4.2 Easy and Deterministic Failure Injection and Environment Manipulation . 15 4.3 Cross-Platform and Multi Language . 16 4.4 Seamless Integration . 17 4.5 Multiple Runtime Engines . 17 4.6 Research Infrastructure . 17 5 Architecture and Implementation 19 5.1 Java-based DSL . -
Oracle Vs. Nosql Vs. Newsql Comparing Database Technology
Oracle vs. NoSQL vs. NewSQL Comparing Database Technology John Ryan Senior Solution Architect, Snowflake Computing Table of Contents The World has Changed . 1 What’s Changed? . 2 What’s the Problem? . .. 3 Performance vs. Availability and Durability . 3 Consistecy vs. Availability . 4 Flexibility vs . Scalability . 5 ACID vs. Eventual Consistency . 6 The OLTP Database Reimagined . 7 Achieving the Impossible! . .. 8 NewSQL Database Technology . 9 VoltDB . 10 MemSQL . 11 Which Applications Need NewSQL Technology? . 12 Conclusion . 13 About the Author . 13 ii The World has Changed The world has changed massively in the past 20 years. Back in the year 2000, a few million users connected to the web using a 56k modem attached to a PC, and Amazon only sold books. Now billions of people are using to their smartphone or tablet 24x7 to buy just about everything, and they’re interacting with Facebook, Twitter and Instagram. The pace has been unstoppable . Expectations have also changed. If a web page doesn’t refresh within seconds we’re quickly frustrated, and go elsewhere. If a web site is down, we fear it’s the end of civilisation as we know it. If a major site is down, it makes global headlines. Instant gratification takes too long! — Ladawn Clare-Panton Aside: If you’re not a seasoned Database Architect, you may want to start with my previous articles on Scalability and Database Architecture. Oracle vs. NoSQL vs. NewSQL eBook 1 What’s Changed? The above leads to a few observations: • Scalability — With potentially explosive traffic growth, IT systems need to quickly grow to meet exponential numbers of transactions • High Availability — IT systems must run 24x7, and be resilient to failure. -
ACID Compliant Distributed Key-Value Store
ACID Compliant Distributed Key-Value Store #1 #2 #3 Lakshmi Narasimhan Seshan , Rajesh Jalisatgi , Vijaeendra Simha G A # 1l [email protected] # 2r [email protected] #3v [email protected] Abstract thought that would be easier to implement. All Built a fault-tolerant and strongly-consistent key/value storage components run http and grpc endpoints. Http endpoints service using the existing RAFT implementation. Add ACID are for debugging/configuration. Grpc Endpoints are transaction capability to the service using a 2PC variant. Build used for communication between components. on the raftexample[1] (available as part of etcd) to add atomic Replica Manager (RM): RM is the service discovery transactions. This supports sharding of keys and concurrent transactions on sharded KV Store. By Implementing a part of the system. RM is initiated with each of the transactional distributed KV store, we gain working knowledge Replica Servers available in the system. Users have to of different protocols and complexities to make it work together. instantiate the RM with the number of shards that the user wants the key to be split into. Currently we cannot Keywords— ACID, Key-Value, 2PC, sharding, 2PL Transaction dynamically change while RM is running. New Replica Servers cannot be added or removed from once RM is I. INTRODUCTION up and running. However Replica Servers can go down Distributed KV stores have become a norm with the and come back and RM can detect this. Each Shard advent of microservices architecture and the NoSql Leader updates the information to RM, while the DTM revolution. Initially KV Store discarded ACID properties leader updates its information to the RM. -
Nosql Databases: Yearning for Disambiguation
NOSQL DATABASES: YEARNING FOR DISAMBIGUATION Chaimae Asaad Alqualsadi, Rabat IT Center, ENSIAS, University Mohammed V in Rabat and TicLab, International University of Rabat, Morocco [email protected] Karim Baïna Alqualsadi, Rabat IT Center, ENSIAS, University Mohammed V in Rabat, Morocco [email protected] Mounir Ghogho TicLab, International University of Rabat Morocco [email protected] March 17, 2020 ABSTRACT The demanding requirements of the new Big Data intensive era raised the need for flexible storage systems capable of handling huge volumes of unstructured data and of tackling the challenges that arXiv:2003.04074v2 [cs.DB] 16 Mar 2020 traditional databases were facing. NoSQL Databases, in their heterogeneity, are a powerful and diverse set of databases tailored to specific industrial and business needs. However, the lack of the- oretical background creates a lack of consensus even among experts about many NoSQL concepts, leading to ambiguity and confusion. In this paper, we present a survey of NoSQL databases and their classification by data model type. We also conduct a benchmark in order to compare different NoSQL databases and distinguish their characteristics. Additionally, we present the major areas of ambiguity and confusion around NoSQL databases and their related concepts, and attempt to disambiguate them. Keywords NoSQL Databases · NoSQL data models · NoSQL characteristics · NoSQL Classification A PREPRINT -MARCH 17, 2020 1 Introduction The proliferation of data sources ranging from social media and Internet of Things (IoT) to industrially generated data (e.g. transactions) has led to a growing demand for data intensive cloud based applications and has created new challenges for big-data-era databases. -
Go in Tidb Yao Wei | Pingcap About Me
Go in TiDB Yao Wei | PingCAP About me ● Yao Wei (姚维) ● TiDB Kernel Expert, General Manager of South Region, China ● 360 Infra team / Alibaba-UC / PingCAP ● Atlas/MySQL-Sniffer ● Infrastructure software engineer Why a new database? Brief History RDBMS NoSQL NewSQL ● Standalone RDBMS 1970s 2010 2015 Present ● NoSQL MySQL Redis Google Spanner PostgreSQL HBase Google F1 Oracle Cassandra TiDB ● Middleware & Proxy DB2 MongoDB ... ... ● NewSQL Architecture Stateless SQL Layer Metadata / Timestamp request TiDB ... TiDB ... TiDB Placement Driver (PD) Raft Raft Raft TiKV ... TiKV TiKV TiKV Control flow: Balance / Failover Distributed Storage Layer TiKV - Overview • Region: a set of continuous key-value pairs • Data is organized/stored/replicated by Regions • Highly layered TiKV Key Space RPC (gRPC) Node A Transaction 256MB MVCC [ start_key, Raft end_key) RocksDB (-∞, +∞) Raft Raft Sorted Map Node B Node C Raft PD - Overview ● Meta data management ● Load balance management Route Info TiKV Client PD Node/Region Management Info Command TiKV TiKV TiKV TiKV … ... TiKV Cluster TiKV - Multi-Raft Multiple raft groups in the cluster, one group for each region. Client RPC RPC RPC RPC Store 1 Store 2 Store 3 Store 4 Region 1 Region 1 Region 2 Region 1 Region 3 Region 2 Region 5 Region 2 Raft Region 5 Region 4 Region 3 Region 5 Group Region 4 Region 3 Region 4 TiKV node 1 TiKV node 2 TiKV node 3 TiKV node 4 TiKV - Horizontal Scale Node B Region 1^ Region 1 Region 2 Region 3 Region 1* Node D Region 2 Region 2 Region 3 Region 3 Node C Node A Add Replica Three -
Cloud Computing Synopsis and Recommendations
Special Publication 800-146 Cloud Computing Synopsis and Recommendations Recommendations of the National Institute of Standards and Technology Lee Badger Tim Grance Robert Patt-Corner Jeff Voas NIST Special Publication 800-146 Cloud Computing Synopsis and Recommendations Recommendations of the National Institute of Standards and Technology Lee Badger Tim Grance Robert Patt-Corner Jeff Voas C O M P U T E R S E C U R I T Y Computer Security Division Information Technology Laboratory National Institute of Standards and Technology Gaithersburg, MD 20899-8930 May 2012 U.S. Department of Commerce John Bryson, Secretary National Institute of Standards and Technology Patrick D. Gallagher, Under Secretary of Commerce for Standards and Technology and Director CLOUD COMPUTING SYNOPSIS AND RECOMMENDATIONS Reports on Computer Systems Technology The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analysis to advance the development and productive use of information technology. ITL’s responsibilities include the development of management, administrative, technical, and physical standards and guidelines for the cost-effective security and privacy of other than national security-related information in Federal information systems. This Special Publication 800-series reports on ITL’s research, guidance, and outreach efforts in computer security and its collaborative activities with industry, government, and academic organizations. National Institute of Standards and Technology Special Publication 800-146 Natl. Inst. Stand. Technol. Spec. Publ. -
Grpc - a Solution for Rpcs by Google Distributed Systems Seminar at Charles University in Prague, Nov 2016 Jan Tattermusch - Grpc Software Engineer
gRPC - A solution for RPCs by Google Distributed Systems Seminar at Charles University in Prague, Nov 2016 Jan Tattermusch - gRPC Software Engineer About me ● Software Engineer at Google (since 2013) ● Working on gRPC since Q4 2014 ● Graduated from Charles University (2010) Contacts ● jtattermusch on GitHub ● Feedback to [email protected] @grpcio Motivation: gRPC Google has an internal RPC system, called Stubby ● All production applications use RPCs ● Over 1010 RPCs per second in total ● 4 generations over 13 years (since 2003) ● APIs for C++, Java, Python, Go What's missing ● Not suitable for external use (tight coupling with internal tools & infrastructure) ● Limited language & platform support ● Proprietary protocol and security ● No mobile support @grpcio What's gRPC ● HTTP/2 based RPC framework ● Secure, Performant, Multiplatform, Open Multiplatform ● Idiomatic APIs in popular languages (C++, Go, Java, C#, Node.js, Ruby, PHP, Python) ● Supports mobile devices (Android Java, iOS Obj-C) ● Linux, Windows, Mac OS X ● (web browser support in development) OpenSource ● developed fully in open on GitHub: https://github.com/grpc/ @grpcio Use Cases Build distributed services (microservices) Service 1 Service 3 ● In public/private cloud ● Google's own services Service 2 Service 4 Client-server communication ● Mobile ● Web ● Also: Desktop, embedded devices, IoT Access APIs (Google, OSS) @grpcio Key Features ● Streaming, Bidirectional streaming ● Built-in security and authentication ○ SSL/TLS, OAuth, JWT access ● Layering on top of HTTP/2 -
Object Migration in a Distributed, Heterogeneous SQL Database Network
Linköping University | Department of Computer and Information Science Master’s thesis, 30 ECTS | Computer Engineering (Datateknik) 2018 | LIU-IDA/LITH-EX-A--18/008--SE Object Migration in a Distributed, Heterogeneous SQL Database Network Datamigrering i ett heterogent nätverk av SQL-databaser Joakim Ericsson Supervisor : Tomas Szabo Examiner : Olaf Hartig Linköpings universitet SE–581 83 Linköping +46 13 28 10 00 , www.liu.se Upphovsrätt Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare – under 25 år från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/. Copyright The publishers will keep this document online on the Internet – or its possible replacement – for a period of 25 years starting from the date of publication barring exceptional circumstances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. -
Database Software Market: Billy Fitzsimmons +1 312 364 5112
Equity Research Technology, Media, & Communications | Enterprise and Cloud Infrastructure March 22, 2019 Industry Report Jason Ader +1 617 235 7519 [email protected] Database Software Market: Billy Fitzsimmons +1 312 364 5112 The Long-Awaited Shake-up [email protected] Naji +1 212 245 6508 [email protected] Please refer to important disclosures on pages 70 and 71. Analyst certification is on page 70. William Blair or an affiliate does and seeks to do business with companies covered in its research reports. As a result, investors should be aware that the firm may have a conflict of interest that could affect the objectivity of this report. This report is not intended to provide personal investment advice. The opinions and recommendations here- in do not take into account individual client circumstances, objectives, or needs and are not intended as recommen- dations of particular securities, financial instruments, or strategies to particular clients. The recipient of this report must make its own independent decisions regarding any securities or financial instruments mentioned herein. William Blair Contents Key Findings ......................................................................................................................3 Introduction .......................................................................................................................5 Database Market History ...................................................................................................7 Market Definitions -
Implementing and Testing Cockroachdb on Kubernetes
Implementing and testing CockroachDB on Kubernetes Albert Kasperi Iho Supervisors: Ignacio Coterillo Coz, Ricardo Brito Da Rocha Abstract This report will describe implementing CockroachDB on Kubernetes, a sum- mer student project done in 2019. The report addresses the implementation, monitoring, testing and results of the project. In the implementation and mon- itoring parts, a closer look to the full pipeline will be provided, with the intro- duction of all the frameworks used and the biggest problems that were faced during the project. After that, the report describes the testing process of Cock- roachDB, what tools were used in the testing and why. The results of testing will be presented after testing with the conclusion of results. Keywords CockroachDB; Kubernetes; benchmark. Contents 1 Project specification . 2 2 Implementation . 2 2.1 Kubernetes . 2 2.2 CockroachDB . 3 2.3 Setting up Kubernetes in CERN environment . 3 3 Monitoring . 3 3.1 Prometheus . 4 3.2 Prometheus Operator . 4 3.3 InfluxDB . 4 3.4 Grafana . 4 3.5 Scraping and forwarding metrics from the Kubernetes cluster . 4 4 Testing CockroachDB on Kubernetes . 5 4.1 SQLAlchemy . 5 4.2 Pgbench . 5 4.3 CockroachDB workload . 6 4.4 Benchmarking CockroachDB . 6 5 Test results . 6 5.1 SQLAlchemy . 6 5.2 Pgbench . 6 5.3 CockroachDB workload . 7 5.4 Conclusion . 7 6 Acknowledgements . 7 1 Project specification The core goal of the project was taking a look into implementing a database framework on top of Kuber- netes, the challenges in implementation and automation possibilities. Final project pipeline included a CockroachDB cluster running in Kubernetes with Prometheus monitoring both of them. -
Comparative Studies of 10 Programming Languages Within 10 Diverse Criteria
Department of Computer Science and Software Engineering Comparative Studies of 10 Programming Languages within 10 Diverse Criteria Jiang Li Sleiman Rabah Concordia University Concordia University Montreal, Quebec, Concordia Montreal, Quebec, Concordia [email protected] [email protected] Mingzhi Liu Yuanwei Lai Concordia University Concordia University Montreal, Quebec, Concordia Montreal, Quebec, Concordia [email protected] [email protected] COMP 6411 - A Comparative studies of programming languages 1/139 Sleiman Rabah, Jiang Li, Mingzhi Liu, Yuanwei Lai This page was intentionally left blank COMP 6411 - A Comparative studies of programming languages 2/139 Sleiman Rabah, Jiang Li, Mingzhi Liu, Yuanwei Lai Abstract There are many programming languages in the world today.Each language has their advantage and disavantage. In this paper, we will discuss ten programming languages: C++, C#, Java, Groovy, JavaScript, PHP, Schalar, Scheme, Haskell and AspectJ. We summarize and compare these ten languages on ten different criterion. For example, Default more secure programming practices, Web applications development, OO-based abstraction and etc. At the end, we will give our conclusion that which languages are suitable and which are not for using in some cases. We will also provide evidence and our analysis on why some language are better than other or have advantages over the other on some criterion. 1 Introduction Since there are hundreds of programming languages existing nowadays, it is impossible and inefficient