DOCSLIB.ORG

Das Große Einmaleins Der Nosql Datenbanken

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Das große Einmaleins der NoSQL Datenbanken Felix Gessert [email protected] @baqendcom Slides: slideshare.net/felixgessert Article: medium.com/baqend-blog About me NoSQL research for PhD dissertation CEO of startup for high-performance serverless development based on caching and NoSQL Why NoSQL? Architecture Typical Data Architecture: Analytics Reporting Data Mining Analytics Data Warehouse The era Data of one-size-fits-all database systems is over Specialized dataNoSQLsystems Operative Database Applications Data Data Management The Database Explosion Sweetspots RDBMS Parallel DWH NewSQL General-purpose Aggregations/OLAP for High throughput ACID transactions massive data amounts relational OLTP Wide-Column Store Document Store Key-Value Store Long scans over Deeply nested Large-scale structured data data models session storage Graph Database In-Memory KV-Store Wide-Column Store Graph algorithms Counting & statistics Massive user- & queries generated content The Database Explosion Cloud-Database Sweetspots Realtime BaaS Managed RDBMS Managed Cache Communication and General-purpose Caching and collaboration ACID transactions transient storage Azure Tables Wide-Column Store Wide-Column Store Backend-as-a-Service Very large tables Massive user- Small Websites generated content and Apps Google Cloud Amazon Elastic Storage MapReduce Managed NoSQL Object Store Hadoop-as-a-Service Full-Text Search Massive File Big Data Analytics Storage How to choose a database system? Many Potential Candidates Question in thisApplicationtutorialLayer: Nested Billing Data Session data Files Application Data How to approach the decision problem? Friend Google Cloud Storage network requirements database Recommen- Cached data Search Index & metrics dation Engine Amazon Elastic MapReduce NoSQL Databases „NoSQL“ term coined in 2009 Interpretation: „Not Only SQL“ Typical properties: ◦ Non-relational ◦ Open-Source ◦ Schema-less (schema-free) ◦ Optimized for distribution (clusters) ◦ Tunable consistency NoSQL-Databases.org: Current list has over 150 NoSQL systems NoSQL Databases Two main motivations: Scalability Impedance Mismatch ID Line Item 1: … Customer Line Item2: … Payment: Credit Card, … User-generated data, Request load Line Items ? Orders Payment Customers Scale-up vs Scale-out Scale-Up (vertical Scale-Out (horizontal scaling): scaling): More RAM More CPU More HDD Commodity Hardware Shared-Nothing Architecture Schemafree Data Modeling RDBMS: NoSQL DB: Item[Price] - Item[Discount] SELECT Name, Age FROM Customers Implicit schema Customers Explicit schema Open Source & Commodity Hardware Commercial DBMS Open-Source DBMS Specialized DB hardware Commodity hardware (Oracle Exadata, etc.) Highly available network Commodity network (Infiniband, Fabric Path, etc.) (Ethernet, etc.) Highly Available Storage (SAN, Commodity drives (standard RAID, etc.) HDDs, JBOD) NoSQL System Classification Two common criteria: Data Consistency/Availability Model Trade-Off Key-Value AP: Available & Partition Tolerant Wide-Column CP: Consistent & Partition Tolerant Document CA: Not Partition Graph Tolerant Key-Value Stores Data model: (key) -> value Interface: CRUD (Create, Read, Update, Delete) users:2:friends {23, 76, 233, 11} Value: users:2:inbox [234, 3466, 86,55] An opaque blob users:2:settings Theme → "dark", cookies → "false" Key Examples: Amazon Dynamo (AP), Riak (AP), Redis (CP) Wide-Column Stores Data model: (rowkey, column, timestamp) -> value Interface: CRUD, Scan Versions (timestamped) Row Key Column content : "<html>…" com.cnn.www contentcontent: :"< "<htmlhtml>…">…" title : "CNN" crawled: … Examples: Cassandra (AP), Google BigTable (CP), HBase (CP) Document Stores Data model: (collection, key) -> document Interface: CRUD, Querys, Map-Reduce JSON Document ID/Key order-12338 { order-id: 23, customer: { name : "Felix Gessert", age : 25 } line-items : [ {product-name : "x", …} , …] } Examples: CouchDB (AP), Amazon SimpleDB (AP), MongoDB (CP) Graph Databases Data model: G = (V, E): Graph-Property Modell Interface: Traversal algorithms, querys, transactions Nodes WORKS_FOR Properties company: since: 1999 Apple salary: 140K name: value: John Doe 300Mrd Edges Examples: Neo4j (CA), InfiniteGraph (CA), OrientDB (CA) Soft NoSQL Systems Not Covered Here Search Platforms (Full Text Search): ◦ No persistence and consistency guarantees for OLTP ◦ Examples: ElasticSearch (AP), Solr (AP) Object-Oriented Databases: ◦ Strong coupling of programming language and DB ◦ Examples: Versant (CA), db4o (CA), Objectivity (CA) XML-Databases, RDF-Stores: ◦ Not scalable, data models not widely used in industry ◦ Examples: MarkLogic (CA), AllegroGraph (CA) CAP-Theorem Only 2 out of 3 properties are Consistency achievable at a time: ◦ Consistency: all clients have the same view on the data Partition Availability ◦ Availability: every request to a non- Tolerance failed node most result in correct response ◦ Partition tolerance: the system has to continue working, even under arbitrary network partitions Impossible Eric Brewer, ACM-PODC Keynote, Juli 2000 Gilbert, Lynch: Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services, SigAct News 2002 CAP-Theorem: simplified proof Problem: when a network partition occurs, either consistency or availability have to be given up Block response until Response before ACK arrives successful replication Consistency Availability Value = V1 Replication Value = V0 N1 N2 Network partition NoSQL Triangle Relational Data models Key-Value Every client can always Wide-Column Document-Oriented read and write A CA AP Oracle, MySQL, … Dynamo, Redis, Riak, Voldemort Cassandra SimpleDB, C CP P All clients share the Postgres, MySQL Cluster, Oracle RAC All nodes continue same view on the data BigTable, HBase, Accumulo, Azure Tables working under network MongoDB, RethinkDB partitions Nathan Hurst: Visual Guide to NoSQL Systems http://blog.nahurst.com/visual-guide-to-nosql-systems PACELC – an alternative CAP formulation Idea: Classify systems according to their behavior during network partitions yes no Partiti on No consequence of the CAP theorem Avail- Con- Laten- Con- ability sistency cy sistency AL - Dynamo-Style AC - MongoDB CC – Always Consistent Cassandra, Riak, etc. HBase, BigTable and ACID systems Abadi, Daniel. "Consistency tradeoffs in modern distributed database system design: CAP is only part of the story." Serializability Not Highly Available Either Global serializability and availability are incompatible: Write A=1 Write B=1 Read B Read A 푤1 푎 = 1 푟1(푏 = ⊥) 푤2 푏 = 1 푟2(푎 = ⊥) Some weaker isolation levels allow high availability: ◦ RAMP Transactions (P. Bailis, A. Fekete, A. Ghodsi, J. M. Hellerstein, und I. Stoica, „Scalable Atomic Visibility with RAMP Transactions“, SIGMOD 2014) S. Davidson, H. Garcia-Molina, and D. Skeen. Consistency in partitioned networks. ACM CSUR, 17(3):341–370, 1985. Where CAP fits in Negative Results in Distributed Computing Asynchronous Network, Asynchronous Network, Unreliable Channel Reliable Channel Atomic Storage Atomic Storage Impossible: Possible: CAP Theorem Attiya, Bar-Noy, Dolev (ABD) Algorithm Consensus Consensus Impossible: Impossible: 2 Generals Problem Fisher Lynch Patterson (FLP) Theorem Lynch, Nancy A. Distributed algorithms. Morgan Kaufmann, 1996. ACID vs BASE ACID BASE „Gold standard“ Model of many Basically for RDBMSs Atomicity NoSQL systems Available Consistency Soft State Eventually Isolation Consistent Durability http://queue.acm.org/detail.cfm?id=1394128 Data Models and CAP provide high-level classification. But what about fine-grained requirements, e.g. query capabilites? Outline NoSQL Foundations and • Techniques for Functional Motivation and Non-functional Requirements • Sharding The NoSQL Toolbox: • Replication Common Techniques • Storage Management • Query Processing NoSQL Systems Decision Guidance: NoSQL Decision Tree Functional Techniques Non-Functional enable Sharding enable Data Scalability Scan Queries Range-Sharding Hash-Sharding Entity-Group Sharding Write Scalability ACID Transactions Consistent Hashing Shared-Disk Functional Central Read Scalability Replication Conditional or Atomic Writes Elasticity Require- Committechniques/Consensus Protocol Operational Synchronous ments from AsynchronousNoSQL ConsistencyRequire- Joins Primary Copy Update Anywhere the databases Write Latencyments Storage Management applicationSorting employ Logging Read Latency Update-in-Place Caching Filter Queries In-Memory Storage Write Throughput Append-Only Storage Query Processing Read Availability Full-text Search Global Secondary Indexing Local Secondary Indexing Write Availability Query Planning Analytics Framework Aggregation and Analytics Materialized Views Durability http://www.baqend.com /files/nosql-survey.pdf Functional Techniques Non-Functional Sharding Data Scalability Scan Queries Range-Sharding Hash-Sharding Entity-Group Sharding Write Scalability ACID Transactions Consistent Hashing Shared-Disk Read Scalability Conditional or Atomic Writes Elasticity Joins Sorting Sharding (aka Partitioning, Fragmentation) Horizontal distribution of data over nodes Shard 1 [G-O] FranzPeter Shard 2 Shard 3 Partitioning strategies: Hash-based vs. Range-based Difficulty: Multi-Shard-Operations (join, aggregation) Sharding Hash-based Sharding Implemented in ◦ Hash of data values (e.g. key) determinesMongoDB, Riakpartition, Redis(, shard) ◦ Pro: Even distribution Cassandra, Azure Table, ◦ Contra: No data locality Dynamo Implemented in Range-based Sharding ◦
Recommended publications
  • Velocity London 2018

    Velocity London 2018

    Building Resilient Serverless Systems @johnchapin | symphonia.io John Chapin • Currently Partner, Symphonia • Former VP Engineering, Technical Lead • Data Engineering and Data Science teams • 20+ yrs experience in govt, healthcare, travel, and ad-tech • Intent Media, RoomKey, Meddius, SAIC, Booz Allen Agenda • What is Serverless? • Resiliency • Demo • Discussion and Questions What is Serverless? Serverless = FaaS + BaaS! • FaaS = Functions as a Service • AWS Lambda, Auth0 Webtask, Azure Functions, Google Cloud Functions, etc... • BaaS = Backend as a Service • Auth0, Amazon DynamoDB, Google Firebase, Parse, Amazon S3, etc... go.symphonia.io/what-is-serverless Serverless benefits • Cloud benefits ++ • Reduced cost • Scaling flexibility • Shorter lead time go.symphonia.io/what-is-serverless Serverless attributes • No managing of hosts or processes • Self auto-scaling and provisioning • Costs based on precise usage (down to zero!) • Performance specified in terms other than host size/count • Implicit high availability, but not disaster recovery go.symphonia.io/what-is-serverless Resiliency “Failures are a given and everything will eventually fail over time ...” –Werner Vogels (https://www.allthingsdistributed.com/2016/03/10-lessons-from-10-years-of-aws.html) K.C. Green, Gunshow #648 Werner on Embracing Failure • Systems will fail • At scale, systems will fail a lot • Embrace failure as a natural occurrence • Limit the blast radius of failures • Keep operating • Recover quickly (automate!) Failures in Serverless land • Serverless is all about using vendor-managed services. • Two classes of failures: • Application failures (your problem, your resolution) • All other failures (your problem, but not your resolution) • What happens when those vendor-managed services fail? • Or when the services used by those services fail? Mitigation through architecture • No control over resolving acute vendor failures.
  • How Are the Leaders of the Most Digitally Savvy Companies

    How Are the Leaders of the Most Digitally Savvy Companies

    Executive Traits for Recognizing the Bountiful Opportunities Ahead CONCLUSION Author By Krishnan Ramanujam President, Business and Technology Services, Tata Consultancy Services Industries are being reordered today by the confluence of four technologies: cloud computing (which makes supercomputing power affordable even for startup firms), artificial intelligence (which ups the IQ of products, people and processes), big data and analytics (which turn operational chaos into coherency), and the internet of things (which lets us track products, people, customers, and premises around the clock/around the world). But why do so many companies ignore the transformation that is happening now around them? In their new book, MIT professors Andrew McAfee and Erik Brynjolfsson argue (as Kuhn did about scientific revolutions) that it is difficult to change long-accepted beliefs. “Existing processes, customers and suppliers, pools of expertise, and more general mindsets can all blind incumbents to things that should be obvious, such as the possibilities of new technologies that depart greatly from the status quo,” they write in Machine, Platform, Crowd: Harnessing Our Digital Future.70 It’s why “so many of the smartest and most experienced people and companies … [are] the least able to see” a transformation that they won’t escape. 70 Andrew McAfee and Erik Brynjolfsson, Machine, Platform, Crowd: Harnessing Our Digital Future (W.W. Norton & Company), published June 2017. http://books. wwnorton.com/books/Machine-Platform-Crowd/ Accessed July 28, 2017. 91 Is such blindness avoidable? I think so. To most successful companies of the last do so, senior executives need to sharpen 10 years—Apple, Amazon, and Netflix— or develop five traits that may have rapidly recognize the potential of AI and sat dormant in them, but which I think automation, cloud computing, IoT and reside in all of us.
  • AWS with Corey Quinn

    AWS with Corey Quinn

    SED 745 Transcript EPISODE 745 [INTRODUCTION] [00:00:00] JM: Amazon Web Services changed how software engineers work. Before AWS, it was common for startups to purchase their own physical servers. AWS made server resources as accessible as an API request, and AWS has gone on to create higher-level abstractions for building applications. For the first few years of AWS, the abstractions were familiar. S3 provided distributed reliable objects storage. Elastic MapReduce provided a managed cloud Hadoop system. Kinesis provided a scalable queuing system. Amazon was providing developers with managed alternatives to complicated open source software. More recently, AWS has started to release products that are completely novel. They’re unlike anything else. A perfect example is AWS Lambda, the first function as a service platform. Other newer AWS products include Ground Station, which is a service for processing satellite data; and AWS DeepRacer, a miniature race car for developers to build and test machine learning algorithms on. As AWS has grown into new categories, the blog announcements for new services and features have started coming so frequently that is hard to keep track of it all. Corey Quinn is the author of Last Week in AWS, a popular newsletter about what is changing across Amazon Web Services. Corey joins the show today to give his perspective on the growing shifting behemoth that is Amazon Web Services as well as the other major cloud providers that have risen to prominence. Corey is the host of the Screaming in the Cloud podcast, which you should check out if you like this episode.
  • K12 Education Leaders! on Behalf of AWS, We Are Excited for You to Join Us for AWS Re:Invent 2020!

    K12 Education Leaders! on Behalf of AWS, We Are Excited for You to Join Us for AWS Re:Invent 2020!

    © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Welcome K12 Education Leaders! On behalf of AWS, we are excited for you to join us for AWS re:Invent 2020! This year’s virtual conference is going to be the industry event of the year. It is free of charge and will offer 5 Keynotes, 18 Leadership Sessions, and unlimited access to hundreds of sessions. We are happy to have you along for the journey! To help you prepare and be ready for this 3-week event, we’ve created this guide of suggested sessions and keynotes for K12 education. This is only a starting point. There is much more content on the AWS re:Invent website. Step 1: Register for re:Invent Once you register you will be able to search all 500+ sessions. Step 2: Register for Edu Rewind AWS education has also created weekly re:Invent Rewind sessions for education organizations to discuss announcements and takeaways throughout the event. Step 3: Follow @AWS_EDU on Twitter to join the conversation. We look forward to seeing you there! Learn 500+ sessions covering core services and emerging technologies will be available at your fingertips over 3 weeks. With hours of content to comb through, our team recommends the following sessions for K12 education professionals. K12 Session Recommendations LEADERSHIP TALKS Reimagine business applications from the ground up BIZ291-L DEC 1, 2020 | 2:15 PM - 3:15 PM EST Harness the power of data with AWS Analytics ANT291-L DEC 9, 2020 | 12:15 PM - 1:15 PM EST AWS security: Where we’ve been, where we’re going SEC291-L DEC 8, 2020
  • Bigchaindb Server Documentation Release 1.2.0

    Bigchaindb Server Documentation Release 1.2.0

    BigchainDB Server Documentation Release 1.2.0 BigchainDB Contributors Nov 13, 2017 Contents 1 Introduction 1 2 Quickstart 3 3 Production Nodes 5 4 Clusters 13 5 Production Deployment Template 15 6 Develop & Test BigchainDB Server 65 7 Settings & CLI 73 8 The HTTP Client-Server API 85 9 The Events API 103 10 Drivers & Tools 107 11 Data Models 109 12 Transaction Schema 117 13 Vote Schema 121 14 Release Notes 123 15 Appendices 125 Python Module Index 169 HTTP Routing Table 171 i ii CHAPTER 1 Introduction This is the documentation for BigchainDB Server, the BigchainDB software that one runs on servers (but not on clients). If you want to use BigchainDB Server, then you should first understand what BigchainDB is, plus some of the spe- cialized BigchaindB terminology. You can read about that in the overall BigchainDB project documentation. Note that there are a few kinds of nodes: •A dev/test node is a node created by a developer working on BigchainDB Server, e.g. for testing new or changed code. A dev/test node is typically run on the developer’s local machine. •A bare-bones node is a node deployed in the cloud, either as part of a testing cluster or as a starting point before upgrading the node to be production-ready. •A production node is a node that is part of a consortium’s BigchainDB cluster. A production node has the most components and requirements. 1.1 Setup Instructions for Various Cases • Quickstart • Set up a local BigchainDB node for development, experimenting and testing • Set up and run a BigchainDB cluster There are some old RethinkDB-based deployment instructions as well: • Deploy a bare-bones RethinkDB-based node on Azure • Deploy a RethinkDB-based testing cluster on AWS Instructions for setting up a client will be provided once there’s a public test net.
  • Analyzing Dynamic Capabilities in the Context of Cloud Platform Ecosystems - a Case Study Approach

    Analyzing Dynamic Capabilities in the Context of Cloud Platform Ecosystems - a Case Study Approach

    Junior Management Science 2(3) (2017) 124-172 Volume 2, Issue 3, December 2017 Advisory Editorial Board: DOMINIK VAN AAKEN JUNIOR FREDERIK AHLEMANN CHRISTOPH BODE ROLF BRÜHL MANAGEMENT JOACHIM BÜSCHKEN LEONHARD DOBUSCH RALF ELSAS DAVID FLORYSIAK SCIENCE GUNTHER FRIEDL WOLFGANG GÜTTEL CHRISTIAN HOFMANN Yasmin Le, The State of the Art in Cryptocurrencies 1 KATJA HUTTER LUTZ JOHANNING Fabian Müller, Einfluss von Commitment und Affekten STEPHAN KAISER auf das Investitionsverhalten in Projekten 11 ALFRED KIESER NATALIA KLIEWER Marcus Pfeiffer, Biases bei betriebswirtschaftlichen Junior Management Science Entscheidungen in Großprojekten und DODO ZU KNYPHAUSEN-AUFSEß SABINE T. KÖSZEGI Lösungsansätze: Aktueller Stand der Theorie ARJAN KOZICA und Empirie 48 TOBIAS KRETSCHMER HANS-ULRICH KÜPPER Simon Hux, Ankereffekt und Risikoprämie anhand einer REINER LEIDL Crowdfunding-Kampagne 73 ANTON MEYER GORDON MÜLLER-SEITZ Anastasia Kieliszek, Corporate Divestment Decision GÜNTER MÜLLER-STEWENS Factors: A Systematic Review 104 BURKHARD PEDELL MARCEL PROKOPCZUK Kevin Rudolph, Analyzing Dynamic Capabilities in the TANJA RABL Context of Cloud Platform Ecosystems - A Case SASCHA RAITHEL Study Approach 124 ASTRID REICHEL KATJA ROST MARKO SARSTEDT DEBORAH SCHANZ ANDREAS G. SCHERER STEFAN SCHMID UTE SCHMIEL CHRISTIAN SCHMITZ PHILIPP SCHRECK GEORG SCHREYÖGG LARS SCHWEIZER DAVID SEIDL journal homepage: www.jums.academy THORSTEN SELLHORN ANDREAS SUCHANEK ORESTIS TERZIDIS ANJA TUSCHKE SABINE URNIK STEPHAN WAGNER BARBARA E. WEIßENBERGER ISABELL M. WELPE HANNES WINNER CLAUDIA B. WÖHLE THOMAS WRONA THOMAS ZWICK Published by Junior Management Science e. V. Analyzing Dynamic Capabilities in the Context of Cloud Platform Ecosystems - A Case Study Approach Kevin Rudolph Technische Universität Berlin Abstract Dynamic capabilities (DCs) refer to a firm’s abilities to continuously adapt its resource base in order to respond to changes in its external environment.
  • Package 'Rethinker'

    Package 'Rethinker'

    Package ‘rethinker’ November 13, 2017 Type Package Title RethinkDB Client Version 1.1.0 Author Miron B. Kursa Maintainer Miron B. Kursa <[email protected]> Description Simple, native 'RethinkDB' client. URL https://github.com/mbq/rethinker Suggests testthat Imports rjson License GPL-3 RoxygenNote 6.0.1 NeedsCompilation no Repository CRAN Date/Publication 2017-11-13 11:20:29 UTC R topics documented: close.RethinkDB_connection . .2 close.RethinkDB_cursor . .2 cursorNext . .3 cursorToList . .3 drainConnection . .4 isCursorEmpty . .5 isOpened . .5 openConnection . .6 print.RethinkDB_connection . .6 print.RethinkDB_cursor . .7 r...............................................7 Index 10 1 2 close.RethinkDB_cursor close.RethinkDB_connection Close RethinkDB connection Description Closes connection and stops all associated callbacks and/or sync cursor. Usage ## S3 method for class 'RethinkDB_connection' close(con, ...) Arguments con Connection to close. ... Ignored. close.RethinkDB_cursor Close cursor Description Closes a given cursor and stops its associated query. Should be called on the current cursor before a new sync query is invoked on the same connection. Usage ## S3 method for class 'RethinkDB_cursor' close(con, ...) Arguments con Cursor to close. ... Ignored. cursorNext 3 cursorNext Pull next object from a cursor Description Pulls a datum from a given cursor, sending continuation queries when needed. Usage cursorNext(cursor, inBatch = FALSE) Arguments cursor Cursor to pull from; a result of r()$...$run(...). inBatch If set to TRUE, enables batch mode, i.e., returning the whole local cache (this is usually NOT the whole data available under cursor) rather than a single result. Values other than TRUE or FALSE are invalid. Value In a default mode, a list representing the returned response JSON, or NULL if no data is available.
  • Download.Oracle.Com/Docs/Cd/B28359 01/Network.111/B28530/Asotr Ans.Htm

    Download.Oracle.Com/Docs/Cd/B28359 01/Network.111/B28530/Asotr Ans.Htm

    SCUOLA DI DOTTORATO IN INFORMATICA Tesi di Dottorato XXIV Ciclo A Distributed Approach to Privacy on the Cloud Francesco Pagano Relatore: Prof. Ernesto Damiani Correlatore: Prof. Stelvio Cimato Direttore della Scuola di Dottorato in Informatica: Prof. Ernesto Damiani Anno Accademico 2010/2011 Abstract The increasing adoption of Cloud-based data processing and storage poses a number of privacy issues. Users wish to preserve full control over their sensitive data and cannot accept it to be fully accessible to an external storage provider. Previous research in this area was mostly addressed at techniques to protect data stored on untrusted database servers; however, I argue that the Cloud architecture presents a number of specific problems and issues. This dissertation contains a detailed analysis of open issues. To handle them, I present a novel approach where confidential data is stored in a highly distributed partitioned database, partly located on the Cloud and partly on the clients. In my approach, data can be either private or shared; the latter is shared in a secure manner by means of simple grant-and-revoke permissions. I have developed a proof-of-concept implementation using an in-memory RDBMS with row-level data encryption in order to achieve fine-grained data access control. This type of approach is rarely adopted in conventional outsourced RDBMSs because it requires several complex steps. Benchmarks of my proof- of-concept implementation show that my approach overcomes most of the problems. 2 Acknowledgements I want to thank
  • Linux on Z Platform ISV Strategy Summary

    Linux on Z Platform ISV Strategy Summary

    Linux on IBM Z / LinuxONE Open Source Ecosystem Status and Strategy for NY/NJ Linux Council Meeting on March 1, 2019 Enyu Wang Program Director, Ecosystem Strategy and Business Development [email protected] As an enterprise platform WHY ARE WE INVESTING IN OPEN SOURCE ECOSYSTEM? IBM Z / Open Source Ecosystem / Mar 1, 2019 / © 2019 IBM Corporation 2 TREND: Enterprise Going Open Source • 83% hiring managers surveyed for the 2018 Open Source Jobs report said hiring open source talent was a priority this year • Some of the biggest trends in enterprise IT, such as containers and hybrid cloud, rely on open source technologies including Linux and Kubernetes IBM Z / Open Source Ecosystem / Mar 1, 2019 / © 2019 IBM Corporation 3 OPEN SOURCE Building Blocks for Enterprise Digital Transformation IBM Z / Open Source Ecosystem / Mar 1, 2019 / © 2019 IBM Corporation 4 OUR MISSION Provide a Rich and Robust Ecosystem to Clients. Help Accelerate their Digital Transformation IBM Z / Open Source Ecosystem / Mar 1, 2019 / © 2019 IBM Corporation 5 Rich Open Source Ecosystem on Linux on Z/LinuxONE Analytics/ Distributions Hypervisors PaaS / IaaS Languages Runtimes Management Database ML LPA R IBM Cloud Private Community Versions LLVM vRealize LXD (Ubuntu) Apache Tomcat DPM Db2 IBM Z / Open Source Ecosystem / Mar 1, 2019 / © 2019 IBM Corporation 6 Building an Open Ecosystem Isn’t Just Porting… IBM Z / Open Source Ecosystem / Mar 1, 2019 / © 2019 IBM Corporation 7 Composition of Open Source Ecosystem on Z – a combination of community based projects and vendor
  • Failure Modes and Effects Analysis (FMEA)

    Failure Modes and Effects Analysis (FMEA)

    A R C 3 3 5 - R Designing for failure: Architecting resilient systems on AWS Adrian Cockcroft Harsha Nippani Vinay Kola VP, Cloud Architecture Solutions Architect Software Engineer Amazon Web Services Amazon Web Services Snap Inc. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda • Risk and resilience • Technical considerations • Customer use case: Snap • Continuous resilience • Related sessions • AWS whitepaper © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Everything fails, all the time” - Werner Vogels (CTO, Amazon.com) Business continuity How much data can you afford to How quickly must you recover? recreate or lose? What is the cost of downtime? Disaster Recovery point (RPO) Recovery time (RTO) Data loss Downtime mission Availability by the numbers Level of availability Percent uptime Downtime per year Downtime per day 1 Nine 90% 36.5 Days 2.4 Hours 2 Nines 99% 3.65 Days 14 Minutes 3 Nines 99.9% 8.76 Hours 86 Seconds 4 Nines 99.99% 52.6 Minutes 8.6 Seconds 5 Nines 99.999% 5.26 Minutes 0.86 Seconds Daily Downtime in Seconds 1 Nine 2 Nines 3 Nines 4 Nines 5 Nines 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 Daily Downtime in Seconds Multi-AZ architecture Region • Enables fault-tolerant applications Availability Zone • AWS Regional services designed to withstand AZ failures • Leveraged in the Amazon S3 design for 99.999999999% durability Availability Zone Availability Zone Multi-AZ Zero blast radius! Well-Architected Framework AWS Shared Responsibility Model Resilient AWS infrastructure
  • Startup Attendee Guide

    Startup Attendee Guide

    © 2020, Amazon Web Services, Inc. its affiliates. All rights reserved. or Welcome! © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. On behalf of AWS, we are excited for you to join us for re:Invent 2020! This year’s virtual conference is going to be the industry event of the year. It will offer 5 Keynotes, 18 Leadership Sessions, and unlimited access to hundreds of sessions. We are happy to have you along for the journey! To be sure you are prepared and ready to take on our 3-week event, we’ve created this guide full of tips and tricks provided directly from re:Invent alumni, teams, and staff on how to maneuver our free virtual event successfully. Step 1 – Register Be sure to take advantage of the resources outlined in this document. We’ve curated them to offer you advice on how to plan a successful 3-week agenda. The full schedule, including dates, times, and locations, goes live in early November. In the meantime, use this document to start thinking about how you’ll plan your 3 weeks. We look forward to seeing you there! Leadership Partners & Overview Keynotes Learn Make a Plan Communities Play Checklist Contact Sessions Sponsors Continuous Content Every Week Access hours of sessions led by AWS experts, hear from cloud leaders, and be the first to learn what's next and new from AWS. AWS re:Invent attendees should expect to: • Learn firsthand about AWS services from recognized world experts. • Focus on key services that matter most to their businesses and interests.
  • Dmitry Omelechko

    Dmitry Omelechko

    Dmitry Omelechko E-mail : [email protected] Phone : skype: dvarkin8 Address: Olevska 3 В, Kyiv, Ukraine Objective I have a more than 15 years of experience in IT industry, and I was involved in design and development of several long- term reliable, scalable and high-load platforms and applications, which gave me solid experience of software design. More than 10 years, I prefer functional programming and I have strong understanding of concurrent, scaled and parallel approaches. I'm good team player with strong self-motivation. I always do my best to build great solution for customers needs. Qualifications Programming Languages. Erlang/OTP Elixir/LFE (pet projects) Clojure / ClojureScript Python LISP (Alegro, SBCL) Perl C Data Science Recommendation Systems. DBMS PostgreSQL / EnterpriseDB / GreenPlum (PL/pgSQL, PL/Python). Riak MongoDB RethinkDB Cassandra Datomic KDB+ (pet) Hazelcast / Aerospike Redis / Memcache Application Servers and Frameworks Cowboy / Mochiweb / Ring / Runch Nginx / Apache / Haproxy RabbitMQ / 0mq / Kafka Storm / Samza Django/web.py/TurboGears/Plone/Zope Dmitry Omelechko 1 Histrix Riak / Redux / GraphQL / Om Next Ejabberd Erlyvideo / Flussonic AWS / Google Cloud / Heroku Networks OSPF/BGP/DNS/DHCP, VPNs, IPSec, Software Farewalls (Iptables, PF). Cisco/Extreme networks/Juniper routers and commutators. Clouds Google Cloud Amazon Cloud and AWS Heroku Operating Systems Ubuntu / Gentoo / Slackware OpenBSD Development Tools Emacs / Vim / IntelliJ Idea. git / svn / cvs Rebars / Erlang.mk / Leiningen EUnit / Common Test / Proper / Tsung Ansible / Terraform Methodologies Scrum / Kanban / TDD Work experience SBTech mar 2016 — present Tech Lead Data Science, Bets Recommendation System, User profiling system. Bet Engines sep 2015 — mar 2016 CTO Own startup. System for calculation huge Mathematical models in Excel.