DOCSLIB.ORG

Containers Demystified DOAG 2019

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Containers Demystified DOAG 2019 Containers Demystified DOAG 2019 Daniel Westermann - Jan Karremans AGENDA Intro History of containers What happened to Docker?? History k8s WHY Automation of the stuff Demo Intro …boring About me Daniel Westermann Principal Consultant Open Infrastructure Technology Leader +41 79 927 24 46 daniel.westermann[at]dbi-services.com EDB containers in Red Hat OpenShift 19.11.2019 Page 4 Who we are The Company Founded in 2010 More than 70 specialists Specialized in the Middleware Infrastructure The invisible part of IT Customers in Switzerland and all over Europe Our Offer Consulting Service Level Agreements (SLA) Trainings License Management dbi services Template - DOAG 2019 19.11.2019 Page 5 Jan Karremans @Johnnyq72 BACKGROUND 25 years of database technology 15 years of consulting 15 years of management 10 years of software development 10 years of technology sales 5 years of community advocacy 5 years of international public speaking EXPERTISE Oracle ACE Alumni EDB Postgres Advanced Server Professional Leader in the PostgreSQL community To Postgres what RedHat is to Linux EnterpriseDB Enterprise-grade Postgres The Most Complete Open Source Database Platform Freeing companies from vendor lock-in • Bruce Momjian • PostgreSQL, Global Development Group, Founding member • PostgreSQL, Core Team member • EnterpriseDB, Senior Database Architect • Mr. Postgres History of containers In the beginning • It was 1960… Bill Joy Co-founder of Sun Microsystems • Sharing 1 single computer with many users • It was then 1979… • Introducing chroot • Bill Joy added chroot to Unix 7 in March / ’82 • Then it was 1990 • Crackers aren’t hackers William (Bill) Cheswick • Bill Cheswick creates the foundations for jail 11 chroot • Create a new root-directory for the current running process and offspring • Basic ”isolation” for processes • Quite easy to “escape” from 12 jail • Made usable on FreeBSD • Partitions a computer into independent mini-systems • Adding virtualization and better security to a system 13 In more recent years • It became a new millenium • The jail-command gets the ability to add an IP address and custom configuration • And now it goes fast, 2004 • Solaris introduces containers (!) by using Solaris Zone technology • 2006, we write… • Google launches cgroups for isolation and limitation of resource usage • Actually frantically trying to avoid the word “Container”! 14 “Today” • Calendar flips to 2008 • cgroups becomes LCX (Linux Containers) as part of the kernel. • Meanwhile we land in 2013 • Google launches the open source project LMCTFY • LMCTFY is contributed to the Docker project libcontainer. • Docker Hub is now home to over 100,000 container images • Many repositories available (RedHat, openSuse, Quai.io, amm) 15 So… • A container runs until it dies. • It fails over • Or it restarts • It is effectively “built to break” • “The avalange has already started, it’s too late for the pebbles to vote.” • Kosh, Babylon 5, “Believers”-episode Mr. Michael Willer - EDB 16 What happened to Docker?? Docker Swarm • Docker Compose • Defining and running Docker multi-container setup (services) • Docker Swarm • Docker container orchestration software 18 History of K8s A short history • There was Google • Based on the Google Borg Cluster Manager • Kubernetes or k8s was originally code-named… • Google & Linux Foundation form Cloud Native Computing Foundation [CNCF] 21 k8s • Written in Go • Pods Logical group of containers • Replica sets Set of instantiations of a specific Pod • Services Logical group of kinned Pods • Volumes Persistent storage • Namespaces Partitioning for dev / test / prod • Configmaps & secrets Configuration data for the pods • Stateful sets EnaBle stateful workloads… like Postgres 22 WHY Big things in little containers Perspectives • Business • Operations • Increase compute density • Manageability • Standardization • Automation • Idempotence • Development • Portability • Flexibility & agility !! 26 Automation of the stuff Why is it important • Lots of repetitive tasks • Everything is aimed at similarity 28 Why databases in containers Top 3 reasons from the field Laetitia Avrot, Postgres expert and Senior Database Consultant at EnterpriseDB: Instantaneous online physical backup with snapshots (made possible thanks to the “pausing PID1 on my container”, something you won’t find with VM) Physical streaming replication by mirroring storage. That’s easy to implement and again, the “pausing PID1 on my container” makes it possible without corruption risk Simplified high availability as a container only stores processes and memory, if it dies, it can “say” it and another container will take it from here. Switchover and failover are way quicker 29 When not to containerize • Just for the sake of… • Wishing for miraculous advantages • To become agile • Implementing containers does not make it micro services 30 And now we stop talking… Questions & Answers CONFIDENTIAL © Copyright EnterpriseDB Corporation, 2019. All rights reserved..
Load more

Recommended publications
  • Systematic Evaluation of Sandboxed Software Deployment for Real-Time Software on the Example of a Self-Driving Heavy Vehicle
    Systematic Evaluation of Sandboxed Software Deployment for Real-time Software on the Example of a Self-Driving Heavy Vehicle Philip Masek1, Magnus Thulin1, Hugo Andrade1, Christian Berger2, and Ola Benderius3 Abstract— Companies developing and maintaining software-only products like web shops aim for establishing persistent links to their software running in the field. Monitoring data from real usage scenarios allows for a number of improvements in the software life-cycle, such as quick identification and solution of issues, and elicitation of requirements from previously unexpected usage. While the processes of continuous integra- tion, continuous deployment, and continuous experimentation using sandboxing technologies are becoming well established in said software-only products, adopting similar practices for the automotive domain is more complex mainly due to real-time and safety constraints. In this paper, we systematically evaluate sandboxed software deployment in the context of a self-driving heavy vehicle that participated in the 2016 Grand Cooperative Driving Challenge (GCDC) in The Netherlands. We measured the system’s scheduling precision after deploying applications in four different execution environments. Our results indicate that there is no significant difference in performance and overhead when sandboxed environments are used compared to natively deployed software. Thus, recent trends in software architecting, packaging, and maintenance using microservices encapsulated in sandboxes will help to realize similar software and system Fig. 1. Volvo FH16 truck from Chalmers Resource for Vehicle Research engineering for cyber-physical systems. (Revere) laboratory that participated in the 2016 Grand Cooperative Driving Challenge in Helmond, NL. I. INTRODUCTION Companies that produce and maintain software-only prod- ucts, i.e.
    [Show full text]
  • An Empirical Study of Virtualization Versus Lightweight Containers
    On Exploiting Page Sharing in a Virtualized Environment - an Empirical Study of Virtualization Versus Lightweight Containers Ashish Sonone, Anand Soni, Senthil Nathan, Umesh Bellur Department of Computer Science and Engineering IIT Bombay Mumbai, India ashishsonone, anandsoni, cendhu, [email protected] Abstract—While virtualized solutions are firmly entrenched faulty application can result in the failure of all applications in cloud data centers to provide isolated execution environ- executing on that machine. On the other hand, each virtual ments, the chief overhead it suffers from is that of memory machine runs its own OS and hence, a faulty application consumption. Even pages that are common in multiple virtual machines (VMs) on the same physical machine (PM) are not cannot crash the host OS. Further, each application can shared and multiple copies exist thereby draining valuable run on any customized and optimized guest OS which is memory resources and capping the number of VMs that can be different from the host OS. For example, each guest OS can instantiated on a PM. Lightweight containers (LWCs) on the run its own I/O scheduler which is optimized for the hosted other hand does not suffer from this situation since virtually application. In addition to this, the hypervisor provides many everything is shared via Copy on Write semantics. As a result the capacity of a PM to host LWCs is far higher than hosting other features such as live migration of virtual machines, equivalent VMs. In this paper, we evaluate a solution using high availability using periodic checkpointing and storage uKSM to exploit memory page redundancy amongst multiple migration.
    [Show full text]
  • Large-Scale Cluster Management at Google with Borg Abhishek Verma† Luis Pedrosa‡ Madhukar Korupolu David Oppenheimer Eric Tune John Wilkes Google Inc
    Large-scale cluster management at Google with Borg Abhishek Vermay Luis Pedrosaz Madhukar Korupolu David Oppenheimer Eric Tune John Wilkes Google Inc. Abstract config file command-line Google’s Borg system is a cluster manager that runs hun- borgcfg webweb browsers tools dreds of thousands of jobs, from many thousands of differ- ent applications, across a number of clusters each with up to Cell BorgMaster tens of thousands of machines. BorgMasterBorgMaster UIUI shard shard BorgMasterBorgMaster read/UIUIUI shard shard It achieves high utilization by combining admission con- shard Scheduler persistent store trol, efficient task-packing, over-commitment, and machine scheduler (Paxos) link shard linklink shard shard sharing with process-level performance isolation. It supports linklink shard shard high-availability applications with runtime features that min- imize fault-recovery time, and scheduling policies that re- duce the probability of correlated failures. Borg simplifies life for its users by offering a declarative job specification Borglet Borglet Borglet Borglet language, name service integration, real-time job monitor- ing, and tools to analyze and simulate system behavior. We present a summary of the Borg system architecture and features, important design decisions, a quantitative anal- Figure 1: The high-level architecture of Borg. Only a tiny fraction ysis of some of its policy decisions, and a qualitative ex- of the thousands of worker nodes are shown. amination of lessons learned from a decade of operational experience with it. cluding with a set of qualitative observations we have made from operating Borg in production for more than a decade. 1. Introduction The cluster management system we internally call Borg ad- 2.
    [Show full text]
  • Resource Monitoring of Docker Containers
    CoMA: Resource Monitoring of Docker Containers Lara Lorna Jimenez,´ Miguel Gomez´ Simon,´ Olov Schelen,´ Johan Kristiansson, Kare˚ Synnes and Christer Ahlund˚ Dept. of Computer Science, Electrical and Space Engineering, Lulea˚ University of Technology, Lulea,˚ Sweden Keywords: Docker, Containers, Containerization, OS-level Virtualization, Operating System Level Virtualization, Virtualization, Resource Monitoring, Cloud Computing, Data Centers, Ganglia, sFlow, Linux, Open-source, Virtual Machines. Abstract: This research paper presents CoMA, a Container Monitoring Agent, that oversees resource consumption of operating system level virtualization platforms, primarily targeting container-based platforms such as Docker. The core contribution is CoMA, together with a quantitative evaluation verifying the validity of the mea- surements reported by the agent for three metrics: CPU, memory and block I/O. The proof-of-concept is implemented for Docker-based systems and consists of CoMA, the Ganglia Monitoring System and the Host sFlow agent. This research is in line with the rising trend of container adoption which is due to the resource efficiency and ease of deployment. These characteristics have set containers in a position to topple virtual machines as the reigning virtualization technology in data centers. 1 INTRODUCTION 2013). On top of a shared Linux kernel, several of what are generally referred to as “containers” run a series of processes in different user spaces Traditionally, virtual machines (VMs) have been the (Elena Reshetova, 2014). In layman’s terms, OS- underlying infrastructure for cloud computing ser- level virtualization generates virtualized instances of vices(Ye et al., 2010). Virtualization techniques kernel resources, whereas hypervisors virtualize the spawned from the need to use resources more effi- hardware.
    [Show full text]
  • Fast Delivery of Virtual Machines and Containers : Understanding and Optimizing the Boot Operation Thuy Linh Nguyen
    Fast delivery of virtual machines and containers : understanding and optimizing the boot operation Thuy Linh Nguyen To cite this version: Thuy Linh Nguyen. Fast delivery of virtual machines and containers : understanding and optimizing the boot operation. Distributed, Parallel, and Cluster Computing [cs.DC]. Ecole nationale supérieure Mines-Télécom Atlantique, 2019. English. NNT : 2019IMTA0147. tel-02418752 HAL Id: tel-02418752 https://tel.archives-ouvertes.fr/tel-02418752 Submitted on 19 Dec 2019 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. THESE DE DOCTORAT DE L’ÉCOLE NATIONALE SUPERIEURE MINES-TELECOM ATLANTIQUE BRETAGNE PAYS DE LA LOIRE - IMT ATLANTIQUE COMUE UNIVERSITE BRETAGNE LOIRE ECOLE DOCTORALE N° 601 Mathématiques et Sciences et Technologies de l'Information et de la Communication SpéciAlité : InformAtique et Applications Par Thuy Linh NGUYEN Fast delivery of Virtual Machines and Containers: Understanding and optimizing the boot operation Thèse présentée et soutenue à Nantes, le 24 Septembre 2019 Unité de recherche : Inria Rennes Bretagne Atlantique Thèse N° : 2019IMTA0147 Rapporteurs avant soutenance : MariA S. PEREZ Professeure, UniversidAd PolitecnicA de MAdrid, EspAgne Daniel HAGIMONT Professeur, INPT/ENSEEIHT, Toulouse, France Composition du Jury : Président : Mario SUDHOLT Professeur, IMT AtlAntique, FrAnce Rapporteur : MariA S.
    [Show full text]
  • Practical Improvements to User Privacy in Cloud Applications
    Practical Improvements to User Privacy in Cloud Applications Raymond Cheng A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy University of Washington 2017 Reading Committee: Thomas Anderson, Chair Arvind Krishnamurthy, Chair Tadayoshi Kohno Program Authorized to Offer Degree: Paul G. Allen School of Computer Science and Engineering c Copyright 2017 Raymond Cheng University of Washington Abstract Practical Improvements to User Privacy in Cloud Applications Raymond Cheng Co-Chairs of the Supervisory Committee: Professor Thomas Anderson Paul G. Allen School of Computer Science and Engineering Professor Arvind Krishnamurthy Paul G. Allen School of Computer Science and Engineering As the cloud handles more user data, users need better techniques to protect their privacy from adversaries looking to gain unauthorized access to sensitive data. Today's cloud services offer weak assurances with respect to user privacy, as most data is processed unencrypted in a centralized loca- tion by systems with a large trusted computing base. While current architectures enable application development speed, this comes at the cost of susceptibility to large-scale data breaches. In this thesis, I argue that we can make significant improvements to user privacy from both external attackers and insider threats. In the first part of the thesis, I develop the Radiatus ar- chitecture for securing fully-featured cloud applications from external attacks. Radiatus secures private data stored by web applications by isolating server-side code execution into per-user sand- boxes, limiting the scope of successful attacks. In the second part of the thesis, I focus on a simpler messaging application, Talek, securing it from both external and insider threats.
    [Show full text]
  • Managing Linux Resources with Cgroups
    Managing Linux Resources with cgroups Thursday, August 13, 2015: 01:45 PM - 02:45 PM, Dolphin, Americas Seminar Richard Young Executive I.T. Specialist IBM Systems Lab Services Agenda • Control groups overview • CPU resource examples • Control groups what is new? • Memory resource examples • Control groups configuration • Namespaces and containers • Assignment and display of cgroup 2 2 Linux Control Groups • What are they? – Finer grain means to control resources among different processes in a Linux instance. Besides limiting access to an amount of resource, it can also prioritize, isolate, and account for resource usage. – Control groups are also known as “cgroups” – Allow for control of resources such as • CPU • IO • Memory • Others… • Network – libcgroup package can be used to more easily manage cgroups. Set of userspace tools. It contains man pages, commands, configuration files, and services (daemons) • Without libcgroup all configuration is lost at reboot • Without libcgroup tasks don’t get automatically assigned to the proper cgroup – Two configuration files in /etc • cgconfig.conf and cgrules.conf 3 Where might they be of value? • Linux servers hosting multiple applications, workloads, or middleware component instances • Resource control or isolation of misbehaving applications Memory leaks or spikes CPU loops Actively polling application code • Need to limit an application or middleware to a subset of resources For example 2 of 10 IFLs 2 of 40 GB of memory or memory and swap (includes limiting filesystem cache) Assign a relative priority to one workload over another Throttle CPU to a fraction of available CPU. • Making more resource available to other workloads in the same server • Making more resource available to other server in the same or other z/VMs or LPARs.
    [Show full text]
  • Containers & Kubernetes
    Containers & Kubernetes A crash course $ whoami - Bob Bob Killen [email protected] Senior Research Cloud Administrator CNCF Ambassador Github: @mrbobbytables Twitter: @mrbobbytables $ whoami - Jeff Jeffrey Sica [email protected] Senior Research Database Administrator Github: @jeefy Twitter: @jeefy What is an application, really ● Code ● Execution environment ○ Language-level libraries (pip or npm) ○ OS-level libraries (libjpg or libmysql) ● Potentially external dependencies/services ○ Database (Postgres, MySQL etc) ○ Caching layer (redis or memcached) What is a Container? A means to easily isolate, package, deliver, and deploy code. A “Shipping Container” for your code. Image Source: HANJIN container by John A. https://twitter.com/IanColdwater/status/1119396748869472256 "I like putting apps into containers because then I can pretend they're not my problem." @sadoperator Image Source: Hyundai What a Container is Not ● A VM* ○ Containers are not a VM. Docker is not a hypervisor. ● Persistent ○ Containers are ephemeral in nature. ● Secure by Default ○ A container is not a security panacea. ○ They DO have the capability of being more secure, but effort is required. * Some “Sandbox” containerizers spin up a container within a VM for each container. Why Are Containers a Thing? ● Workloads are becoming multi-tier and more complex. ● Application(s) and their associated infrastructure are more dynamic. Application infrastructure now must “react” to changes and events. ● Developers can work locally and push globally. ● Facilitates a “microservice” architecture. Why Are Containers a Thing? ● Reproducible ● Portable ● Easy to “plug-in” and link components What is a Container...Really? What is a Container...Really? A process What is a Container...Really? A process (with some additional properties) What is a Container...Really? ● Isolated ○ Kernel handles application process namespace separation.
    [Show full text]
  • Computer Security 04
    Computer Security 04. Confinement Paul Krzyzanowski Rutgers University Fall 2019 September 30, 2019 CS 419 © 2019 Paul Krzyzanowski 1 Compromised applications • Some services run as root • What if an attacker compromises the app and gets root access? – Create a new account – Install new programs – “Patch” existing programs (e.g., add back doors) – Modify configuration files or services – Add new startup scripts (launch agents, cron jobs, etc.) – Change resource limits – Change file permissions (or ignore them!) – Change the IP address of the system • Even without root, what if you run a malicious app? – It has access to all your files – Can install new programs in your search path – Communicate on your behalf September 30, 2019 CS 419 © 2019 Paul Krzyzanowski 2 How about access control? • Limit damage via access control – E.g., run servers as a low-privilege user – Proper read/write/search controls on files … or role-based policies • ACLs don't address applications – Cannot set permissions for a process: “don’t allow access to anything else” – At the mercy of default (other) permissions • We are responsible for changing protections of every file on the system that could be accessed by other – And hope users don’t change that – Or use more complex mandatory access control mechanisms … if available Not high assurance September 30, 2019 CS 419 © 2019 Paul Krzyzanowski 3 We can regulate access to some resources POSIX setrlimit() system call – Maximum CPU time that can be used – Maximum data size – Maximum files that can be created – Maximum
    [Show full text]
  • Disertacija.Pdf
    UNIVERZITET SINGIDUNUM BEOGRAD DEPARTMAN ZA POSLEDIPLOMSKE STUDIJE I MEĐUNARODNU SARADNJU UNAPREĐENJE PERFORMANSI OBRAZOVNIH SISTEMA PRIMENOM BEŽIČNIH MESH MREŽA - DOKTORSKA DISERTACIJA- Mentor: Kandidat: prof. dr Mladen Veinović Aleksandar Zakić, master BEOGRAD, 2015. UNIVERSITY OF SINGIDUNUM BELGRADE DEPARTMANT OF POSTGRADUATE STUDIES AND INTERNATIONAL COOPERATION DOCTORAL IMPROVING PERFORMANCE LEARNING SYSTEM USING WIRELESS MESH NETWORK - DOCTORAL DISSERTATION- Mentor: Candidate: prof. Mladen Veinović, PhD Aleksandar Zakić, master BELGRADE, 2015. SAŽETAK Nastava iz oblasti bežičnog umrežavanja i mobilnih mreža predstavlja neophodnu oblast bez koje današnje savremeno prihvatanja znanja iz računarskih nauka ne bi moglo da se sprovede. Studenti uče apstraktne pojmove o radu ovih sistema, ali se i svakodnevno susreću s različitim tehnologijama umrežavanja. Jedan od novih načina za povezivanje apstraktnih koncepata jeste korišćenje savremenih tehnologija kao što su računarstvo u oblaku (Cloud computing), emulatori i virtualizacija. Suština unapređenja performansi obrazovnih sistema se ogleda u izgradnji distribuiranih aplikacija primenom savremenih tehnologija. Za izgradnju ovakvog okruženja potrebno je da se studenti i predavači aktivno uključe u proces razvoja novih tehnologija, da bi stvorili optimalne uslove za eksperimente i nova naučna ostvarenja u oblasti informacionih i komunikacionih tehnologija. ABSTRACT Teaching in the field wireless and mobile network represent necessary area without which today society and acceptance of computer knowledge can’t be conducted. Students learn abstract notions about this systems, but they also on daily basis come across different technologies of networking. One of the newest ways for connecting abstract concepts is use of modern technologies like Cloud computing, emulators and virtualization. The essence of improvement performance of education systems is shown through building distributed applications by using modern technology.
    [Show full text]
  • Container Security Powered by Devsecops- Happiest Minds
    Container Security powered by DevSecOps Meaning of Container Containers are a type of software, which isolates environment running within a host machine’s kernel that allows us to run application-specific code. Decoupling the containers-based applications can be deployed across an environment which can be either on the public/private cloud, personal laptop and data center as well. Evolution of containers Below timeline shows a brief history of containers History of containers 2000 :Free 2004 : Solaris 2006 : Process BSD jails containers containers 1979 : Unix 2001 : Linux 2005 : Open VZ version-7 Vserver Unix V7 (1979) Chroot system call was introduced in 1979 in Unix version -7. Changing the root (/) directory of process and corresponding child process to a new location in the file system. It was the beginning of process isolation, by parting file access to for each process. FreeBSD Jails BSD (Berkeley Software Distribution) incorporated chroot in 1982, FreeBSD Jails facili- tated administrators to partition a FreeBSD system into several independent, chunks of systems called Jails. Also, FreeBSD Jails can be assigned with IP address for each system and configuration. Evolution of Jails ease administration which will isolate the service and customer security. Container Security powered by DevSecOps 02 Linux VServer Linux Vserver like FreeBSD Jails, partition resources, this can be implemented by patching the Linux kernel. Solaris container Solaris Containers constitutes system resource controls (file systems, network addresses, memory), and boundary separated by zones that leveraged features like snapshots and cloning of zone file system provided by zones. Open VZ Open VZ is an OS-level virtualization technology for Linux which uses a patched Linux kernel for virtualization, isolation, resource management and check pointing.
    [Show full text]
  • Adoption of Container-Based Virtualization in IT Education
    Paper ID #16188 Adoption of Container-Based Virtualization in IT Education Dr. Tae-Hoon Kim, Purdue University - Calumet Dr. Keyuan Jiang, Purdue University - Calumet Mr. Vivek Singh Rajput c American Society for Engineering Education, 2016 Adoption of Container-based Virtualization in IT Education Abstract Hands-on experience through the lab is one of the key components in Information Technology education because it provides students with an opportunity to learn and observe how to apply the concepts. Generally, the lab in IT education requires a variety of equipment such as PCs, servers, switches, and so forth. And virtual environment becomes an essential component in IT education since virtually multiple systems can be created, modified, tested, and deleted easily with little or no cost. Hypervisor virtual machines have been heavily used in IT education for last decade, but sharing the large resources with the host machine becomes a major limitation of such technology. Recently, emergence of container-based virtual machines provides a lightweight virtual environment over hypervisor-based machines. This emerging technology has not been explored well and adopted in IT education. In this paper, we will investigate and explore container-based virtual machines to provide framework for its adoption in IT education. We will describe step-by-step instructions on creating a cluster in single host and across different hosts using container-based virtualization technology. Afterward, we will evaluate and compare the performance of such implementation with a cluster built on physical machine. 1 Introduction The practical exercises included in laboratory-based course play an important role in engineering and science educations.
    [Show full text]