Large Scale Distributed Computing - Summary

Total Page：16

File Type：pdf, Size：1020Kb

Large Scale Distributed Computing - Summary Author: @triggetry, @lanoxx Source available on github.com/triggetry/lsdc-summary December 10, 2013 Contents 1 Preamble 3 2 CHAPTER 1 3 2.1 Most common forms of Large Scale Distributed Computing . 3 2.2 High Performance Computing . 3 2.2.1 Parallel Programming . 5 2.3 Grid Computing . 5 2.3.1 Definitions . 5 2.3.2 Virtual Organizations . 6 2.4 Cloud Computing . 6 2.4.1 Definitions . 6 2.4.2 5 Cloud Characteristics . 7 2.4.3 Delivery Models . 7 2.4.4 Cloud Deployment Types . 7 2.4.5 Cloud Technologies . 7 2.4.5.1 Virtualization . 7 2.4.5.2 The history of Virtualization . 8 2.5 Web Application Frameworks . 9 2.6 Web Services . 9 2.7 Multi-tenancy . 9 2.8 BIGDATA .................................................. 10 3 CHAPTER 2 11 3.1 OS / Virtualization . 11 3.1.1 Batchsystems . 11 3.1.1.1 Common Batch processing usage . 11 3.1.1.2 Portable Batch System (PBS) . 12 3.1.2 VGE - Vienna Grid Environment . 12 3.2 VMs, VMMs . 12 3.2.1 Why Virtualization? . 12 3.2.2 Types of virtualization . 13 3.2.3 Hypervisor vs. hosted Virtualization . 14 3.2.3.1 Type 1 and Type 2 Virtualization . 14 3.2.4 Basic Virtualization Techniques . 14 3.3 Xen ...................................................... 14 3.3.1 Architecture . 14 3.3.2 Dynamic Memory Control (DMC) . 16 3.3.3 Balloon Drivers . 16 3.3.4 Paravirtualization . 16 3.3.5 Domains in Xen . 17 3.3.6 Hypercalls in Xen . 17 3.4 VMWare . 17 3.4.1 Hosted vs. Hypervisor Architecture . 17 3.5 Cloud Management . 18 3.6 OpenNebula . 18 1 3.7 Eucalyptus . 19 3.7.1 Components . 19 3.8 Virtualization - Glossary . 19 4 CHAPTER 3 22 4.1 Self adaptable Clouds: Cloud Monitoring and Knowledge Management . 22 4.1.1 Traditional MAPE Loop . 22 4.1.2 SLA . 22 4.1.3 LoM2His Framework . 22 4.1.4 Cloud Characteristics . 23 4.1.5 How to make Clouds energy-efficient . 23 4.1.6 How to avoid SLA violations . 23 4.1.7 How to structure actions . 25 4.2 Policy Modes . 25 4.3 Cloud Market . 25 4.4 Cloud Characteristics . 25 4.5 Cloud Enabling Technologies . 25 4.6 Problems when providing virtual goods . 26 4.7 Resource markets in Research . 26 4.8 Commercial Resource Providers . 27 4.9 Liquidity Problems in Markets . 27 4.10 The importance of SLAs in markets . 27 4.11 Managing SLAs . 27 4.12 The SLA Template Lifecycle . 27 4.13 SLA Mapping in Double Auctions . 27 4.14 Consequences of few resource types . 29 4.15 Mapping the SLA Landscape for High Performance Clouds . 29 4.16 SLA Mapping Approach . 29 5 CHAPTER 4 30 5.1 Map-Reduce Overview . 30 5.2 Map Reduce Sequence of Actions . 31 5.3 Master Data Structures . 32 5.4 Fault Tolerance . 32 5.4.1 Worker Failure . 32 5.4.2 Master Failure . 32 5.5 Data Flow . 32 5.6 Partitioning function . 33 5.7 Combiner function . 33 5.8 Input and Output Types . 34 5.9 Hadoop . 34 5.10 Hive . 34 5.10.1 HiveQL . 35 5.11 HadoopDB . ..

Copy Link

Recommended publications

Cna Laboratory Enhancement by Virtualisation Centria
Akintomide Akinsola CNA LABORATORY ENHANCEMENT BY VIRTUALISATION Bachelor’s thesis CENTRIA UNIVERSITY OF APPLIED SCIENCES Degree Programme in Information Technology June 2015 ABSTRACT Unit Date Author /s Kokkola-Pietarsaari June 2015 Akintomide Akinsola Degree program me Information Technology Name of thesis CNA LABORATORY ENHANCEMENT BY VIRTUALISATION Instructor Pages 28 + 3 Supervisor The role of the Cisco Networking Academy of Centria University of Applied Sciences in the Media and Communication Technology specialisation is an essential one. An improvement in the infrastructure of the CNA laboratory directly leads to an improvement in the quality of education received in the laboratory. This thesis work described the creation of an alternative arrangement using Linux Ubuntu as a supplementary option in the studying of networking subjects in the CNA laboratory of the university. Linux Ubuntu is a free software available to download and with some adjustments and modifications it can be made to function just as properly as a Windows operating system in the experience of learning networking. The process of creating and deploying a customised Ubuntu image deployed via the Virtual Machine, was discussed in this thesis work. Linux is a UNIX-like software, the knowledge of which is valuable to students as they study to become professionals. An introduction to several applications such as Minicom, Wireshark, and Nmap was also discussed in this thesis. A simple laboratory experiment was designed to test the performance and functioning of the newly created system. Expertise in more than one operating system expands the horizon of future students of the UAS, an opportunity older students did not have, and capacitates them to become proficient engineers.
[Show full text]
Virtual Machines
Virtual Machines Today l VM over time l Implementation methods l Hardware features supporting VM Next time l Midterm? *Partially based on notes from C. Waldspurger, VMware, 2010 Too many computers! An organization can have a bit too many machines – Why many? Running different services (email, web) to ensure • Each has enough resources • … fails independently • ... survives if another one gets attack • ... services may require different Oses – Why too many? Hard and expensive to maintain! Virtualization as an alternative – A VMM (or hypervisor) creates the illusion of multiple (virtual) machines 2 Virtualization applications Server consolidation – Convert underutilized servers to VMs, saving cost – Increasingly used for virtual desktops Simplified management – Datacenter provisioning and monitoring – Dynamic load balancing Improved availability – Automatic restart – Fault tolerance – Disaster recovery Test and development Cloud support – Isolation for clients 3 Types of virtualization Process virtualization – Language-level: Java, .NET, Smalltalk – OS-level: processes, Solaris Zones, BSD Jails – Cross-ISA emulation: Apple 68K-PPC-x86 Device virtualization – Logical vs. physical: VLAN, VPN, LUN, RAID System virtualization – Xen, VMware Fusion, KVM, Palacios … 4 System virtualization starting point Physical hardware – Processors, memory, chipset, I/O devices, etc. – Resources often grossly underutilized App App App App Software OS – Tightly coupled to physical hardware – Single active OS instance Hardware – OS controls hardware CPU MEM NIC 5 Adding a virtualization layer Software abstraction – Behaves like hardware – Encapsulates all OS and App App App App application state Virtualization layer OS OS – Extra level of indirection AppVM App AppVM App – Decouples hardware, OS – Enforces isolation VirtualizationOS layer – Multiplexes physical hardware across VMs Hardware CPU MEM NIC 6 Virtual Machine Monitor Classic definition* … an efﬁcient, isolated duplicate of the real machine.
[Show full text]
Enhancing Cloud Security and Privacy: Time for a New Approach?
View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Aberdeen University Research Archive Enhancing Cloud Security and Privacy: Time for a New Approach? Bob Duncan Alfred Bratterud Andreas Happe Computing Science Dept. of Computer Science Dept. Digital Safety & Security University of Aberdeen Oslo and Akershus University Austrian Inst. of Tech. GmbH Aberdeen, UK Oslo, Norway Vienna, Austria Email: [email protected] Email: [email protected] Email: [email protected] Abstract—Achieving cloud security is not a trivial problem goal of limiting the dependence of the company on the people and developing and enforcing good cloud security controls is a who use the system, thus removing as many opportunities for fundamental requirement if this is to succeed. The very nature human error as possible. of cloud computing can add additional problem layers for cloud security to an already complex problem area. We discuss why this We are concerned with achieving both good security and is such an issue, consider what desirable characteristics should be good privacy. While it is possible to have security without aimed for and propose a novel means of effectively and efﬁciently privacy, it is not possible to have privacy without security. Thus achieving these goals through the use of unikernel based systems. our approach in this paper will be to ﬁrst ensure a good level of The main thrust of this paper is to discuss the key issues which security can be achieved, and to that end, we start by listing the need to be addressed, noting which of those might be covered by our proposed approach.
[Show full text]
(Type I) Hypervisors
(Type I) Hypervisors Lucas Hansen Hypervisors in General ● Virtualize computing systems ● Host and Guest ● Different from emulation ● Instructions execute directly on the hardware ● Two rather different types Type II ● Arguably more widely known ● Programs like VirtualBox and vFusion ● Hypervisors run as a software program within a running operating system ● Host OS still controls 100% of its hardware. ○ It’s process manager schedules the hypervisor just as it does any other process within its scope ○ Guest is only as secure as the host ● Generally Slower ● Used in personal computing environments Type I ● Mainly used in datacenters ● “Bare-Metal Hypervisor” ● To the guests, they serve basically the same function as Type 2 ● Very thin layer between multiple guests and the hardware ● Guests are the only entities running “true” operating systems. ● Sole purpose is to act as a monitor for multiple OS’es ● Qubes OS ○ Previously presented in class History ● 60’s - SIMMON for testing and CP-40/67 for production (Mainframe) ● 70’s-80’ much of the industry development done by IBM in various product lines ○ Support for virtual memory ● 2003 - Project Xen Released ● 2005 - Chipmakers add virtualization support to CPUs ○ Moore’s Law/increase in computing power ○ Proliferation of clustered computing ○ Companies deciding to use services requiring many applications requiring a wide variety of Operating Systems and hardware ● 2008 - VMware ESX(i) ○ Most popular server hypervisor Use Cases ● Resource Pool Sharing ○ Increases hardware utilization ● Reduce
[Show full text]
Architecture for Enabling Edge Inference Via Model Transfer from Cloud Domain in a Kubernetes Environment
future internet Article Architecture for Enabling Edge Inference via Model Transfer from Cloud Domain in a Kubernetes Environment Pekka Pääkkönen *, Daniel Pakkala, Jussi Kiljander and Roope Sarala VTT Technical Research Centre of Finland, 90571 Oulu, Finland; daniel.pakkala@vtt.fi (D.P.); jussi.kiljander@vtt.fi (J.K.); roope.sarala@vtt.fi (R.S.) * Correspondence: pekka.paakkonen@vtt.fi Abstract: The current approaches for energy consumption optimisation in buildings are mainly reactive or focus on scheduling of daily/weekly operation modes in heating. Machine Learning (ML)-based advanced control methods have been demonstrated to improve energy efficiency when compared to these traditional methods. However, placing of ML-based models close to the buildings is not straightforward. Firstly, edge-devices typically have lower capabilities in terms of processing power, memory, and storage, which may limit execution of ML-based inference at the edge. Secondly, associated building information should be kept private. Thirdly, network access may be limited for serving a large number of edge devices. The contribution of this paper is an architecture, which enables training of ML-based models for energy consumption prediction in private cloud domain, and transfer of the models to edge nodes for prediction in Kubernetes environment. Additionally, pre- dictors at the edge nodes can be automatically updated without interrupting operation. Performance results with sensor-based devices (Raspberry Pi 4 and Jetson Nano) indicated that a satisfactory prediction latency (~7–9 s) can be achieved within the research context. However, model switching led to an increase in prediction latency (~9–13 s). Partial evaluation of a Reference Architecture for edge computing systems, which was used as a starting point for architecture design, may be considered as an additional contribution of the paper.
[Show full text]
In Perfect Xen, a Performance Study of the Emerging Xen Scheduler
View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by DigitalCommons@CalPoly IN PERFECT XEN, A PERFORMANCE STUDY OF THE EMERGING XEN SCHEDULER A Thesis presented to the Faculty of California Polytechnic State University San Luis Obispo In Partial Fulfillment of the Requirements for the Degree Master of Science in Computer Science by Ryan Hnarakis December 2013 Copyright 2013 Ryan Hnarakis ALL RIGHTS RESERVED ii COMMITTEE MEMBERSHIP TITLE: Perfect Xen, A Performance Study of the Emerging Xen Scheduler AUTHOR: Ryan Hnarakis DATE SUBMITTED: December 2013 COMMITTEE CHAIR: Phil Nico, PhD Associate Professor of Computer Science COMMITTEE MEMBER: Franz Kurfess, PhD Professor of Computer Science COMMITTEE MEMBER: Lynne Slivovsky, PhD Professor of Electrical Engineering iii Abstract In Perfect Xen, A Performance Study of the Emerging Xen Scheduler by Ryan Hnarakis Fifty percent of Fortune 500 companies trust Xen, an open-source bare- metal hypervisor, to virtualize their websites and mission critical services in the cloud. Providing superior fault tolerance, scalability, and migration, vir- tualization allows these companies to run several isolated operating systems simultaneously on the same physical server [13]. These isolated operating sys- tems, called virtual machines, require a virtual traffic guard to cooperate with one another. This guard known as the Credit2 scheduler along with the newest Xen hypervisor was recently developed to supersede the older schedulers. Since wasted CPU cycles can be costly, the Credit2 prototype must undergo sig- nificant performance validation before being released into production. Fur- thermore, leading commercial virtualization products, including VMWare and Microsoft Hyper-V frequently adopt Xen's proven technologies.
[Show full text]
Wireless and Mobile System Infrastructure
Virtual Machine Systems Question Can a “small” operating system simulate the hardware of some machine so that Another operating system can run in that simulated hardware? More than one instance of that operating system run on the same hardware at the same time? More than one different operating system can share the same hardware at the same time? Answer: Yes Virtual Machine A virtual machine provides interface identical to underlying bare hardware i.e., all devices, storages, memory, page tables, etc. Virtual Machine Operating System creates illusion of multiple processors Each VM executes independently No sharing, except via network protocols History – CP67 / CMS IBM Cambridge Scientific Center Ran on IBM 360/67 Alternative to TSS/360, which never sold very well Replicated hardware in each “process” Virtual 360/67 processor Virtual disk(s), virtual console, printer, card reader, etc. Cambridge Monitor System (CMS) A single user, interactive operating system Commercialized as VM370 in mid-1970s History (cont.) Various other attempts with other machines VMware Workstation Servers (for IT centers) “Classic” Virtual Machine Copy of a real machine “Any program run under the VM has an effect identical with that demonstrated if the program had been run in the original machine directly” 1 Isolated from other virtual machines “…transforms the single machine interface into the illusion of many” 2 Efficient “A statistically dominant subset of the virtual processor’s instructions is executed directly by the real processor” 2 Also known as a “system VM” 1 “Formal Requirements for Virtualizable Third-Generation Architectures”, G. Popek and R. Goldberg, Communications of the ACM, 17(7), July 1974 2 “Survey of Virtual Machine Research”, R.
[Show full text]
Cs 370: Operating Systems [Virtualization]
CS370: Operating Systems [Fall 2018] Dept. Of Computer Science, Colorado State University CS 370: OPERATING SYSTEMS [VIRTUALIZATION] Shrideep Pallickara Computer Science Colorado State University November 13, 2018 CS370: Operating Systems [Fall 2018] L25.1 Dept. Of Computer Science, Colorado State University Frequently asked questions from the previous class survey ¨ Belady‘s anomaly and local frame replacement policies? ¨ Multiprogramming? November 13, 2018 CS370: Operating Systems [Fall 2018] L25.2 Professor: SHRIDEEP PALLICKARA Dept. Of Computer Science, Colorado State University SLIDES CREATED BY: SHRIDEEP PALLICKARA L25.1 CS370: Operating Systems [Fall 2018] Dept. Of Computer Science, Colorado State University Topics covered in this lecture ¨ Virtualization November 13, 2018 CS370: Operating Systems [Fall 2018] L25.3 Professor: SHRIDEEP PALLICKARA Dept. Of Computer Science, Colorado State University What we will look at Why? Advantages Memory and I/O Virtualization Virtualization Brief History Type 1 and Type 2 hypervisors Requirements Techniques for efficiency November 13, 2018 CS370: Operating Systems [Fall 2018] L25.4 Professor: SHRIDEEP PALLICKARA Dept. Of Computer Science, Colorado State University SLIDES CREATED BY: SHRIDEEP PALLICKARA L25.2 CS370: Operating Systems [Fall 2018] Dept. Of Computer Science, Colorado State University WHY VIRTUALIZATION November 13, 2018 CS370: Operating Systems [Fall 2018] L25.5 Dept. Of Computer Science, Colorado State University Firms often have multiple, dedicated servers: e-mail, FTP, e-commerce, web, etc. ¨ Load: Maybe one machine cannot handle all that load ¨ Reliability: Management does not trust the OS to run 24 x 7 without failures ¨ By putting one server on a separate computer, if one of the server crashes? ¤ At least the other ones are not affected ¨ If someone breaks into the web server, at least sensitive e-mails are still protected ¤ Sandboxing November 13, 2018 CS370: Operating Systems [Fall 2018] L25.6 Professor: SHRIDEEP PALLICKARA Dept.
[Show full text]
Comparative Performance Analysis of the Virtualization Technologies in Cloud Computing
International Journal of Engineering Research & Technology (IJERT) ISSN: 2278-0181 Vol. 3 Issue 9, September- 2014 Comparative Performance Analysis of the Virtualization Technologies in Cloud Computing 1 2 Zinnia Sultana Samjhana Rayamajhi Department of Computer Scienceand Engineering, Department of Computer Scienceand Engineering, International Islamic University Chittagong. International Islamic University Chittagong. Chittagong, Bangladesh. Chittagong, Bangladesh. Abstract—A hypervisor or virtual machine monitor (VMM) metal hypervisors which runs directly on a hardware, in a is a piece of computer software, firmware or hardware that sense they are both hypervisor and an OS. creates and runs virtual machines which makes multi-tenancy possible. Multi-tenancy allows multiple tenants to coexist in the II. RELATED RESEARCH same physical machine sharing its resources and at the same time, creates an isolated environment for each of them. Cloud service providers (CSP) can maximize their infrastructures In recent years virtualization has gained popularity using this architecture by allocating resources from physical in many different areas such as server consolidation, machines that are not being fully used. Multi tenancy can be information security and cloud computing due to an obtained by virtualization, which is the future in the IT world. increase in hardware performance of about ten fold in the This research paper provides concept of virtualization along past decade and the goal to reduce capital and operational with the performance comparison of some common costs within the data center. [3] virtualization technologies using many benchmarks which is chosen as it gives a good idea how the hypervisor’s performance In recent history there have been many work of is.
[Show full text]
Cyber-Physical Application Monitoring Across Multiple Clouds ∗
Cyber-Physical Application Monitoring across Multiple Clouds ∗ Ayman Noora,b, Karan Mitrac, Ellis Solaimana, Arthur Souzad, Devki Nandan Jhaa, Umit Demirbagaa,e, Prem Prakash Jayaramanf, Nelio Cachod, Rajiv Ranjana aNewcastle University, Newcastle upon Tyne, UK bTaibah University, Madinah, Saudi Arabia cLule˚aUniversity of Technology, Skellefte˚a,Sweden dFederal University of Rio Grande do Norte, Natal, Brazil eBartin University, Bartin, Turkey fSwinburne University of Technology, Melbourne, Australia Abstract Cyber-physical systems (CPS) integrate cyber-infrastructure comprising com- puters and networks with physical processes. The cyber components monitor, control, and coordinate the physical processes typically via actuators. As CPS are characterized by reliability, availability, and performance, they are expected to have a tremendous impact not only on industrial systems but also in our daily lives. We have started to witness the emergence of cloud-based CPS. However, cloud systems are prone to stochastic conditions that may lead to quality of service degradation. In this paper, we propose M2CPA - a novel framework for multi-virtualization, and multi-cloud monitoring in cloud-based cyber-physical systems. M2CPA monitors the performance of application components running inside multiple virtualization platforms deployed on multiple clouds. M2CPA is validated through extensive experimental analysis using a real testbed compris- ing multiple public clouds and multi-virtualization technologies. Keywords: Cyber-physical system, Monitoring, Linear Road benchmark, QoS, Virtualization, Cloud Computing ∗ Preprint submitted to Computers and Electrical Engineering June 17, 2019 1. Introduction CPS is an interdisciplinary approach for combining communication devices, computation, and actuation for performing time-constrained actions in a predic- tive and adaptive manner [1, 2]. This is done using a feedback loop within the physical system, which enables the embedded and network systems to monitor and control the physical processes.
[Show full text]
Wireless and Mobile System Infrastructure
Virtual Machine Systems Question Can a “small” operating system simulate the hardware of some machine so that Another operating system can run in that simulated hardware? More than one instance of that operating system run on the same hardware at the same time? More than one different operating systems can share the same hardware at the same time? Answer: Yes Virtual Machine A virtual machine provides interface identical to underlying bare hardware i.e., all devices, storages, memory, page tables, etc. Virtual Machine Operating System creates illusion of multiple processors Each VM executes independently No sharing, except via network protocols History – CP67 / CMS IBM Cambridge Scientific Center Ran on IBM 360/67 Alternative to TSS/360, which never sold very well Replicated hardware in each “process” Virtual 360/67 processor Virtual disk(s), virtual console, printer, card reader, etc. Cambridge Monitor System (CMS) A single user, interactive operating system Commercialized as VM370 in mid-1970s History (cont.) Various other attempts with other machines VMware Workstation Servers (for IT centers) “Classic” Virtual Machine Copy of a real machine “Any program run under the VM has an effect identical with that demonstrated if the program had been run in the original machine directly” 1 Isolated from other virtual machines “…transforms the single machine interface into the illusion of many” 2 Efficient “A statistically dominant subset of the virtual processor’s instructions is executed directly by the real processor” 2 Also known as a “system VM” 1 “Formal Requirements for Virtualizable Third-Generation Architectures”, G. Popek and R. Goldberg, Communications of the ACM, 17(7), July 1974 2 “Survey of Virtual Machine Research”, R.
[Show full text]
A Study on Resource Pooling, Allocation and Virtualization Tools Used for Cloud Computing
International Journal of Computer Applications (0975 – 8887) Volume 168 – No.2, June 2017 A Study on Resource Pooling, Allocation and Virtualization Tools used for Cloud Computing Aaqib Rashid Amit Chaturvedi, PhD Ph.D. Scholar, Mewar Univ, MCA Deptt, Govt Engg College, Chittorgarh, Rajasthan, India Ajmer, Rajasthan, India ABSTRACT b. Virtual Server Pools: The virtual server, also known as In cloud computing various types of resources like CPU, virtual machine (VM), is a form of virtualization software Memory, OS, Application Software etc. are used. A cloud that emulates a physical server and is used by cloud server, which has sufficient resources all the time for its providers to share the same physical server with multiple clients as resource pools, efficiently and dynamically allocates cloud consumers by providing cloud consumers with or deallocates these resources, is considered good for its individual virtual server instances. Figure 1 shows three clients. virtual servers being hosted by two physical servers. The number of instances a given physical server can share is Here, in this paper, we have presented a study on various limited by its capacity. types of resource pools like physical server pools, virtual server pools, storage pools, network pools etc available in cloud computing, various allocation strategies, and virtualization tools used in cloud computing to manages these resources for client’s request. This study will be useful to understand the underlying technologies of cloud computing. General Terms VMM [Virtual Machine Monitor], VM [Virtual Machine], hypervisor, cloud computing, Servers, Pools. Keywords Resource pooling, cloud computing, virtualization, hypervisor, virtual machine, allocation, deallocation. Figure 1: Virtual Server Pools 1.
[Show full text]