A Service for Provisioning Compute Infrastructure in the Cloud
Total Page：16
File Type：pdf, Size：1020Kb
UPTEC IT 19013 Examensarbete 30 hp Augusti 2019 A Service for Provisioning Compute Infrastructure in the Cloud Tony Wang Institutionen för informationsteknologi Department of Information Technology Abstract A Service for Provisioning Compute Infrastructure in the Cloud Tony Wang Teknisk- naturvetenskaplig fakultet UTH-enheten The amount of data has grown tremendously over the last decade. Cloud computing is a solution to handle large-scale computations and immense data sets. However, Besöksadress: cloud computing comes with a multitude of challenges that scientist who are using the Ångströmlaboratoriet Lägerhyddsvägen 1 data have to tackle. Provisioning and orchestration cloud infrastructure is a challenge Hus 4, Plan 0 in itself with a wide variety of applications and cloud providers that are available. This thesis explores the idea of simplifying the provisioning of computing cloud applications Postadress: in the cloud. The result of this work is a service which can seamlessly provision and Box 536 751 21 Uppsala execute cloud computations using different applications and cloud providers. Telefon: 018 – 471 30 03 Telefax: 018 – 471 30 00 Hemsida: http://www.teknat.uu.se/student Handledare: Salman Toor Ämnesgranskare: Sverker Holmgren Examinator: Lars-Åke Nordén UPTEC IT 19013 Tryckt av: Reprocentralen ITC Contents 1 Introduction 1 2 Background 1 2.1 Cloud Computing Concepts and Obstacles . 2 2.2 Scientific Computing . 4 2.3 HASTE Project . 4 2.4 Motivation . 5 2.5 Purpose . 6 3 Related Work 6 4 System Implementation 8 4.1 System Overview . 9 4.2 Terraform . 12 4.3 REST Service . 14 4.4 Message Queue . 15 4.5 Data Aware Functionality . 16 4.6 Negotiator Module . 17 4.6.1 Resource Availability . 17 4.6.2 Terraform Configuration Generation . 18 4.6.3 Executing Terraform Scripts . 18 4.7 Tracing . 19 4.8 Infrastructure Implementations . 20 4.8.1 Spark Standalone Cluster . 22 4.8.2 HarmonicIO cluster . 24 iii 4.8.3 Loading Microscopy Images . 26 4.8.4 Single Container Application . 27 4.9 Simple Web User Interface . 28 5 Results 28 5.1 Spark Standalone Cluster . 30 5.2 HarmonicIO Cluster . 30 5.3 Image Loader . 31 5.4 Running a Trivial Container . 32 6 Discussion & Evaluation 33 6.1 Comparison Against Other Methods . 33 6.1.1 SparkNow . 34 6.1.2 KubeSpray . 34 6.1.3 Manual Provisioning . 35 6.2 Future Development Complexity . 35 6.3 Tracing . 36 6.4 Data Aware Function . 37 6.5 Security Issues . 37 6.6 Limitations of This Service . 38 7 Future Work 39 8 Conclusion 39 iv 2 Background 1 Introduction There has been a tremendous growth in data over the past decade. This trend can be observed in almost every field. The Large Hadron Collider experiment at CERN [2] and Square Kilometre Array project [7] are examples of scientific experiments dealing with data beyond the petascale. This requires efficient, scalable and resilient platforms for the management of large datasets. Furthermore, to continue with the analysis, it is required to make these large datasets available to the computational resources. Recently, together with the cloud infrastructures, a new concept has emerged to offer Infrastructure-as-a- Code (IaC). IaC enables run-time orchestration, contextualization and high-availability of resources using programmable interfaces [4]. The concept allows mobility and high- availability of customized computational environments. AWS Cloud Foundry, Open- Stack HOT and Google AppEngine are the platforms aligned with the concept of IaC. However, it is still overwhelming and time-consuming to capitalize on this concept. In order to satisfy researchers and to seamlessly access the customized computational en- vironment for the analysis, it is required to create a level of abstraction that hides the platform-specific details and intelligently place computational environment close to the datasets required for the analysis. This thesis proposes a software service that aims to support the researchers in the Hi- erarchical Analysis of Spatial and Temporal Data (HASTE [3]) project to seamlessly compute applications on different cloud services. The main capabilities of the software are its cloud agnostic ability, tracing of the build process of the compute infrastructure and its ability to be data aware, meaning that it can locate the data resource that is used in the proposed computation. 2 Background Cloud computing is appearing as a new trend in the ICT sector due to the wide ar- ray of services the cloud can provide. Many companies such as Google and Amazon are offering different kind of cloud services, Google App Engine1 and Amazon Web services (AWS)2 respectively. Each service manages their own infrastructure in their own fashion. The cloud providers control large pools of computers and profit from the cloud by renting out user requested resources. The users are billed on a timely, pay per month or on a usage basis where users pay depending on the workload of the rented resources. Other than for commercial use, cloud computing is expanding in scientific 1https://cloud.google.com/appengine/ 2https://aws.amazon.com/ 1 2 Background research using platforms such as OpenStack3 to provide computation. However cloud computing comes with many challenges that are tackled by businesses using the cloud for commercial use along with scientist that are looking for the cloud to run scientific computations. 2.1 Cloud Computing Concepts and Obstacles The term cloud computing has existed since the 1960s however the concept gained pop- ularity in 2006. There has been no clear definition of the term cloud computing. The definition cloud computing is however described by the National Institute of Standards and Technology (NIST) [13] who defines cloud computing as model for enabling con- venient on demand network access to a shared set of configurable computing resources that can be rapidly provisioned and released with minimal effort. Generally speaking, the cloud can be divided into four different architectural layers. Zhang et. al. [24] describes the layers in the following way. The lowest level can be described as the hardware layer, this is where the bare metal resides as routers, switches, power and cooling systems. Next is the infrastructure layer which creates a set of re- sources that are configured on the hardware through virtualization technologies. Above the infrastructure layer is the platform layer where operating systems and application frameworks lie. The final layer is the application layer where software applications are deployed. The business model of the cloud can be categorized in different services that are derived from the architectural layers. NIST defines the services as following. Infrastructure as a Service (IaaS) provides processing, storage and networks. The user should have the ability to deploy and run software on the infrastructure such as operating systems and applications. Examples of IaaS providers are Google and Amazon. Platform as a Ser- vice (PaaS) allows the user to use the cloud infrastructure through provided tools. The user does not control the underlying networks, operating or storage but only the self de- ployed applications. Software as a Service (SaaS), the highest level of user abstraction where the user is only capable of accessing the cloud through the provider’s interface commonly in the form of a thin client interface or a web browser. As mentioned in the introduction, the newly coined concept infrastructure as a Code (IaC) is on the rise. The principle of IaC is to treat the infrastructure as code. Thereafter use the code to provision and configure the infrastructure, more importantly provision- ing virtual machines (VMs) in IaaS. The code represents and gives the desired state 3https://www.openstack.org/ 2 2 Background of the infrastructure without having to walk through manual steps and previous config- urations [16]. This concepts allows the ability to apply similar software engineering techniques related to programming and software development when building one’s in- frastructure. This means that a blueprint or state of the infrastructure can be version controlled, shared and re-used. The end purpose of IaC is to improve the quality of one’s infrastructure [21]. Another important concept is the container concept which is growing in the cloud com- puting field and containers are most often used at the application level to replace vir- tual machines. There are many advantages of using containers. Containers are more lightweight than VMs, start time and resources usage is decreased [14]. Docker4 is one of the most well-known and used tool for containerizing applications. Docker provides services that builds and assembles applications. Each Docker container is based on an system image, a static snapshot of a system configuration. A Docker container is run by the client, when a Docker container is ready to be run, it looks for the Docker image on the machine or on a remote registry to download the Docker image. Once the image is ready, Docker creates a container and allocates a file system with a read and write layer and creates a network to interact with the host machine [19]. The main principles of using Docker containers are to avoid conflicting dependencies, an example could be if two websites need to run two different versions of a framework, then each version can be installed in a separate container. Also, all dependencies are bound to a container which means that the need to re-install dependencies disappear if the application is to be re-deployed. Furthermore, Docker containers are not very platform dependent. The only requirement is that the operating system runs Docker [14]. As a consequence of multiple providers many individuals who are using cloud services are facing the issues of adapting to each and every different cloud provider. One of the main obstacles is the concept of vendor lock-in [11], meaning that the cost of changing vendor is too high to justify the change which leads to the problem of being locked into one vendor because of the cost of changing vendor.