A STUDY OF THE EFFECTIVENESS OF CLOUD INFRASTRUCTURE
CONFIGURATION
A Thesis
Presented to the
Faculty of
California State Polytechnic University, Pomona
In Partial Fulfillment
Of the Requirements for the Degree
Master of Science
In
Computer Science
By
Bonnie Ngu
2020 SIGNATURE PAGE
THESIS: A STUDY OF THE EFFECTIVENESS OF CLOUD INFRASTRUCTURE CONFIGURATION
AUTHOR: Bonnie Ngu
DATE SUBMITTED: Spring 2020
Department of Computer Science
Dr. Gilbert S. Young ______Thesis Committee Chair Computer Science
Yu Sun, Ph.D ______Computer Science
Dominick A. Atanasio ______Professor Computer Science
ii ACKNOWLEDGEMENTS
First and foremost, I would like to thank my parents for blessing me with the opportunity to choose my own path. I would also like to thank Dr. Young for all the encouragement throughout my years at Cal Poly Pomona. It was through his excitement and passion for teaching that I found my passion in computer science. Dr. Sun and
Professor Atanasio for taking the time to understand my thesis and providing valuable input.
Lastly, I would like to thank my other half for always seeing the positive side of things and finding the silver lining. It has been an incredible chapter in my life, and I could not have done it without all the love and moral support from everyone.
iii ABSTRACT
As cloud providers continuously strive to strengthen their cloud solutions, companies, big and small, are lured by this appealing prospect. Cloud providers aim to take away the trouble of the brick and mortar of physical equipment. Utilizing the cloud can help companies increase efficiency and improve cash flow. Many companies have adopted cloud solutions in their operations in the past few years. Large amounts of money are spent developing and installing software to improve their competitive advantage. Cloud computing enables businesses to utilize software on the internet as a service.
The proposed project will examine and explore multiple ways to configuring cloud infrastructure in Amazon Web Services. Infrastructure as code has been an emerging trend. Each with different strengths and weaknesses. The choices and adaptability of code as infrastructure has grown tremendously along with cloud services.
As the popularity of cloud services grow, the demand of infrastructure as code has grown with it.
iv TABLE OF CONTENTS
SIGNATURE PAGE ...... ii
AKNOWLEDGEMENTS ...... iii
ABSTRACT ...... iv
LIST OF TABLES ...... vii
LIST OF FIGURES ...... viii
CHAPTER 1: INTRODUCTION...... 1
CHAPTER 2: RELATED WORK ...... 3
2.1 Model Driven Infrastructure...... 3
2.2 Self-provisioning Network Infrastructure ...... 3
2.3 Quality of Infrastructure Code ...... 4
CHAPTER 3: DEVELOPMENT OPERATIONS...... 5
3.1 Infrastructure as Code (IaC) ...... 6
CHAPTER 4: BATTLE OF THE CLOUDS ...... 8
4.1 General specifications ...... 8
4.2 Pricing ...... 8
4.3 Compute Services...... 9
4.4 Storage Services ...... 9
4.5 Database Services...... 10
4.6 Overall...... 10
v CHAPTER 5: AMAZON WEB SERVICES...... 11
5.1 Configuration Manager Options...... 12
5.2 PowerShell Desired Configuration (DSC) ...... 14
5.3 AWS AppConfig ...... 20
CHAPTER 6: RESEARCH GOAL ...... 25
CHAPTER 7: METHODOLOGY ...... 26
7.1 Initial Configuration ...... 26
7.2 Configuration Update ...... 26
7.3 Tapered Configuration ...... 26
7.4 Timeline ...... 27
7.4.1 Round 1 ...... 27
7.4.2 Round 2 ...... 27
7.4.3 Round 3 ...... 27
7.4.4 Round 4 ...... 27
7.4.5 Round 5 ...... 28
CHAPTER 8: RESEARCH FINDINGS ...... 29
8.1 Challenged Encountered...... 29
8.2 Evaluation of Results and Analysis ...... 29
CHAPTER 9: CONCLUSION AND FUTURE WORKS ...... 43
REFERENCES ...... 45
vi LIST OF TABLES
Table 1 Results of Round 1: Initial Configuration ……………………………… 34
Table 2 Results of Round 2: Configuration update – installation …………….… 35
Table 3 Results of Round 3: Configuration change – upload file ………………. 36
Table 4 Results of Round 4: Configuration change – script used to pull file …... 37
Table 5 Results of Round 5: Configuration tampering …………………………. 38
vii LIST OF FIGURES
Figure 1 Cost of AWS vs Azure Resources ……………………………………..... 9
Figure 2 Common AWS Architecture for Web Applications …………………… 13
Figure 3 DSC Configuration for Web Application servers ……………………… 17
Figure 4 DSC Output Log ……………………………………………………….. 18
Figure 5 Using All Nodes in DSC ……………………………………………….. 19
Figure 6 Implementing All Node Functionality …………………………………. 20
Figure 7 Service Quotas for AppConfig …………………………………………. 24
Figure 8 Results of Test 1 Initial Configuration of DSC vs AppConfig ……….... 30
Figure 9 Average Time for all Test Rounds with DSC vs AppConfig ...... 31
Figure 10 Median Time for all Test Rounds with DSC vs AppConfig …...…...…. 31
Figure 11 Maximum Time for all Test Rounds with DSC vs AppConfig ………... 32
Figure 12 Minimum Time for all Test Rounds with DSC vs AppConfig …………33
Figure 13 Overall Performance Metrics of Round 1 (DSC vs AppConfig) ………. 34
Figure 14 Overall Performance Metrics of Round 2 (DSC vs AppConfig) ………. 35
Figure 15 Overall Performance Metrics of Round 3 (DSC vs AppConfig) ………. 35
Figure 16 Overall Performance Metrics of Round 4 (DSC vs AppConfig) ………. 36
Figure 17 Overall Performance Metrics of Round 5 (DSC vs AppConfig) ………. 36
viii CHAPTER 1
INTRODUCTION
Technology has become a necessity of today’s generation. The demand for the advancement of technology has pushed companies to release more often to stay competitive. Many companies face challenges in scaling because they simply do not have the required resources and time to do it properly. Those issues are the reasons cloud providers were born.
Companies has turned to cloud service providers to maintain their competitive advantage. A cloud service provider is a third-party company offering cloud-based platform, infrastructure, application, and or storage services. A way to visualize the role of a cloud provider would be a homeowner would pay for utilities such as electricity or water and would only pay usage consumption each month. Similarly, companies pay cloud providers for the usage. Cloud providers are based on the pay-per-used model.
Companies can grow without worrying about added infrastructure cost. Businesses can take advantage of the scalability and flexibility without being limited by the physical constraints such as on-premises servers, building multiple datacenters with redundancies to increase reliability. Businesses can focus on building their product instead of spending time figuring out the hardware cost and maintenance.
Another advantage of the cloud is the ability to scale very rapidly. For example, if a small business maintains an average of 5,000 traffic hit per day but once a year, they offer a massive sale on Black Friday. On Black Friday, they attract 100,000 hits. In the traditional model, the company will have to own enough resources to handle 100,000 hits if they do not want to lose customers during the sale. The company will carry with them unused resources though out the year to handle the traffic for one day.
1 Cloud solutions provide a lower upfront, storage, and maintenance cost. However, traditional on-premise servers can be cheaper in the long term. Cloud allows for high speed scaling up or down as the company sees fit but usage cost is a recurring monthly expense. Careful analysis should be considered when choosing between cloud or on- premise. It is also important to take into consideration maintenance and configurability.
Cloud providers such as AWS and Google cloud has made it simple to configure and secure servers making cloud a desirable option. The growing popularity of cloud computing has driven the interest of developers to develop a solution to quick and reliable configuration of cloud resources.
2 CHAPTER 2
RELATED WORK
2.1 Model Driven Infrastructure
A cloud provider offers different types of infrastructure, scripts are highly dependent on the targeted cloud provider. As a result, managing infrastructure as code
(IaC) has become time consuming. The study presents a model driven infrastructure provisioning tool called ARGON. A tool to implement domain specific language for modeling cloud infrastructure and provides transformation engines to automate infrastructure provisioning for different cloud providers. ARGON can model load balancers that distributes incoming traffic across multiple virtual servers hosting the application and automate the IaC process for different cloud providers. By utilizing
ARGON’s infrastructure modeling process, a high-level configuration of cloud infrastructure can be created [19].
2.2 Self-Provisioning Network Infrastructure
Intel manages about 80 datacenters around the world which contains thousands of servers. The complexity of these servers makes it difficult to configure the network safely and securely. Each server required unique and intricate configurations. The system must navigate complex decision trees using fuzzy logic to choose optimal solution while abiding to rules created to ensure configuration safety [5]. The configuration logic utilized a PHP web-based application which acts as a fuzzy logic decision engine.
Automated scripts written in XML using a semi-proprietary schema are used to dynamically detect target device hardware/software platform and runs the correct sequence of commands for the targeted platform.
3 A challenge was to create an automated approach on top of their existing legacy manual process [5]. The problem was solved by leveraging an existing internal library of automated scripts previously used for commercial network configuration engine and using them as building blocks for the provisioning tool. A working implementation of network port configuration was successful able to demonstrate a highly flexible network provisioning system. Utilizing internal automated scripts, they were able to create a consistent, rapid, and predictable system with an easy to use interface. This advancement will reduce to workload on individual contributors and allow them to focus more time on higher value projects.
2.3 Quality of Infrastructure code
The concept of code smells was introduced by Kent Beck and Martin Fowler [1].
The concept described flaws in code that can lead to problems in the future. Code smell does not produce a runtime error but instead indicates an underlying problem that needs improvement. An example of code smell is the Duplication Block (DB) smell. It is a sequence of identical statements that are repeated throughout the application code and should be a method to encompass the repetition.
A recent research was done by Bent et al. to study a model for the tool Puppet.
Puppet is a tool used to define the required installed software products and their configuration as infrastructure as code (IaC) [3]. The principle of code smells was applied to puppet to validate the quality of the infrastructure code. The experiment was able to distinguish between implementation and designs. Code smells on IaC can be applied to different areas of technology and can use common code refactoring techniques to improve code base [3].
4 CHAPTER 3
DEVELOPMENT OPERATIONS
As the demand for faster, entirely continuous, stable, and reliable software grows the emergence of Infrastructure as Code (IaC) approach has become increasingly popular.
As the trend towards cloud providers increase so does IaC. IaC is based upon software development practices and supports code-centric tools using scripts to specify creation, modification, and execution. In order to solve these challenges, a growth of DevOps
(Development and Operations) came. DevOps promotes the continuous collaboration between developers and operations by optimizing the software delivery cycle and time to production. DevOps makes a significant impact to software delivery by using agile methodologies. The idea behind DevOps involves writing code to deploy and update infrastructure. Online companies such as Amazon, Facebook and Google are famous for operating in a “perpetual development” mindset [4]. These high-performance companies rely on their continuous delivery models to be able to deploy updates multiple times a day. Many companies are now adopting this frequent small incremental delivery model.
Continuous delivery gives companies a competitive advantage by treating software development as a strategic capability and combining a set of practices to ensure rapid and safe code deployment [6]. As part of this new and quick code deployment process, deployment pipelines need to be automated and require little to no intervention.
Reducing manual intervention awards developer teams to put their focus on features and implementation rather than publishing features.
5 3.1 Infrastructure as Code
There is a considerable amount of IaC tools available to orchestrate cloud infrastructure provisioning. Currently, there are multiple choices in tools and strategies.
Amazon Web Services offers CloudFormation to define, manage, and execute infrastructure resources while third parties created Terraform and Ansible. Cloud provisioning tools are not all built the same. Different cloud provisioning tools utilize different approaches to managing cloud infrastructure. Most popular approaches include code-centric or model. [7] Model-driven techniques are popular in academia but in real life application, they appear slower. [8]
An advantage of cloud providers is the ability to rapidly scale up or down [9]. In order to be able to increase the number of servers hosting the application, the infrastructure of the application must be able to reliably scale-up and scale-out. In the
DevOps community, a common topic is to treat server as cattle. Essentially, this means each individual piece of infrastructure must be disposable but also easily build up when needed. Each resource should not be a pet to keep around but as cattle and dedicated to work but replaceable.
In traditional server management, servers are view as pets. If there is an issue with the server, the system administrator will work on fixing the problem but in this new cattle mindset, the dead server will be destroyed and replaced. In the definition of pet servers, they are servers that are indispensable and cannot be down because it will affect operations and profit. Typically, they are manually built without much automation.
Hence, these are referred to as pets because their configuration has been hand-crafted and want to be kept alive because recreating will take an enormous amount of effort. On the
6 other hand, cattle servers are an array of servers that are build using automated tools and designed to handle failures. During a failure of one or multiple servers, the system is designed to reboot and run through steps to self-recovery without human intervention
[11]. Failures can happen at any time and it is safe to assume so. Systems should be designed to gracefully handle failures without service interruption. Self-healing systems take the responsibility away from system administrators and developers so they can focus on producing revenue generating task instead of maintenance.
The ability to rapidly scale is heavily reliant on infrastructure as cattle.
Companies want to be able to build consistent, identical, and working resources to host their applications in an instant. Being able to scale out with automation also implies switching cloud providers will be less painful. Theoretically, if AWS was no longer an option or the best option, the company will be able to reuse their automation, possibly with some adjustments, reuse and build out their application in another cloud provider.
There is flexibility in using automation because the infrastructure is designed to be destroyed and rebuilt.
Although, cattle servers are designed to be destroyed, there may be some repercussions that should be made aware. The problem with respawning or rebooting servers before troubleshooting is that the problem often goes away but we missed the opportunity to diagnose the problem and track down the root cause. Automation is great and incredibly important and saves time troubleshooting but if the root cause is unknown, it can keep occurring in other servers and become a larger problem in the future.
7 CHAPTER 4
BATTLE OF THE CLOUDS
Although some companies see benefits in going multi-cloud – using a combination of two or more cloud providers, we are going to focus on comparison of single providers to increase simplicity. It is important to understand the advantages and pricing of each cloud provider before using a mixture. Each cloud providers have similar resources and features as their competitors to maintain competitiveness in the cloud rankings. Various cloud providers have claimed supremacy in the cloud domain. Practical usage of a single provider creates the illusion of high availability and security [10]. AWS and Azure has been the top providers in the market for a while. But the biggest question is which provider suits my need? Let us compare the two biggest cloud giants to get a better understanding of the similarities and differences.
4.1 General Features
AWS entered the market in 2006 followed by Azure four years later in 2010 [12].
While Azure holds 30% of the market share, AWS holds 40%. It is also important to take into consideration the ability to integrate open source tools from the community as well as platform ecosystem. While Azure is still building up their Linux ecosystem and open source integration, they excel in hybrid compatibility.
4.2 Pricing
Both AWS and Azure charges by the pay as you go model [18]. AWS typically charges per hour while Azure is billed based on minutes. Regarding short-term outlook,
Azure will be the cheaper option because of the small incremental charges. However,
8 AWS is the better option when architecture is scaled up. We will do a cost comparison on the general set of resources below to gain a deeper understanding.
4.3 Compute Services
Computational resources are one of the core services in cloud computing. With large amounts of data being generated, there is an increase need for faster processing.
Spawning multiple instances up or down in a matter of minutes is an attractive aspect of cloud computing. In terms of compute, AWS offers EC2, Elastic Beanstalk, Lambda,
ECS. On Azure, we can find Virtual machine, App service, Functions, and Container service.
Instance Type AWS Azure Aws Price Azure Price
General Purpose M5.xlarge B4MS $0.192/hour $0.166/hour
Compute optimized C5.xlarge F4s V2 $0.17/hour $0.169/hour
Memory optimized R5.xlarge E4 v3 $0.252/hour $0.252/hour
GPU instances G3s.xlarge NC 6 $0.75/hour $0.90/hour
Figure 1 Cost of AWS vs Azure Resources
4.4 Storage Services
Both AWS and Azure provide long running and reliable storage services. AWS has AWS
S3, EBS, and Glacier whereas Azure Storage Services have Blob Storage, Disk Storage, and Standard Archive. Both providers have comparable options with AWS offering high availability and automated replications of S3 storage across region if a region server ever failed on their end.
9 4.5 Database Services
Both AWS and Azure provide different services to handle structured and unstructured data to suit the needs of their customers. AWS offers RDS, Redshift, and
DynamoDB while Azure offers SQL Database, SQL Warehouse, and DocumentDB.
Azure largely supports SQL server while AWS RDS supports different database engines including MariaDB, Aurora, MYSQL, PostgreSQL, and Oracle. The database services alone can be a deciding factor in choosing cloud providers if limitation to SQL is a deal breaker.
4.6 Overall
As seen in the comparisons, both AWS and Azure can provide similar products with slight variance in pricing using larger computational instances. It is difficult to pick a clear winner; however, Azure is tightly integrated with Microsoft products such as
Windows server, Office, and SQL. Microsoft has done a great job in providing their strongest and well-loved services in their cloud implementation. On the other hand,
Amazon’s biggest strength is their dominance in the cloud market and being able to offer an expansive list of services.
10 CHAPTER 5
AMAZON WEB SERVICES
In 2000, when days were more simply, Amazon was an e-commerce company.
They were faced with struggles to scale as fast as the market demanded. These issues pushed the company to build solid internal systems to capture their growth. Like most startups, their code and architectural design was not setup to handle future expansions.
Instead of an organized development platform, they had created a jumble mess. This began Amazon’s decoupling adventure. Amazon began untangling their product and creating well-documented APIs that set the standard for future internal products.
As the teams worked, they noticed a large amount of time is spend building infrastructure for their applications such as database, compute, and storage units. They were not building their applications faster than before. The problem was each team was building their own resources for an individual project with no preparation to scale or reuse. Each project would start with a new set of infrastructures. To solve this problem, a common set of infrastructure services was produced. All teams leveraged this common set for their product and reduced the build out time. They were able to dedicate more energy to application development.
While building out their application, Amazon found they had the opportunity to offer a broad selection of their products. The team at Amazon has become good at running compute, storage, and database resources at a reliable, scalable, and cost effectiveness. At this point, they began to articulate the vision of AWS. The idea of providing infrastructure resources to other developers facing similar issues could be a market opportunity.
11 In the summer of 2003, a set of services required to operate a system in internet hosting was gathered. The selection intended to include a selection of necessary and expansive components. Amazon wanted to develop a service to help companies build out their application like they have done for themselves.
A few years later, in 2006, the first Infrastructure as a Service (IaaS) was launched. Back then, it was a new concept to behold, but today, it has become a lucrative market. It took a few years for competitors to launch their own IaaS which gave AWS time to attract a vast amount of the market share. While competitors were still building out a cloud service, AWS had the advantage to tune and refine their cloud resources to be reliable and stable. Today, AWS still holds the largest market share of cloud providers.
An impressive list of companies has chosen AWS as their cloud provider such as Netflix,
Twitch, LinkedIn, Facebook, ESPN, and Twitter. The top EC2 spenders on AWS includes Netflix at $19 million/month and Twitch at $15 million/month.
Amazon may have been in the market longest and has the top market percentage, however, building large scale computing infrastructures over distributed platforms presents several challenges [13]. There are several factors that play a role in the efficiency and accuracy such as network latency, fault tolerance, and application compatibility. However, there are multiple choices in tools to leverage and the best tool is dependent on the application’s nature.
5.1 Configuration Manager Options
While we have explored the importance of scaling up and building out architectural cloud resources it is important to understand how this works and the configuration needed behind the scenes. Cloud offers the ability to rapidly increase
12 virtual machines running your application to be able to handle traffic and usage loads.
But a big question is how is this possible and what is the magic behind the ability to serve large amounts of application usage without downtime and decreased user experience.
The most common architectural design utilizing a virtual machine is an API
Gateway provisioning the traffic then directing traffic to a DNS record in Route53 which propagates the traffic route based on the set up. This set up is consistently used to govern the flow of the application.
Amazon EC2
Amazon Amazon Amazon Classic Load Cloud Front Route 53 API Balancer Gateway
Amazon AWScloud EC2
Figure 2 Common AWS Architecture for Web Applications
AWS offers Elastic Beanstalk as a solution to rapidly grow the application load handle threshold by increasing the number of instances in each environment [14]. Elastic
Beanstalk helps developers deploy quickly and manage applications. It is essentially multiple AWS services coupled together to perform ideally. Each service in Elastic
Beanstalk is charged their standard rates and there are no extra charges for utilizing
Elastic Beanstalk as a combined service. AWS Elastic Beanstalk provides capacity provisioning, load balancing, auto-scaling, and health monitoring. Basically, Elastic
Beanstalk provides most important needs of an application. The autoscaling feature will
13 take care of scaling up or down base of capacity limitations implemented, and load will automatically be divided amongst available and healthy servers.
Elastic Beanstalk seems sounds like the tool to gracefully handle a large portion of the challenge’s companies face. However, are the new instances spun up with the
Elastic Beanstalk autoscaling consistently configured identically is something we should investigate. The idea behind Elastic Beanstalk is to replicate saved configuration settings so instances do not need to be reconfigured when scaling up.
There are numerous automation tools used to configure instances without manual intervention. I have done a comparison between the effectiveness, speed, reliability, and cost of two different methods to configure windows web servers. The study was done on
Windows 2012 platform in the EC2 service provided in AWS. First, let’s examine two different methods to configure an EC2 virtual instance.
5.2 PowerShell Desired State Configuration
Windows DSC is a management tool in Power Shell. It allows for management of
Windows based infrastructure with configuration as code. It is commonly used for configuration, deployment, and management of systems. DSC is free tool offered by
Windows to solve the needs of infrastructure as code (IaC) in window systems. DSC uses
PowerShell to create Managed Object Format (MOF) files which Windows Management
Instrument (WMI) can use to configure the windows-based machine. This ecosystem of windows tool works together to ensure machines stay consistent and always in the desired state.
In order to maintain consistency in infrastructure, DSC can be leveraged. A common technique in systems management is to use base images. The base server image
14 includes all the necessary configurations. The server image is used to spin up new instances. While this method is automated and decreases time spent on server configuration a problem arises when there is a change needed in the configuration such as a new certificate update. We could update the image with the new certificate and rebuild the existing servers with the new image but recreating servers and testing the new update can take a significant amount of time. We could write a script to apply those changes, but in scale, rebuilding hundreds of servers will take time and coordination. However, with
DSC, the configuration is updated in the master script then sent out to respective servers.
We can use DSC to define the desired state to all existing and new servers. Each server will reach out in periodic intervals to check if the desired state has been changed.
PowerShell DSC takes the components configured with PowerShell and converts them into MOF files or WMI. The converted files are used to configure the server. There are two methods (Push and Pull) used to apply the configuration.
Push Method: Use the command Start-DscConfiguration to push the desired state configuration to the server. This cmdlet will only be available when the server is already online.
Pull Method: Reach out to the server to pull the desired state configuration and apply it.
This command requires a pull server that hosts configuration of several servers to be running. The servers need to be configured to poll the pull server and determine if there is a new MOF file available.
Firstly, the environment needs to be Windows Server 2012 or later with
PowerShell 4.0 or newer. A DSC configuration is a special PowerShell function that is defined to configure single or multiple target computers which are referred to as nodes
15 hence, needs a version of PowerShell capable to translating DSC scripts. Also, the server needs to be configured to receive PowerShell remote commands.
Example:
A configuration of a server hosting a website needs to include basic components such as enabling IIS, Web-Server, Web-Mgmt-Tools, and Web-Default-Doc. These settings can be configured through DSC using PowerShell function block. A list of available Windows features is listed under Get-WindowsFeature.
Simple configuration script
Each windows component in the contained in a Windows Server block within the script. Each configured instance needs a unique name because the name is used as a key to convert into the MOF file. Inside each component, there is an Ensure property with value Present when the component should be included in the configuration. An output path is provided after the configuration blocks, so DSC knows where to store the generated MOF file.
16 Figure 3 DSC Configuration for Web Application servers
Once the PowerShell DSC scripts is completed, run as a PowerShell script to generate the MOF file. Then, run the MOF file to apply the configuration.
PS C: \U$ers \Bonni e > Start- DscConf i gu 0 ation -Wai t -\erbose -~ath · I
The configuration will run then applied. It will display the time it took to complete the configuration.
17 Figure 4 DSC Output Log
A name is given to the configuration function block and the node block specified the target node (localhost) to be configured. The function calls on four resources - Web-
Server, Web-Mgmt-Tools, and Web-Default-Doc. We can make the DSC script more dynamic by separating the configuration data from the configuration. A DSC configuration file is a set of Hash tables that contain more Hash tables or Arrays. We can separate the server from the configuration data by using the AllNodes functionality block to capture configurations.
18 Figure 5 Using All Nodes in DSC
Then refined the DSC script to be more generic. The MyWebServerConfiguration call needs an additional .psd1 parameter to specify the file containing the configuration data. To verify the status of the configuration, Get-DscConfiguration will report the current configurations of the server.
19 MyWebServerC onfigu rat ion
Node $AllNodes . NodeName
( $ Feature $Node. Windows Features)
l.Jindrn,,sFeature $Feature. Name
Name = $Feature. Name Ensure = $Feature. Ensure
MyWebServer(onfigurat ion -Out putPat h -( onfigu rat ionData
Start-DscConfiguration -Wait -Verbose -Path
Figure 6 Implementing All Node Functionality
DSC can detect configuration drift in a server. The Local Configuration Manager
(LCM) can be set up to automatically correct configuration when a server is not in the desirable state. The LCM will detect configuration drift on resources within the DSC script. For example, if the Web-Default-Doc is removed, DSC will see this as a configuration drift. However, if someone were to install an additional resource, this would not be considered a configuration drift.
5.3 AWS App Config
The AWS team saw challenges faced by developers in server configuration and came up with a custom AWS service to fully configure infrastructure in AWS. This is an intriguing tool and deserved to be analyzed because the comparison between an AWS native and a third-party tool could provide insight on the reliability of the AWS ecosystem. Internal compliance with internal policies can be intimidating when you have hundreds of servers build over years. With frequent business requirements, code, and
20 infrastructure changes servers can sync out of compliance. App Config is a tool offered to codify policies and automatically triggered.
Safe Deployment of Application Configuration Settings with AWS AppConfig
AWS App Config is a configuration service AWS launched to feed the need of faster configuration than traditional code deployments with precision and confidence.
The service is used internally across their AWS, amazon.com, kindle, and Alexa teams.
The configuration deployment system can perform changes dynamically in seconds. The configuration system is separates from code which means the applications and services that use the application does not need to be restarted. The changes take effect immediately without impacting application performance.
On Nov 2019, AWS launched their internal tool, App Config. It is currently available to customers in all AWS regions. This new service is part of the AWS Systems
Manager and will provide users with the capability to quickly release application configuration changes without dependency on code across applications hosted on
Amazon Elastic Compute Cloud (EC2) instances, containers, and serverless functions.
Amazon currently hold the top market in terms of cloud providers making this new tool a game changer.
AppConfig can be used to configure applications through the terminal or through scripted files for repeated use on multiple applications. AWS provides a stable CLI for all their services as an option to automate infrastructure deployments. Scripted infrastructure speeds up deployment and reduces human intervention and error. In addition, AppConfig
21 also includes an automated safety control to monitor configuration. A periodic poll is utilized to ensure configurations are immediately available.
Getting Started with AWS AppConfig
Step 1:
Create an application inside the AppConfig.
Step 2:
Define environments. An environment is used to segregate deployment groups such as non-production, beta, and production. It could also be used to represent subcomponents of an application for example, web, mobile, or backend components that need different configuration rules. A possible way to automate rollbacks due to a bad configuration is to set up CloudWatch alarms in each environment and trigger a rollback watch event is fired.
Step 3:
Define configuration profiles. This is the source of the configuration data to be deployed. Configuration profiles can and should be setup with validators to ensure the configuration data is correct prior deployment. Validators are added to verify the new configuration will not cause an application failure. One or more validators can be included in the form of AWS Lambda or JSON Schema. For example, a validator can ensure critical and fragile properties are not tampered in the new configuration.
Syntactic validation: you can use a JSON schema. AppConfig validates your configuration by using the JSON schema to ensure that configuration changes adhere to the application requirements.
22 Semantic validation: you can call an AWS Lambda function that runs your configuration before you deploy it.
Step 4:
Finally, set deployment strategies within the application. Deployment strategies is a powerful feature in AppConfig that allows percentage wise deployment to the total instance to be configured. A defined 'bake' time provides control over the period in which the full deployment is to be completed. If something goes wrong during the deployment and bake time, a roll back will be triggered ensuring the stability of the application.
This new capability of AWS Systems manager allows developers to quickly create, manage, and deploy application configurations. Configuration changes are simplified by residing in a centralized location. AppConfig is well integrated with other
AWS services such as parameter store, Systems Manager (SSM) documents, and S3.
Centralizing configuration reduces errors when updates are needed by configurable rules to ensure configuration is valid - similar to integrated unit tests for code. By using a deployment strategy and targeting specific servers, application is not interrupted, and risk is minimized. For instance, if the new configuration is incompatible and is set to deploy to 50% of servers, the other 50% will be running previous compatible configuration can still serve traffic and keep the application active.
There are 3 steps to implement AppConfig:
1. Configure AppConfig to work with the application
2. Enable periodic polling for changes and configuration
3. Deploy a new configuration
23 AppConfig can seem limitless but there are default service quotas to limit misuse. The service quotas are generally enough for moderate usage but can be increased for enterprise usage levels.
Restrictions Resource Onota AnnConfi2 annlications 100 Denlovment stratea;es 20 Environments oer AnnC'.nnfi2 =lication 20 Configuration profiles per AppConfig 100 aoolication Figure 7 Service Quotas for AppConfig
24 CHAPTER 6
RESEARCH GOAL
Following the introductory tests conducted with the tools, an in-depth analysis comparison has been conducted to test the reliability and speed of DSC and AppConfig.
In order to determine the most ideal configuration management between an AWS native service and a Windows native service in the situation of configuring Windows Server
2012 virtual EC2 machines. DSC and AppConfig will be configured to handle identical configurations and number of instances. The hypothesis is that AppConfig will be easier and more reliable because of the native nature of the AWS resources coupled together. It will have an advantage over DSC because Amazon designed the tools to work seamlessly in their cloud network.
25 CHAPTER 7
METHODOLOGY
This study examined the effectiveness and speed of DSC and AppConfig tools to configure infrastructure. In the Spring of 2020, multiple tests based on the time it took to reach and configure the server, validity of the configuration, and the time it took to push out new changes. Log timers were implemented to record the time of deployment and configuration. There was a total of 40 EC2 virtual machines created in AWS for this experiment. The experiment was conducted in US East (North Virginia) Region in the same availability zone.
7.1 Initial Configuration
This analysis was completed by first instantiating 40 t2.medium EC2 instances on
AWS. Right after the initial instantiation, configurations scripts were executed with started log timers to record the completion time. The set of servers were segregated in half. There were 20 servers dedicated to DSC configuration, and 20 other servers dedicated to AppConfig.
7.2 Configuration Update
After the initial configuration is completed, the configuration was changed in order to simulate a real scenario. Companies have on going evolving business requirements and it is important to be able to capture the performance of each change.
7.3 Tampered Configuration
Small and Enterprise companies typically have multiple servers hosting their application to create redundancy and reliability. However, it is difficult to ensure the servers are always properly configured because a team member can unknowingly make a
26 change, or the server could have an underlying problem which misconfigures itself. For example, an inappropriately scheduled task can be removing unused software to free disk space but remove necessary components.
7.4 Timeline
7.4.1 Round 1
Create the 40 ec2 servers and log the time taken to finish initial configuration of both DSC and AppConfig.
7.1.2 Round 2
Change configuration by including extra software installation. In this round, telnet was installed though the configurations. Telnet is a type of client-server protocol that supports command line interface. A common tool used to track down server communication issues.
7.1.3 Round 3
A small zip file (less than 50kb) uploaded to the C: drive of the EC2 virtual servers. This is a good test because it is common to deploy files to the servers either in the form of scripts, new web certificates, or executables to be ran.
7.1.4 Round 4
A script is sent to the server. The script will pull down a small zip file (less than
50kb) into a folder in the C: Drive. This experiment will gauge the time taken to execute a custom script. This is especially useful when the tools provided by the configuration service does not fit the needs of the configuration requirements.
27 7.1.5 Round 5
The configuration was tampered with by removing the anti-virus installation with was set to required. A script to remove the tool was ingested by all the EC2s in the experiment pool. Then a script was used to run through all servers and poll for the installed services and check to see if the anti-virus was again installed.
28 CHAPTER 8
RESEARCH FINDINGS
The five experiments have been conducted in Spring 2020 and the results have been recorded and statistical data have been pulled. The results are presented in the following tables to represent the timed scores in minutes. The instance number represents their unique identifier in the experiment. The table has been structured with two (2) instances per row to compare the results of the DSC server versus AppConfig server.
8.1 Challenges Encountered
Over the course of the experiment, there were a few difficulties encountered.
Firstly, AppConfig is a relatively new tool and did not have strong online thread discussions about best practices and troubleshooting complications. Although,
AppConfig was straight forward and going through the console interface set up was simple. AWS presented AppConfig in a simple and intuitive interface. However, the absence of a supporting online community is a downside but will probably not be a problem in a few more months when the tool gains more traction.
On the flipside, DSC had a strong online community with extensive discussion regarding different types of implementation. However, DSC required more set up in order to ensure compatibility with each AWS EC2. In addition, DSC felt overwhelming to set up without comprehensive knowledge.
8.2 Evaluation of Results and Analysis
Table 1-5 shows the raw test results of all five situations presented. Each table displays the time taken by each server in minutes (rounded to nearest two decimal points), the average, median, maximum, and minimum. In Table 1, while testing initial
29 configuration of the EC2 servers, AppConfig gained better results than DSC. The results posed an interesting observation with outliers in the DSC servers. The minimum was recorded at 3.73 minutes while the maximum was 18.17 minutes which bought up the average time taken to completion as seen in Figure 1. After investigating further into the anomaly, it appeared DSC did not consistently finish the script without timing out and full configuration was completed in the post runs to check for configuration updates. In addition, it appeared DSC did not always trigger initial configuration as soon as the server is up and that added completion time.
Initial Configuration 20 18 16 14 12
10 DSC 8 AppConfig 6 Time Time taken (Seconds) 4 2 0 Instance Number
Figure 8 Results of Test 1 Initial Configuration of DSC vs AppConfig
In Figure 2, we can see that AppConfig was able to perform better than DSC in the testing scenarios except for Round 4 which DSC managed to perform slightly better than AppConfig. However, it is important to point our Round 5 in Figure 2 which indicates DSC took 3.85 minutes more than AppConfig to correct the server
30 configuration from a critical misconfiguration.
Average Time Taken 16
14
12
10
8 DSC 6 •
4 Time Time Taken (Seconds) 2
0 1 2 3 4 5 Round Number
Figure 9 Average Time for all Test Rounds with DSC vs AppConfig
Median Time Taken 16
14
s) d s) 12 ncoe 10 (Sn (Sn 8
keTa DSC 6 • e mTi e 4
2
0 1 2 3 4 5 Round Number
Figure 10 Median Time for all Test Rounds with DSC vs AppConfig
31 In Figure 2, we can see the scores without the outlier biased. The median scoring time for AppConfig was on average 2.41 minutes better than DSC. AppConfig scored a better median in Round 1, 2, 3, and 5 but scored 22 seconds behind DSC in Round 4.
Maximum Time Taken 25
20
15
■ DSC 10 ■ AppConfig
Time Time Taken (Seconds) 5
0 1 2 3 4 5 Round Number
Figure 11 Maximum Time for all Test Rounds with DSC vs AppConfig
As seen in Figure 4, DSC’s maximum time recorded is consistently higher than
AppConfig in all five rounds of testing. This proves DSC will likely take longer to configure servers in the worst-case scenario. In the best-case scenario, we see similar results. DSC at best, took longer to finish server configurations. In Figure 5, DSC took at least an average of 5.69 minutes to configure while AppConfig averaged to 4.59 minutes.
Hence, in the best-case scenario, AppConfig is likely to perform faster than DSC.
32 Minimum Time Taken 10 9 8 7 6 5 DSC 4 • AppConfig 3 •
Time Time Taken (Seconds) 2 1 0 I 1 2 3 4 5 Round Number
Figure 12 Minimum Time for all Test Rounds with DSC vs AppConfig
We can compare the overall metrics of each round individually to gain a better understanding of the performance of each tool. In round 1, the virtual servers were initiated then DSC and AppConfig were used to configure the new server. While we look at the average, median, minimum, and maximum results of both tools, AppConfig was able to preform better than DSC in all measurements. Hence, in the initial configuration situation, AppConfig is a clear performance winner.
33 Performance of Round 1 20 18 16 14 12 10
Seconds Seconds 8 6 4 2 0 Average Median Max Min
DSC AppConfig
Figure 13 Overall Performance Metrics of Round 1 (DSC vs AppConfig)
In round 2, AppConfig performed slightly better than DSC in all metrics apart from the minimum. DSC was able to get 1 out of the 20 servers to outperform the 20
AppConfig servers. Although this is recommendable, the minimum metric should be taken into consideration lightly. It is reasonable to evaluate the maximum time a server could take to be fully configured or the average time a user can expect the server to be configured. The maximum and average gives a good indication on the general performance of a tool.
34 Performance of Round 2 20 18 16 14 12 10
Seconds Seconds 8 6 4 2 0 Average Median Max Min
DSC AppConfig
Figure 14 Overall Performance Metrics of Round 2 (DSC vs AppConfig)
Performance of Round 3 16
14
12
10
8
Seconds Seconds 6
4
2
0 Average Median Max Min
DSC AppConfig
Figure 15 Overall Performance Metrics of Round 3 (DSC vs AppConfig)
35 Performance of Round 4 16
14
12
10
8
Seconds Seconds 6
4
2
0 Average Median Max Min
DSC AppConfig
Figure 16 Overall Performance Metrics of Round 4 (DSC vs AppConfig)
Performance of Round 5 25
20
15
Seconds Seconds 10
5
0 Average Median Max Min
DSC AppConfig
Figure 17 Overall Performance Metrics of Round 5 (DSC vs AppConfig)
36 In figure 15 and 18, AppConfig performed better than DSC in all metrics.
However, in figure 14 and 16, DSC managed to perform better in at least one of the metrics by a small margin. In comparison, AppConfig was able to perform by a larger margin in most rounds and metrics. Depending on the testing case, the AWS native tool,
AppConfig, was able to outperform DSC in most measurement metrics. In round 1, DSC took an average of 41% longer than AppConfig. With the best AppConfig performance in round 3 which performed 133% better than DSC. This large disparity between results can be due to the native nature of AppConfig. AppConfig utilizes an AWS system manager installed on the server itself to apply configurations. This could be giving AppConfig an advantage over DSC which relies on third-party servers to notify there has been a configuration change and the server needs to be updated.
37 Table 1 Results of Round 1: Initial Configuration
Instance # DSC Time Taken AppConfig Time Taken 1/21 5.31 2.46 2/22 4.31 3.74 3/23 6.34 4.47 4/24 7.3 6.47 5/25 5.54 5.42 6/26 5.63 2.36 727 4.98 2.63 8/28 6.42 2.74 9/29 5.63 4.55 10/30 3.73 3.93 11/31 4.93 5.28 12/32 4.25 3.75 13/33 15.52 7.31 14/34 7.42 9.23 15/35 5.24 3.11 16/36 18.17 9.54 17/37 6.18 3.47 18/38 4.71 4.93 19/29 5.28 3.28 20/40 4.91 4.51 Average 6.59 4.66 Median 5.43 4.20 Max 18.17 9.54 Min 3.73 2.36
38 Table 2 Results of Round 2: Configuration update - installation
Instance # DSC Time Taken AppConfig Time Taken 1/21 10.73 10.31 2/22 10,63 9.28 3/23 10.52 6.32 4/24 11.34 7.49 5/25 10.72 8.29 6/26 11.53 6.29 727 5.32 10.32 8/28 6.88 15.20 9/29 9.97 8.35 10/30 7.43 9.38 11/31 9.82 6.93 12/32 6.32 7.23 13/33 9.83 7.90 14/34 12.32 10.81 15/35 11.19 10.29 16/36 13.67 7.29 17/37 13.97 9.01 18/38 11.25 12.94 19/29 18.53 8.24 20/40 12.83 16.93 Average 10.75 9.44 Median 10.73 8.68 Max 18.53 16.93 Min 5.32 6.29
39 Table 3 Results of Round 3: Configuration change – upload file
Instance # DSC Time Taken AppConfig Time Taken 1/21 5.32 2.35 2/22 6.36 5.39 3/23 7.25 3.47 4/24 4.92 6.12 5/25 9.55 3.97 6/26 6.90 4.58 727 8.93 6.31 8/28 10.11 2.63 9/29 4.63 1.47 10/30 9.37 3.87 11/31 8.91 5.31 12/32 10.84 2.45 13/33 7.96 2.56 14/34 12.88 2.85 15/35 9.10 3.62 16/36 8.44 4.16 17/37 13.94 2.68 18/38 10.07 5.21 19/29 8.10 3.74 20/40 10.45 1.76 Average 8.70 3.73 Median 8.92 3.68 Max 13.94 6.31 Min 4.63 1.47
40 Table 4 Results of Round 4: Configuration change – script used to pull file
Instance # DSC Time Taken AppConfig Time Taken 1/21 6.32 7.39 2/22 6.49 9.02 3/23 5.29 8.38 4/24 7.39 9.26 5/25 6.92 9.06 6/26 9.19 8.19 727 10.83 10.41 8/28 13.74 9.32 9/29 9.04 8.19 10/30 7.21 10.38 11/31 10.24 5.29 12/32 6.39 6.29 13/33 9.03 9.28 14/34 8.77 11.38 15/35 10.38 10.46 16/36 11.19 9.48 17/37 12.40 11.09 18/38 7.42 8.48 19/29 9.20 10.57 20/40 8.73 9.48 Average 8.81 9.07 Median 8.90 9.27 Max 13.74 11.28 Min 5.29 5.29
41 Table 5 Results of Round 5: Configuration tampering
Instance # DSC Time Taken AppConfig Time Taken 1/21 15.33 9.38 2/22 16.35 8.94 3/23 17.43 10.38 4/24 14.30 11.83 5/25 11.48 7.91 6/26 10.39 9.63 727 19.24 8.91 8/28 10.75 8.60 9/29 12.63 10.31 10/30 10.38 12.38 11/31 12.49 9.41 12/32 15.09 9.91 13/33 16.00 8.19 14/34 14.94 9.73 15/35 15.48 7.55 16/36 17.92 10.32 17/37 9.48 10.53 18/38 9.92 9.47 19/29 10.56 8.84 20/40 9.67 10.61 Average 13.49 9.64 Median 13.47 9.55 Max 19.24 12.38 Min 9.48 7.55
42 CHAPTER 9
CONCLUSION AND FUTURE WORKS
The results on Figure 2-5 indicates AppConfig was able to complete server configurations in less time than DSC. While we filter and disregard outliers in the solution set, DSC still was not able to perform better than AppConfig. Because
AppConfig is a native AWS tool, it had the advantage of being able to reach servers quicker and more reliably. It was proven, in the scenarios covered by the experiment,
AppConfig was the winner. The process to create configurations in AppConfig were feasibly simple to an individual without previous exposure.
There were a couple of factors that could have influenced the results of this experiment. AppConfig is a tool specifically designed to configure infrastructure such as
EC2s in the AWS environment so ideally, this provided a competitive advantage over the third-party tool – DSC. Although AppConfig has reaped better scores in AWS server configuration, it is limited to configuring compatible AWS resources. On the other hand,
DSC is a tool that can be migrated to different cloud providers and on-premise servers.
Going forward, I will continue to explore wide-spread generic applicability of cloud configuration tools. To improve the performance of either configuration tools, we could decrease the pull time and check for updates often. Configuration management tools are very powerful and can be included in unlimited architectural designs. There is not a best way to implement but there is a good way to implement based on the application design. As we saw in the results, there are positives and negatives of each tool.
43 Future work can be done on reducing the polling times of both AppConfig and
DSC as we know the frequency in the polling phase can affect the length of time taken for a server to realize configuration is outdated. However, we must take into consideration the resource usage to implement frequent polling. A good balance of resource usage as well as configuration efficiency should be achieved to produce optimal results.
44 REFERENCES
[1] M. Fowler, K. Beck, J. Brant, W. Opdyke, D. Roberts, Refactoring: Improving the
Design of Existing Code., Boston, MA, USA:Addison-Wesley Longman
Publishing Co., Inc., 1999.
[2] E. Van der Bent, J. Hage, J. Visser, G. Gousios, "How good is your puppet? an
empirically defined and validated quality model for puppet", 2018 IEEE 25th
International Conference on Software Analysis Evolution and Reengineering
(SANER), pp. 164-174, March 2018.
[3] J. Schwarz, A. Steffens and H. Lichter, "Code Smells in Infrastructure as
Code," 2018 11th International Conference on the Quality of Information and
Communications Technology (QUATIC), Coimbra, 2018, pp. 220-228.
[4] D. Feitelson, E. Frachtenberg, K. Beck, "Development and Deployment at
Facebook", IEEE Internet Computing, vol. 17, no. 4, pp. 8-17, July 2013.
[5] P. Apel, G. Botts and A. Fullam, "Self-provisioning of network infrastructure for
server landings in complex data center environments," 2013 IFIP/IEEE
International Symposium on Integrated Network Management (IM 2013), Ghent,
2013, pp. 912-915.
[6] J. Gmeiner, R. Ramler and J. Haslinger, "Automated testing in the continuous
delivery pipeline: A case study of an online company," 2015 IEEE Eighth
International Conference on Software Testing, Verification and Validation
Workshops (ICSTW), Graz, 2015, pp. 1-6.
45 [7] Sultan F. and Chan, L., 2000, “The adoption of new technology: the case of
object-oriented computing in software companies”, IEEE Trans. Engg. Mgmt. 47,
1 pp
[8] M. Torchiano, F. Tomassetti, F. Ricca, A. Tiso, G. Reggio, "Relevance benefits
and problems of software modelling and model driven techniques—A survey in
the Italian industry", J. Syst. Softw., vol. 86, no. 8, pp. 2110-2126, Aug. 2013.
[9] J. Weinman, "Hybrid Cloud Economics," in IEEE Cloud Computing, vol. 3, no. 1,
pp. 18-22, Jan.-Feb. 2016.
[10] Y. Demchenko, C. Ngo, C. de Laat, T. W. Wlodarczyk, C. Rong and W. Ziegler,
"Security Infrastructure for On-demand Provisioned Cloud Infrastructure
Services," 2011 IEEE Third International Conference on Cloud Computing
Technology and Science, Athens, 2011, pp. 255-263.
[11] P. Bellini, D. Cenni and P. Nesi, "A Knowledge Base Driven Solution for Smart
Cloud Management," 2015 IEEE 8th International Conference on Cloud
Computing, New York, NY, 2015, pp. 1069-1072.
[12] Patrizio, A., & Harvey, C. (2020, March 17). AWS vs. Azure vs. Google: Cloud
Comparison. Retrieved March 25, 2020, from
https://www.datamation.com/cloud-computing/aws-vs-azure-vs-google-cloud-
comparison.html
[13] P. Riteau, "Building Dynamic Computing Infrastructures over Distributed
Clouds," 2011 First International Symposium on Network Cloud Computing and
Applications, Toulouse, 2011, pp. 127-130.
46 [14] R. Miller. “How AWS Came to Be.” TechCrunch, TechCrunch, 2 July 2016,
techcrunch.com/2016/07/02/andy-jassys-brief-history-of-the-genesis-of-aws/.
[15] M. K. M. Murthy, Y. Patel, H. A. Sanjay and M. N. Ameen, "Prediction Based
Dynamic Configuration of Virtual Machines in Cloud Environment," 2012
International Symposium on Cloud and Services Computing, Mangalore, 2012,
pp. 124-128.
[16] A. Rahman, "Characteristics of Defective Infrastructure as Code Scripts in
DevOps," 2018 IEEE/ACM 40th International Conference on Software
Engineering: Companion (ICSE-Companion), Gothenburg, 2018, pp. 476-479.
[17] B. Karakostas, "Towards Autonomic Cloud Configuration and Deployment
Environments," 2014 International Conference on Cloud and Autonomic
Computing, London, 2014, pp. 93-96.
[18] F. Khodadadi, R. N. Calheiros and R. Buyya, "A data-centric framework for
development and deployment of Internet of Things applications in clouds," 2015
IEEE Tenth International Conference on Intelligent Sensors, Sensor Networks and
Information Processing (ISSNIP), Singapore, 2015, pp. 1-6.
[19] J. Sandobalin, E. Insfran and S. Abrahão, "ARGON: A Model-Driven Infrastructure
Provisioning Tool," 2019 ACM/IEEE 22nd International Conference on Model
Driven Engineering Languages and Systems Companion (MODELS-C), Munich,
Germany, 2019, pp. 738-742.
47