Guide to the Open Cloud Current trends and open source projects November 2016 AUTHORS Libby Clark and Mark Hinkle The Linux Foundation www.linuxfoundation.org Contents Introduction 3 What is the Open Cloud Today, Why It’s Important, and How It’s Advancing 5 Emerging Trends 6 Cloud Native Applications 6 Containers 7 Unikernels 7 Why is the Open Cloud Important? 7 Profile Methodology 9 Profiles 11 Infrastructure as a Service (IaaS) 11 Platform as a Service (PaaS) 12 Virtualization, Containers and Cloud Operating Systems 13 Micro or Minimalist OSes 13 Virtualization 14 Management and Automation 15 Unikernels 17 DevOps CI/CD 19 Complete CI/CD cycle 19 Configuration management 20 Logging and Monitoring 21 Software-Defined Networking (SDN) 22 Networking for containers 23 Software-Defined Storage 24 2 Guide to the Open Cloud: Current trends and open source projects Introduction This third annual Guide to the Open Cloud aims to help companies stay informed on the latest open source cloud technologies and trends. The Linux Foundation’s cloud experts have curated a list of the most useful, influential, and promising open source projects with which IT managers and practitioners can build, manage, and monitor their current and future mission-critical cloud resources. In doing so, we’re also helping companies identify projects they can best leverage for external R&D through strategic open source participation. Contributing knowledge and code to open source projects not only helps companies meet their business objectives, but it creates thriving communities that keep projects strong and relevant over time, advances the technology, and benefits the entire open source cloud ecosystem. Cloud computing is the cornerstone of the digital economy. Companies across industries now use the cloud—private, public or somewhere in between—to deliver their products and services. Forty-one percent of all enterprise workloads are currently running in some type of public or private cloud1. That number is expected to rise to 60 percent by mid-2018. And some 95 percent of companies are at least experimenting in the cloud2. Enterprises are continuing to shift workloads to the cloud as their expertise and experience with the technology increases. From banking and finance to automotive and healthcare, companies are facing the reality that they’re now in the technology business. In this new reality, cloud strategies can make or break an organization’s market success. And successful cloud strategies are built on Linux and open source software. Some of the most successful public companies today are built around cloud-native applications—a fashionable term that simply means they’re designed to run in the cloud. Netflix, Facebook, LinkedIn, Twitter, and Amazon have all leveraged open source components within a distributed, microservices-based architecture to quickly deliver new products and services that are cost-effective and responsive to market demands and changes. By breaking applications up into microservices, or distinct, single-purpose services that are loosely coupled with dependencies and explicitly described through service endpoints, they have significantly increased the overall agility and maintainability of applications and used that to gain competitive advantage. 1 451 Research, “Voice of the Enterprise: Cloud Transformation - Workloads and Key Projects,” Sept. 1, 2016. 2 RightScale, “2016 State of the Cloud Report,” January, 2016. 3 Guide to the Open Cloud: Current trends and open source projects The rest of the market has scrambled to replicate this architecture and approach, cobbling together their own solutions using custom scripts and open source software—often using the open source versions of these web giants’ own infrastructure (ie Google’s borg, which became Kubernetes; Twitter’s Mesos project, VMware’s Cloud Foundry, etc.) This experimentation has set off a chain of innovation with four notable trends, still playing out today: • Increasing consumption of public cloud resources • Adoption of container technologies like Docker and others (Fifty-three percent of organizations are either investigating or using containers in development or in production3.) • The rise of DevOps as the most effective method for application delivery in the cloud • An explosion in available open source tooling as user companies like Walmart and Capital One release their management software under open source licenses. While deployment and management remain a challenge, microservices architecture is now becoming mainstream. In a recent Nginx survey4 of 1,800 IT professionals, 44 percent said they’re using microservices in development or in production. Adoption was highest among small and medium-sized businesses. Not coincidentally, the use of public cloud is also predominant among SMBs5, which are more nimble and faster to respond to market changes than large enterprises with legacy applications and significant on-premise infrastructure investments. Many reports tout hybrid cloud as a fast-growing segment of the cloud. Demand is growing at a compound rate of 27 percent, “far outstripping growth of the overall IT market,” according to researcher MarketsandMarkets6. And IDC predicts that more than 80 percent of enterprise IT organizations will commit to hybrid cloud architectures by 20177. However, this growth is happening predominantly among large enterprises with legacy applications and the budget and staffing to build private clouds. They turn to cloud for storage and scale-out capabilities, but keep most critical workloads on premise. In the mid-market, hybrid cloud adoption stands at less than 10 percent, according to 451 Research. Hybrid cloud is, then, a good transition point for legacy workloads and experimenting with cloud implementation. But it suffers from several challenges with more advanced cloud implementations, including management complexity and cost. “Most organizations are already using a combination of cloud services from different cloud providers. While public cloud usage will continue to increase, the use of private cloud and hosted private cloud services is also expected to increase at least through 2017. The increased use of multiple public cloud providers, plus growth in various types of private cloud services, will create a multi-cloud environment in most enterprises and a need to coordinate cloud usage using hybrid scenarios. 3 Cloud Foundry, ”Hope Vs. Reality: Containers in 2016,” June 2016. 4 Nginx, “The Future of App Development and Delivery,” March 2016. 5 RightScale, “2016 State of the Cloud Report,” January, 2016. 6 Wall Street Journal, “Special Report: CIOs Say Hybrid Cloud Takes Off,” Oct. 2015. 7 IDC, “IDC FutureScape: Worldwide Cloud 2016 Predictions — Mastering the Raw Material of Digital Transformation,” 2016. 4 Guide to the Open Cloud: Current trends and open source projects “Although hybrid cloud scenarios will dominate, there are many challenges that inhibit working hybrid cloud implementations. Organizations that are not planning to use hybrid cloud indicated a number of concerns, including: integration challenges, application incompatibilities, a lack of management tools, a lack of common APIs and a lack of vendor support,” according to Gartner’s 2016 Public Cloud Services worldwide forecast.8 Over the long term, workloads are shifting away from hybrid cloud to a public cloud market dominated by providers like AWS, Azure, and Google Compute. “The share of enterprise workloads moved to the public cloud is expected to triple over the next five years,” from 16 percent to 41.3 percent of workloads running in the public cloud, according to a recent JP Morgan survey of enterprise CIOs9. Among this group, 13 percent said they view AWS as “intrinsic to future growth.” AWS still very much dominates the cloud industry and has continued to grow at an incredible rate. It looks like hosted services are going to be a huge second line of business for the company. Based on its July 2016 earnings release, Amazon’s web business is on pace to be a $9 billion business this year.10 The shift to the public cloud will continue to accelerate as costs come down and security concerns are allayed. Security is still a top concern among companies considering moving workloads to the public cloud, according to Gartner, despite a strong track record of security and increased transparency from cloud providers. Rather, security is still an issue largely due to companies’ inexperience and improper use of cloud services. As companies gain more experience with public cloud use, security concerns will become less of a hindrance. As cloud adoption grows, open source technologies will continue to be the source of innovation and the foundation for new markets and ecosystems. Each of the projects listed in this report is actively involved in creating the future of IT infrastructure on which companies will deliver their products and services, in the coming year and beyond. Let this report be your guide to become familiar with these projects, the categories of technology in which they are influential, and the ways in which they can help companies remain competitive in this age of digital transformation. What is the open cloud today, why it’s important, and how it’s advancing Many companies are still building their own cloud stacks—assembling open source components to run a private or hybrid infrastructure. And you’ll find these components on our list, as they were in the past. But the drive is no longer to build enterprise alternatives to large, proprietary public clouds. Instead, more companies are turning to public cloud to run their applications and services, and as they do so the nature of the open cloud today is changing. 8 Gartner, “Forecast Analysis: Public Cloud Services, Worldwide, 2Q16 Update,” Sept., 2016. 9 Business Insider, “CIOs See Momentous Shift in Public Cloud Usage,” April 2016. 10 Amazon, “Q2 2016 Financial Results,” July 2016. 11 Gartner, “Gartner Says Worldwide Public Cloud Services Market to Grow 17 Percent in 2016,” Sept. 2016. 5 Guide to the Open Cloud: Current trends and open source projects As cloud computing continues to evolve, the infrastructure is becoming more flexible and programmable.