IDC TECHNOLOGY SPOTLIGHT Sponsored by: IBM
Demand for qualified data scientists and advanced analysts is outpacing supply, and the evolving business climate is making insight-driven decisions even more important. Automation and artificial intelligence (AI) will be key to helping organizations alleviate these problems and empower them along their digital transformation journey.
The Business Value of Cloud Services for Data and Analytics Workloads December 2020 Written by: Chandana Gopal, Research Director, Business Analytics Solutions
Introduction AT A GLANCE The pandemic will be remembered globally in history as a turning point. Everything will either be pre-pandemic or post-pandemic as humans have KEY STATS had to change the way they live, behave, and work. The pandemic has » In 2015, big data and analytics (BDA) served as a wake-up call to organizations that realized that digital software revenue in the public cloud transformation is not optional. For the foreseeable future, enterprises domain was 12.5% of the total market. have to rethink the way they do business, become more resilient, leverage » BDA revenue is estimated to be 35% in tools and technologies to fill gaps, and learn a new set of digital capabilities 2020 and forecast to be 57% by 2024. that allow them to capitalize on opportunities and face challenges posed » 67% of executives stated they currently by this unprecedented global event. lack staff with data analysis skills. » 55% of IDC survey respondents said that As stated in IDC's 2020 CIO Sentiment Survey, successful recovery from and business expectations are too high to be transition to the next normal post-pandemic require organizations to met with their current circumstances. understand the changing business environment and to adapt strategies that enable the business through the adoption of essential capabilities. More than ever, enterprises must overcome their weaknesses, identify opportunities, and successfully transform themselves to remain competitive in the post-pandemic era. The 151 executives surveyed cited spending on analytics as the highest priority for investment to give enterprises the ability to be resilient in the face of the pandemic and move quickly into innovation as they look beyond into the next normal. The need for digital transformation has to be balanced with the reality that most enterprises are struggling to keep up with demand for analytics resources. IT and business users are mandated to achieve higher targets but with fewer resources (financial, human, and technology). To overcome these challenges, enterprises need a solution not only to address their core data preparation, governance, visualization, and analysis requirements but also to provide a scalable, manageable, and secure platform that enables rapid development and deployment of artificial intelligence (AI)–driven analytic applications. It's also crucial to address the needs of different personas such as executives looking for the latest performance metrics on a mobile device, analysts exploring root causes of a problem or evaluating key drivers of an output variable, or frontline employees needing a recommendation in the flow of their work, embedded into an operational application.
IDC TECHNOLOGY SPOTLIGHT The Business Value of Cloud Services for Data and Analytics Workloads
For most enterprises, achieving the lofty goal of building a secure, scalable, performant, and easy-to-use data and analytics platform in-house is not realistic because it is not their core competency, and that is where the experts come in. Cloud-based providers of analytics technologies and services have invested time and money into building cutting-edge platforms, leveraging the knowledge gained from implementing hundreds of analytics projects on their platforms that they can pass on to their customers. Enterprises get the benefit of the expertise and the economies of scale provided by their cloud-based "as a service" provider that has invested in building enhancements into cloud platforms, artificial intelligence, and machine learning (ML)–based automation to help them bridge the resource gap. The big data and analytics (BDA) market is well on its way to migrating to the cloud. According to IDC research, BDA software revenue in the public cloud domain was 12.5% of the total market in 2015, is estimated to be 35% in 2020, and is forecast to be 57% by 2024. In addition, according to IDC's Worldwide Whole Cloud Forecast, 2020–2024, public cloud services for infrastructure, platforms, and various software-as-a-service (SaaS) offerings continue to be the largest engine of growth for the whole cloud market. Cloud in all its permutations (hardware/software/services/as a service as well as public/private/hybrid/multi/edge) will play ever greater, and even dominant, roles across the IT industry for the foreseeable future. By 2024, total worldwide spending on cloud services, the hardware and software components underpinning cloud services, and the professional and managed services opportunities around cloud services will surpass $1.0 trillion while sustaining a double-digit CAGR of 15.7%. Analytics in the cloud allows enterprises to harness the computing power of cloud-based data environments, with flexible and intuitive analytics that allows users to truly make data a core driving force of their digital strategy. With the infusion of AI and machine learning, cloud-based data environments become more and more powerful but remain easy to use. Cloud-based providers are also leveraging AI and machine learning to surface insights on how usage patterns can further improve business process and value. AI and automation compensate for the scarcity of data scientists and allow humans to focus on value-added activities. IDC also predicts that the disruptions and changes triggered by the global pandemic as well as individuals' and enterprises' responses to those changes will accelerate digitalization globally, with 65% of global GDP digitalized in the next two years. Having a digital transformation strategy and a data and analytics strategy will be a critical success factor for enterprises to succeed beyond the pandemic.
#US47086020 Page 2
IDC TECHNOLOGY SPOTLIGHT The Business Value of Cloud Services for Data and Analytics Workloads
The Intelligent Enterprise Every enterprise needs to unlock the full power of its data capital to accelerate digital transformation. As shown in Figure 1, data and analytics is a core element of an intelligent enterprise. Enterprises need to leverage their systems, models, processes, and architectures to empower their workers with data. A comprehensive data and analytics strategy takes into account the current state of an enterprise's infrastructure and assets and then implements a platform that can integrate data from myriad sources, including on-premises and cloud sources, and make insights accessible to employees at all levels within the organization. FIGURE 1: Data Flow Necessary for Digital Transformation
Bot
IoT
INTERNAL INTELLIGENT EXTERNAL AR/VR PROCESSES CORE PROCESSES
Mobile
People
Assets
Connected Processes API Source: IDC, 2020
The same technologies that are driving digital transformation within organizations are also rapidly transforming work as we know it. Work is no longer constrained by a physical place or a specific time of the day. Distributed, empowered teams need to access resources and collaborate effectively and securely. According to IDC FutureScape: Worldwide Future of Work 2021 Predictions, enterprises need to embrace the idea that the hybrid workforce is here to stay and need to provide technology parity and a consistent and productive experience to all workers. They must focus on agility, adaptability, and business continuity. Traditional technology models and infrastructure will not provide the flexible, dynamic work environment required to be resilient and competitive in today's world. Challenges There is no shortage of challenges facing today's enterprises in making digital transformation a reality. Today's demands on business and IT teams span a broad range of usage patterns with their varied data access and processing requirements. These demands vary and can be conflicting. As shown in Figure 2, which presents results from a 2020 IDC survey of business intelligence (BI) end users, respondents faced many challenges related to their business strategy with data and analytics and technology.
#US47086020 Page 3
IDC TECHNOLOGY SPOTLIGHT The Business Value of Cloud Services for Data and Analytics Workloads
FIGURE 2: Enterprise Challenges to Data Analytics Strategy Business Challenges
There is a lack of a data strategy and leadership
Lack of staff with data analysis skills
Business expectations for data analysis are too high
There is a general lack of trust in results or outputs of data analysis
Lack of understanding of the value of data in general
0 20 40 60 80 100 (% of respondents)
Not challenging Neutral Very challenging
Technical Challenges
Inability to synthesize data into relevant or actionable information
Substandard IT infrastructure to run BI and analytics
Don't have access to the right tools
Tools available only to a limited number of users
0 20 40 60 80 100 (% of respondents)
Not challenging Neutral Very challenging
n = 300
Source: IDC survey of BI end users, 2020
#US47086020 Page 4
IDC TECHNOLOGY SPOTLIGHT The Business Value of Cloud Services for Data and Analytics Workloads
Most enterprises found that they didn't understand the value of data or didn't trust the results or the outputs of the data analysis and insights that were generated by tools and software. They also lacked staff with data analysis skills and indicated that they didn't have access to the right tools and had substandard IT infrastructure to run their analytics. These challenges include aging legacy technologies, suboptimal performance from their infrastructure, limited resources to manage their IT landscape, and tools that are complicated. As AI/ML-generated recommendations are embedded into more analytics tools, IDC's survey also highlighted the fact that it is important for users to trust the results of the tools. IDC's survey showed the following challenges and patterns:
» Analysts and data scientists had a lower level of trust in AI/ML-generated recommendations than executives and managers. » AI/ML-generated recommendations are trusted more when the tools providing the recommendations are newer and modern. » Trusting the recommendations from tools leads to better usage and acceptance. Respondents who trusted their AI/ML-generated recommendations accepted those recommendations. » 67% of executives stated that they lack staff with data analysis skills, and 55% of respondents said that business expectations are too high to be met with their current circumstances. According to CIO.com, LinkedIn listed "data scientist" as the most promising job in 2019 and noted that most companies cannot fill these jobs fast enough. Most organizations have too much data and not enough data scientists and analysts to glean insights from data to power their decision making. The resource gap clearly cannot be filled solely by hiring people with advanced data skills. Organizations can address the gap by using more automation in areas where machines can augment human capabilities and by looking for avenues such as embracing an as-a-service cloud provider that can provide both technical infrastructure and the expertise in data and analytics so that enterprises can focus on what they do best for their customers. In addition to the lack of skilled data scientists and advanced analysts, 56% of respondents in IDC's survey of analytics users indicated that there is often a lack of collaboration on data and analytics-related projects, and almost 50% cited that there is too much redundant data-related work in their organizations. These challenges often stem from the fact that siloed data sets and analytics tools inhibit collaboration between user groups and personas. Benefits There are many benefits to investing in a modern solution for data and analytics that leverages automation and machine learning. According to IDC research, during the full-year 2019, global spending on cloud-based analytics data management solutions grew 47%, while spending on similar noncloud technology declined by 1%. In addition, IDC's survey of business intelligence and analytics users showed that with modern tools and platforms for data and analytics:
» 59.3% of respondents saw an improvement in time to insights. » 66.4% of respondents indicated that the results of analysis are more frequently influencing action. » 65.3% of respondents reported greater trust in the results of analysis. » 78% of respondents indicated that their decision making was influenced by data and analytics.
#US47086020 Page 5
IDC TECHNOLOGY SPOTLIGHT The Business Value of Cloud Services for Data and Analytics Workloads
Figure 3 shows that respondents in more data-driven organizations saw higher percentages in benefits from investing in a modern data and analytics platform than their peers in less data-driven organizations. FIGURE 3: Benefits and Deployment Models of Data and Analytics Solutions Benefits of Investing in Modern Data and Analytics Solutions 80 70 60 50 40 30
(% of (% respondents) 20 10 0 Overall respondents Somewhat data driven Completely data driven
Faster time to insights Greater trust in results of analysis Results of analysis are more frequently influencing action Deployment Model 100 90 80 70 60 50 40 30
(% of of (% respondents) 20 10 0 Overall respondents Somewhat data driven Completely data driven
On-premises Cloud (public and private) Source: IDC, 2020
#US47086020 Page 6
IDC TECHNOLOGY SPOTLIGHT The Business Value of Cloud Services for Data and Analytics Workloads
Key Trends This section discusses the three main trends driving the modernization of data and analytics.
Digital Transformation Continues to Spur Growth Across All Data and Analytics Software Markets » Assumption: Digital transformation is still a key driver for growth in BDA spending. In an IDC survey conducted in 2020, 87% of CXOs said an intelligent enterprise was a key investment area for them. The rise of the digital-native data worker enables enterprises to achieve significant improvements in business outcomes.
» Impact: COVID-19 has had a negative impact in the short term, but enterprises have realized that to be resilient to events like the pandemic, they have to digitally transform, and that data and analytics are keys component of DX. Many organizations are investing in improving their data fluency.
Demand for Cloud-Based Software Services Continues at a Strong Pace » Assumption: Growth in public cloud–based data and analytics offerings outpaced that of on-premises deployments across all data and analytics segments. The pandemic will hasten the timeline where cloud deployments become larger than on-premises deployments. Public cloud deployments represented 30.5% of the overall data and analytics software market in 2019 and are expected to grow at a CAGR of 23.2% through 2024 compared with a CAGR of -1.2% for on-premises/other software deployment methods.
» Impact: The demand for cloud-based software and services will continue to grow as users look to cloud platforms for faster implementation times and lower costs in data and analytics initiatives.
Increase in AI/ML-Infused BDA Offerings Broadens User Base » Assumption: An increased amount of AI-driven automation is being introduced into data and analytics offerings across all user segments.
» Impact: This AI-based automation makes data and analytics products easier to use and broadens the user base and drives adoption. Vendors are using AI and ML to determine how users are leveraging products and helping them bridge knowledge and resource gaps. IDC's survey of analytics users indicated that 72% of respondents consider AI infusion of their analytics software a very positive development. Considering IBM Cloud Pak for Data as a Service IBM Cloud Pak for Data as a Service is a fully managed set of data services available on the IBM cloud. It provides organizations with a unified data and AI platform that can connect to a variety of IBM and non-IBM sources of data that customers need to access. IBM Cloud Pak for Data as a Service can be deployed in customers' environment of choice and can be fully managed by IBM, allowing customers to simplify their IT management while optimizing implementation timelines. IBM Cloud Pak for Data as a Service and IBM Cloud Satellite provide customers the ability to handle distributed workloads and offer connectors to multicloud sources. In its road map, IBM Cloud Pak for Data as a Service will allow users to run data science and AI workloads in any cloud and still leverage IBM's managed services. This will allow users to leverage AI in any part of the analytics life cycle while maintaining any data residency restrictions that the customer might have.
#US47086020 Page 7
IDC TECHNOLOGY SPOTLIGHT The Business Value of Cloud Services for Data and Analytics Workloads
Challenges IDC has identified several challenges with the adoption of AI/ML-driven automation in data and analytics that IBM needs to help enterprises address as well as the correlated opportunities:
» Misinformation About AI ■ Challenge: In recent years, the commercial sphere of automation or AI has been subject to misrepresentation of the scope of automation. Grandiose proclamations about the ability of automation to solve all challenges, or be the end of humanity, do a disservice to both enterprises and individuals trying to plan for the appropriate level of investment in automation. ■ Opportunity: The purpose of AI is to augment the capabilities of humans and lead to significantly higher throughput in a business process and greater job satisfaction/reduction in the mundane. Understanding the role of the machine and the role of the human is key in managing expectations of AI-based automation.
» Lack of Change Management ■ Challenge: As with any transformation, the challenge of changing the status quo begins with managing expectations of people. Change management is key and often overlooked. People have to be retrained appropriately, and they have to trust the automation. In addition, their usage of the technology has to be evaluated to determine if it is being used effectively. ■ Opportunity: Enterprises should engage business and IT users in the planning of the data and analytics architecture and help them understand where AI-driven automation can augment their work and build trust and a sense of ownership that will enhance usage and outcomes.
» Shortage of Experts ■ Challenge: Leveraging cloud-based platforms such as Cloud Pak for Data as a Service does not mean that BI teams or data scientists are no longer needed. There is a shortage of these highly skilled qualified experts whose efficiencies are compromised when they are involved in manual, time- consuming, and repetitive tasks and processes. ■ Opportunity: Leveraging managed services can help move the workload off these specialized teams by augmenting the capabilities of the business user or process owner.
» Unclear Use Cases ■ Challenge: Not all data and projects are best suited for automation, or the scope of improvement is not defined clearly at the outset. Projects without clearly defined objectives or metrics often result in frustrated users and/or bloated budgets because the finish line is unknown. ■ Opportunity: Enterprises that take the time to identify the best processes for automation and determine which use cases provide the highest ROI will have the most success.
#US47086020 Page 8
IDC TECHNOLOGY SPOTLIGHT The Business Value of Cloud Services for Data and Analytics Workloads
» Unknown Costs and Overruns ■ Challenge: Enterprises often are not sure about the cost of running workloads in the cloud. They don't do their homework about the cost of compute, storage, and services and are unpleasantly surprised when costs are more than what they anticipated. ■ Opportunity: Organizations that do their due diligence when defining their requirements and estimating costs are positioned for success because they can build their business cases and are better positioned to get approval for investment. Conclusion Demand for qualified data scientists and advanced analysts will continue to outpace supply, and the evolving business climate will make it more important for organizations to make insight-driven decisions. Enterprises will continue to be resource constrained even after the pandemic and will have to manage to do more with less. Automation and AI will be key to helping organizations alleviate some of their problems as well as empower them along their digital transformation journey. Enterprises should keep the following considerations in mind as they chart their data and analytics strategies:
» Having more data or more technology or more people is not a strategy for success in the digital economy. To become more data driven in decision making, begin with evaluating your current state and identifying gaps at all levels of your organization. Ask yourself the following questions: Do people have access to data? Do they have access to the right tools? If so, are they using them appropriately? What are the biggest pain points and challenges to becoming more data driven? Is IT enabling the business or fighting fires? The more data-savvy executive leadership is, the more successful the enterprise will be with digital initiatives.
» Consider a technology partner that addresses a broad set of decision-making capabilities needed by executives, managers, business analysts, data scientists, frontline employees, and automated systems/bots.
» Consider the total cost of ownership, including cost of compute and storage for highly variable as well as stable workloads, solution maintenance costs and staff utilization efficiency, and software subscription costs. Estimate the difference between using a managed service and doing it yourself, and make sure you identify hidden costs for both options.
About the Analyst
Chandana Gopal, Research Director, Business Analytics Solutions Chandana Gopal is Research Director for IDC Business Analytics Solutions market research and advisory practice. Ms. Gopal's core research coverage includes demand and supply trends in business intelligence advanced and predictive analytics, and enterprise performance management markets.
#US47086020 Page 9
IDC TECHNOLOGY SPOTLIGHT The Business Value of Cloud Services for Data and Analytics Workloads
MESSAGE FROM THE SPONSOR
About IBM IBM Cloud Pak for Data as a Service provides a core set of pre-integrated data and AI capabilities, fully managed on the IBM Cloud. Unifying market leading solutions for collecting, organizing, and analyzing data the platform helps organizations infuse AI across their business while simplifying the underlying IT and management. IBM handles provisioning, 24x7 management and version upgrades as well as helping ensure high availability and security, so you can focus on your business needs. Get started quickly and scale as you grow with this fully integrated, managed data and AI platform. Through integration with IBM Cloud Satellite, IBM Cloud Pak for Data as a Service allows customers to securely extend their managed IBM Cloud Pak for Data workloads to run across distributed IT environments. Try IBM Cloud Pak for Data as a Service today at no cost to get hands-on experience with the platform.
The content in this paper was adapted from existing IDC research published on www.idc.com.
IDC Research, Inc. This publication was produced by IDC Custom Solutions. The opinion, analysis, and research results presented herein are drawn from 5 Speen Street more detailed research and analysis independently conducted and published by IDC, unless specific vendor sponsorship is noted. IDC Framingham, MA 01701, USA Custom Solutions makes IDC content available in a wide range of formats for distribution by various companies. A license to distribute IDC content does not imply endorsement of or opinion about the licensee. T 508.872.8200 External Publication of IDC Information and Data — Any IDC information that is to be used in advertising, press releases, or promotional F 508.935.4015 materials requires prior written approval from the appropriate IDC Vice President or Country Manager. A draft of the proposed Twitter @IDC document should accompany any such request. IDC reserves the right to deny approval of external usage for any reason. idc-insights-community.com Copyright 2020 IDC. Reproduction without written permission is completely forbidden. www.idc.com
#US47086020 Page 10