Data Lifecycle and Analytics in the AWS Cloud A Reference Guide for Enabling Data-Driven Decision-Making DATA LIFECYCLE AND ANALYTICS IN THE AWS CLOUD AWS THE IN ANALYTICS AND LIFECYCLE DATA CONTENTS PURPOSE Contents INTRODUCTION Purpose 3 CHALLENGES 1. Introduction 4 2. Common Data Management Challenges 10 3. The Data Lifecycle in Detail 14 LIFECYCLE Stage 1 – Data Ingestion 16 INGESTION Stage 2 – Data Staging 24 Stage 3 – Data Cleansing 31 Stage 4 – Data Analytics and Visualization 34 STAGING Stage 5 – Data Archiving 44 4. Data Security, Privacy, and Compliance 46 CLEANSING 5. Conclusion 49 ANALYTICS 6. Further Reading 51 Appendix 1: AWS GovCloud 53 Appendix 2: A Selection of AWS Data and Analytics Partners 54 Contributors 56 ARCHIVING SECURITY Public Sector Case Studies Financial Industry Regulation Authority (FINRA) 9 CONCLUSION Brain Power 17 READING DigitalGlobe 21 US Department of Veterans Affairs 23 APPENDICES Healthdirect Australia 27 CONTRIBUTORS Ivy Tech Community College 35 UMUC 38 UK Home Office 40 2 © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. DATA LIFECYCLE AND ANALYTICS IN THE AWS CLOUD AWS THE IN ANALYTICS AND LIFECYCLE DATA CONTENTS PURPOSE Purpose of this guide INTRODUCTION Data is an organization’s most valuable asset and the volume and variety of data that organizations amass continues to grow. The demand for simpler data analytics, cheaper data storage, advanced predictive tools CHALLENGES like artificial intelligence (AI), and data visualization is necessary for better data-driven decisions. LIFECYCLE INGESTION The Data Lifecycle and Analytics in the AWS Cloud guide helps organizations of all sizes better understand the data lifecycle so they can optimize or establish an advanced data analytics practice in their STAGING organization. The document guides readers through five stages of the data lifecycle, including data ingestion, data staging, data cleansing, CLEANSING data analysis (including AI/machine learning (ML) inference and deep learning tools) and visualization, data archiving, and overall ANALYTICS data security. This guide is written primarily for IT professionals, as well as for chief ARCHIVING data officers, data scientists, data analysts, and data engineers. Technical professionals of diverse training and experience can learn how to extract more value from their data by taking advantage of the Amazon Web Services (AWS) Cloud to support data-driven decision-making. Business leaders and managers will also benefit from the guide’s overview sections and customer case studies. One could spend countless hours searching online to understand the many services available to turn data into insights. This reference guide aggregates important definitions, workflows, and the relevant AWS services for each stage of the workflow, to simplify the learning process. Here are some data lifecycle and management questions this guide will help you answer: • How do you collect and analyze high-velocity data across a variety of data types – structured, unstructured, and semistructured? • How do you scale up IT resources to run thousands of concurrent queries against your data – and then scale back down automatically to lower costs? • How do you analyze your data across platforms, so users can view, search, and run queries on multiple data repositories? • How do you cost-effectively store petabytes of data and share them on- demand with users around the world? • How do you get your data to answer questions about past scenarios and patterns, while predicting future events? 3 © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. DATA LIFECYCLE AND ANALYTICS IN THE AWS CLOUD AWS THE IN ANALYTICS AND LIFECYCLE DATA INTRODUCTION CONTENTS PURPOSE INTRODUCTION CHALLENGES LIFECYCLE INGESTION STAGING CLEANSING ANALYTICS ARCHIVING SECURITY CONCLUSION READING APPENDICES CONTRIBUTORS Introduction DATA LIFECYCLE AND ANALYTICS IN THE AWS CLOUD AWS THE IN ANALYTICS AND LIFECYCLE DATA INTRODUCTION CONTENTS PURPOSE Data within organizations is measured in petabytes, and it grows exponentially each year. INTRODUCTION IT teams are under pressure to quickly orchestrate data storage, analytics, and visualization projects that get the most from their organizations’ CHALLENGES number one asset: their data. They’re also tasked with ensuring customer privacy and meeting security and compliance mandates. These challenges LIFECYCLE are cost-effectively addressed with cloud-based IT resources, as an INGESTION alternative to fixed, conventional IT infrastructure (e.g. owned data centers and computing hardware managed by internal IT departments). By modernizing their approach to data lifecycle management, and leveraging STAGING the latest cloud-native analytics tools, organizations reduce costs and gain operational efficiencies, while enabling data-driven decision-making. CLEANSING ANALYTICS What is Big Data? A dataset too large or complex for traditional data processing mechanisms is called “big data”. Big data also encompasses a set of data management ARCHIVING challenges resulting from an increase in volume, velocity, and variety of data. These challenges cannot be solved with conventional data storage, SECURITY database, networking, compute, or analytics solutions. Big data includes CONCLUSION structured, semi-structured, and unstructured data. “Small data,” on the other hand, refers to structured data that is manageable within existing READING databases. Whether your data is big or small, the lifecycle stages are APPENDICES universal. It’s the data management and IT tools that will differ in terms of scale and costs. CONTRIBUTORS To support advanced analytics for big and small data projects, cloud services support a variety of use cases: descriptive analytics that address what happened and why (e.g. traditional queries, scorecards, and dashboards); predictive analytics that measure the probability of a given event in the future (e.g. early alert systems, fraud detection, preventive maintenance applications, and forecasting); and prescriptive analytics that answer, “What should I do if ‘x’ happens?” (e.g. recommendation engines). With AWS, it’s also technically and economically feasible to collect, store, and share larger datasets and analyze them to reveal actionable insights. 5 © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. DATA LIFECYCLE AND ANALYTICS IN THE AWS CLOUD AWS THE IN ANALYTICS AND LIFECYCLE DATA INTRODUCTION CONTENTS AWS offers a complete cloud platform designed for big data across data PURPOSE lakes or big data stores, data warehousing, distributed analytics, real-time INTRODUCTION streaming, machine learning, and business intelligence services. These cloud-based IT infrastructure building blocks – along with AWS Cloud capabilities that meet the strictest security requirements – can help address CHALLENGES a wide range of analytics challenges. LIFECYCLE What is the Data Lifecycle? INGESTION As data is generated, it moves from its raw form to a processed version, to outputs that end users need to make better decisions. All data goes STAGING through this data lifecycle. Organizations can use AWS Cloud services in each stage of the data lifecycle to quickly and cost-effectively prepare, process, and present data to derive more value from it. The five data CLEANSING lifecycle stages include: data ingestion, data staging, data cleansing, ANALYTICS data analytics and visualization, and data archiving. ARCHIVING SECURITY CONCLUSION Data Sources Data Ingestion READING APPENDICES CONTRIBUTORS Data Staging Data Cleansing Data Analytics & Data Archiving Visualization 6 © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. DATA LIFECYCLE AND ANALYTICS IN THE AWS CLOUD AWS THE IN ANALYTICS AND LIFECYCLE DATA INTRODUCTION CONTENTS 1. The first stage is data ingestion. Data ingestion is the movement of PURPOSE data from an external source to another location for analysis. Data can INTRODUCTION move from local or physical disks where value is locked (e.g. in an IT data center), to the cloud’s virtual disks. There, it can be closer to end users and where machine learning and analytics tools can be applied. During data CHALLENGES ingestion, high value data sources are identified, validated, and imported while data files are stored and backed up in the AWS Cloud. Data in LIFECYCLE the cloud is durable, resilient, secure, cost-effectively stored, and most INGESTION importantly, accessible to a broad set of users. Common data sources include transaction files, large systems (e.g. CRM, ERP), user-generated data (e.g. clickstream data, log files), sensor data (e.g. from Internet-of- STAGING Things or mobile devices), and databases. AWS services available in this stage include Amazon Kinesis, AWS Direct Connect, AWS Snowball/ CLEANSING Snowball Edge/Snowmobile, AWS DataSync, AWS Database Migration Service, and AWS Storage Gateway. ANALYTICS 2. The second stage is data staging. Data staging involves performing housekeeping tasks prior to making data available to users. Organizations ARCHIVING house data in multiple systems or locations, including data warehouses, SECURITY spreadsheets, databases, and text files. Cloud-based tools make it easy to stage data or create a data lake in one location (e.g. Amazon S3), CONCLUSION while avoiding disparate storage mechanisms. AWS services available Amazon S3, Amazon Aurora, Amazon RDS, READING in this stage include Amazon DynamoDB. APPENDICES CONTRIBUTORS 3. The third stage is data cleansing. Before data is