Building Blocks of Intelligence Suite

Melissa Coates Blog: sqlchick.com Analytics Architect Twitter: @sqlchick SentryOne

Presentation content last updated: April 21, 2017 Building Blocks of Cortana Intelligence Suite

Agenda 1. Azure Overview 2. Tour of Cortana Intelligence Suite: Purpose, Use Cases, Building Blocks 3. Resources for Samples & Tutorials Cortana Intelligence Suite

Azure Overview Azure Platform Microsoft Azure is a cloud and infrastructure for building, deploying, and managing applications and services through a global network of Microsoft-managed datacenters.

Azure provides services and supports many different programming languages, tools and frameworks, including both Microsoft-specific and third-party software and systems.

http://azureplatform.azurewebsites.net/ Azure Objectives

Extend data center Separate compute Scalability infrastructure from storage

Shared Simplified infrastructure = Simplified management = development = reduced cost of reduced cost of ownership faster time to value ownership Open source Self-service interoperability Integration of services provisioning

Built-in high availability & Shared code base with disaster recovery (some) on-premises resources Types of Cloud Deployments IaaS ✓ Azure Virtual Machines IaaS PaaS Infrastructure-as-a-Service ✓ Azure SQL Data Warehouse ✓ Azure SQL Database ✓ Azure HDInsight PaaS ✓ Azure Data Lake Store Platform-as-a-Service ✓ Azure Stream Analytics SaaS SaaS ✓ Power BI ✓ SharePoint Online Software- ✓ Office 365 ✓ Exchange Online as-a- Service ✓ Azure Data Catalog ✓ Azure Data Factory ✓ Azure Machine Learning Types of Cloud Deployments On-Premises Cloud Shared High Infrastructure Scalability (Lower Cost)

Dedicated Infrastructure (Higher Cost of Ownership) Limits to Scalability More Control Less Control (Higher Administration Effort) (Lower Administration Effort) Cortana Intelligence Suite Formerly known as Cortana Analytics Suite Cortana Intelligence in a Sentence

Collection of Azure services Team Data Science Process “Cortana Intelligence (TDSP) is a platform and a process to perform advanced analytics from start to finish“

Source: Chris Testa O’Neill Data Science Team at Microsoft Cortana Intelligence Suite Objectives

Big Data and Advanced Analytics with less cost and effort

“Intelligent action” from or automated systems

Enable opportunities for automation and innovation: ✓ Templates ✓ Preconfigured solutions ✓ Interoperability ✓ Easier to operationalize solutions Cortana Intelligence Suite = a marketing term for ✓ Open standards a bundle of integrated services Cortana Intelligence Suite Sample Scenarios Cortana Intelligence Components

Information Big Data Machine Learning Visualization Management Stores & Analytics

Power BI SQL Data Machine Data Data Warehouse HDInsight Factory Catalog Learning Intelligence

Cortana Bot Cognitive Data Lake Data Lake Stream Event Hub Framework Services Store Analytics Analytics

Plus other solution components Azure Azure Express Azure IoT Hub Blob Document Azure SQL Azure Applications Virtual Azure such as: Automation Active Route Key Storage DB Database Virtual Network Backup Machine Directory Vault Storage Layers vs. Compute Services

Compute

SQL Data Azure SQL SQL HDInsight Storage Warehouse Database

Machine Data Lake Azure Batch SQL Data Azure SQL Learning Analytics Warehouse Database

Stream Event Analytics SQL Hub Document Data Lake Blob Server DB Store Storage

IoT Hub Cortana Intelligence Suite

Scenario:

Loan Repayments Loan Repayment Scenario Current State:

Organizational data Operational Operational • Customer Reporting Data Store application (income, assets) Analytics & Batch ETL • Loan history Organizational reporting Data tools • Payment activity OLAP Historical Data Analytics Warehouse Semantic Third party data Layer • Credit bureau Third Party Data Data Marts history Master Data Loan Repayment Scenario Desired State:

Streaming Data Alerts Predictive Analytics Near-Real-Time Monitoring model to predict repayment ability Devices & Advanced Sensors Data Lake Analytics Fraud alerts re: line of credit withdrawals Social Media Self-Service Operational Reports & Sentiment analysis for Data Store Hadoop Machine Models phone records Demographics Learning Data Text analytics for e- Batch ETL records Federated Queries Data OLAP Operational Organizational Analyze comments made Data Warehouse Semantic Reporting Layer on social media Data Historical Third Party Master Marts In-Memory Analytics Data Data Model Cortana Intelligence Suite

Tour of Cortana Intelligence Suite Cortana Intelligence Suite

Azure Data Lake

ADLS + ADLA: Generally Available as of November 2016 HDInsight: Generally Available as of October 2013 Azure Data Lake A collection of 3 services:

Big Data Data Lake queries-as-a-service Analytics

Big Data HDInsight cluster-as-a-service

Big Data storage as-a-service Data Lake Store https://info.microsoft.com/CO-Azure-WBNR-FY16-01Jan-19-Azure-Data-Lake-Overview-Registration.html Azure Data Lake Store ‘storage Big Data - as - a - service’ house all types of data: One data its in native format quantities of disparate sources of A repository o o o architectural platform (ex: (ex: sales, inventory) Traditional operational data (ex: tweets, e Human (ex: IoT, logs) Machine - generated data - for analyzing large generated data - mail) to Purpose Big Data Storage ✓ Storage to support analytic applications ✓ HDFS (Hadoop Distributed ) for the cloud, with no size limitations ✓ Stores data in its native format: objective is to not reformat

File System Optimized for Analytics WebHDFS-Compatible ✓ An alternative to general purpose Azure ✓ Accessible to all HDFS- Blob Storage compliant projects ✓ Parallel read scans (If integrated with ✓ Scaled out over multiple machines HDInsight) ✓ Low latency writes

✓ Large file sizes Azure Data Lake Store Lake Data Azure Common Use Cases

Big Data Analytic Workloads ✓ Agility: reduce up-front effort for ingestion of data (defer work to ‘schematize’) ✓ Optimized to work with ADL Analytics ✓ Also supports HDInsight (Hadoop)

Influx of Data Active Archive ✓ Acquire multi-structured data ✓ Rarely used data ✓ Persist data in its native format ✓ No limits on account/file sizes Data Science ✓ Analytic sandbox Data Warehouse Support ✓ Raw & curated data zones

✓ Staging area for DW Azure Data Lake Store Lake Data Azure Azure Data Lake Store Structuring a Data Lake Data a Structuring Metadata Metadata | Security | Governance | Information Management Staging Zone Staging Temp Zone Raw Data/ Transient/ AzureData Lake Store CuratedData Analytics Sandbox Zone 1 Ingest web logs Building Blocks into the data lake Repository for Ingestion of New Types of Data for purpose of analyzing website visits

2 Ingest social media data into the data lake for Web Logs 1 purpose of analyzing 2 Data Lake comments Social Media Store

Data Azure Data Lake Store Lake Data Azure Azure Data Lake Analytics Big Data ‘queries - as - a - service’ Purpose Big Data Processing ✓ Ability to process any data, regardless of size or structure ✓ Query scalability: resources allocated for each query ✓ YARN application built on open standards ✓ Optimized to work with Azure Data Lake Store

Simplification ✓ Abstracts away the cluster nodesfocuses on convenience, efficiency, and scalability ✓ U-SQL = familiar SQL and C# to reduce learning curve ✓ Separation of ADL Analytics from ADL Store: easier to manage,

debug, and optimize Azure Data Lake Analytics Lake Data Azure U-SQL SQL + C# ✓ New big data query processing language ✓ Applies ‘schema on read’ logic ✓ Mix of multiple SQL dialects (T-SQL and ANSI SQL) ✓ Native extensibility of user code written in C# ✓ Full C# expressions ✓ Reuse in assemblies ✓ Define custom types, functions, etc. ✓ Automatically scales and parallelizes across nodes

https://info.microsoft.com/CO-Azure-WBNR-FY16-01Jan-19-Azure-Data-Lake-Overview-Registration.html Azure Data Lake Analytics Lake Data Azure Common Use Cases Focus on Business Logic ✓ Focus on jobs rather than on infrastructure for a cluster ✓ Abstracts away the cluster nodes and focuses on convenience, efficiency, and scalability File Management Various Size Workloads ✓ Scheduled batch processes to ✓ Scalability on an individual manage ADLS or Blob storage job basis (U-SQL is *not* currently suitable ✓ Objective is to not reserve for ad hoc query workloads) capacity that’s not needed

U-SQL is a Fit

✓ Skillsets and preferences are a fit (U-SQL = SQL + C#) Azure Data Lake Analytics Lake Data Azure Building Blocks 1 U-SQL: execute federated query Batch Analysis of Data Stored in the Data Lake from SQL DW + ADL Store to analyze data

2 U-SQL: write Data Lake 1 results back to Analytics SQL Data Warehouse ADL Store U-SQL 2 1 Web Logs

Data Lake Social Media Store

Data Azure Data Lake Analytics Lake Data Azure Azure HDInsight Hadoop - based distribution for ‘Big Data’solutions distribution based … Purpose Big Data Processing ✓ A Big Data ‘cluster-as-a-service’ for distributed data processing, scaling, and querying capabilities ✓ Supports the Apache Hadoop open source ecosystem: Hive, Spark, R, Solr, Storm, etc ✓ Considered a ‘compute’ service ✓ Linux or Windows

3 Ways to Manage HDInsight Hortonworks Partnership ✓ As a service ✓ Based on Hortonworks Data ✓ On-demand (ADF) Platform (HDP) distribution

Azure HDInsight Azure ✓ Inside a virtual machine ✓ Microsoft + Hortonworks joint engineering team Azure HDInsight ✓ Data Exploration ✓ Development/POC ✓ ✓ ✓ Big DataScenarios Common UseCases Part of data scientist’s toolbox on before investing out proof of concept Inexpensive way to test Integration with other open source projects SQL caneasily do with ADL Analytics You want to manage a cluster & go beyond what U Leveraging the Hadoop ecosystem - prem big big data cluster in in an Volume Volume | Variety | Velocity ✓ Data EngineProcessing and analytics systems data movement for data sent to DW Computations, transformations and ✓ One Large - Time Data Loads Time batch jobs (ex: Sqoop) - Azure Data Lake Pricing Capability Analytical Source Open Storage Data vs HDInsight onADLA Deciding No U Scale per per U Scale Pay AzureLake Data Store - SQL batchSQL processing jobs Azure Data Lake Analytics (ADLA) - as - you - - go SQL job SQL Permanent& on cluster data Big Yes SQL projects Spark, (Hive, Pig, open source all Supports AzureStorage Blob or AzureLake Data Store - on - Hadoop, etc) Hadoop, HDInsight - demand Cortana Intelligence Suite

Azure SQL Data Warehouse

Generally Available as of July 2016 Relational Data Warehousing in Azure*

SQL Server in a Virtual Machine Azure SQL Database Azure SQL Data Warehouse (IaaS) (PaaS) (PaaS) Run SQL Server workloads A relational database-as-a-service A DW-as-a-service (DWaaS) (including SSAS, SSRS, (DBaaS). Close feature parity with optimized for performance SSIS, etc) in an Azure SQL Server. and large scale, distributed Virtual Machine. workloads.

Best for: Best for: Best for: ✓ Migrating/extending ✓ < 1TB data volume (sharding ✓ Larger data volumes on existing database across DBs is not suitable for DW MPP architecture ✓ Administer all aspects workloads) ✓ Ability to scale up/down/ ✓ Bring your own license ✓ OLTP with scaling & pooling pause on-demand ✓ Dev/test scenarios needs (unpredictable workloads) ✓ Combining relational + ✓ Reduced administration of DB, nonrelational data *Excluding: Other technologies in a VM such as O/S, and hardware Oracle, and Big Data technologies like Hive, etc Azure SQL DW https://azure.microsoft.com/en - us/documentation/articles/sql - data - warehouse - overview - what - is/ ✓ ✓ ✓ differences ( family Part distributions architecture processing (MPP) parallel Massively workloads Large of the SQL Server Server the SQL of - scale DW with some key with ) across Azure SQL DW Purpose ✓ PolyBase ✓ ✓ ✓ ✓ MPP Scale ✓ Elastic Scale queries & data loads T Clustered columnstore indexes used by default Built on SQLServer(with some differences & limits) Massively parallel processing (MPP) Cloud demand or on schedule Scale up/down on - SQL for Hadoop - based, multi - Out Query EngineQuery Out - - tenant, platform ✓ ✓ ✓ ✓ Storage +Compute ability ability independently of data storage Increase/decrease/pause compute compute billing Data Warehouse Units (DWUs) controls vs. compute Separate billing & scaling for storage Storage and compute is decoupled - as - a - service (PaaS) Azure SQL DW ✓ Data Variety ✓ Varying Workloads ✓ ✓ Analytical andAdHocWorkloads Common UseCases ( Integration with various data source types and data intensive intensive analytical operations) up/down (ex: data loads or ability to scale ‘compute’ Large OLTP Batch inserts and updates i.e., takes advantage of PolyBase) workloads are *not* suitable for SQL DW w orkloads which which suit the ✓ Cloud Large Scale on environments in cloud than Easier to provision large - premises structures DW - scale Azure SQL DW Two Ways of Using PolyBase Two WaysofUsing 1 Joe Green Amanda Name Driver where the data currently resides. consolidatedquery. Objective:avoid data movement from Querying Brown Warehouse SQL SQL Data Azure Azure Escort Civic Car Model of relational+ semi/unstructured data in asingle WA WA State Telemetry Storage Azure Azure Blob data results query T - SQL SQL Joe Green Amanda Name Driver Brown Escort Civic Model Car WA WA State 25 58 Speed Avg 10 91 Driven Miles Azure SQL DW Two Ways of Using PolyBase Two WaysofUsing 2 behavior. parallelprocessing DW SQL dueto its method forloading the CurrentlyisPolyBase to target. Data movement Source Data recommended PolyBase from sourcefrom Script Control Node Compute Compute Compute Node Node Node Data Storage Building Blocks 1 Load staging data from on-premises Modern Data Warehouse with a Data Lake source systems to the data lake

2 PolyBase used for loading from the data lake to the SQL Data data warehouse Line of Business Warehouse Systems Data Lake 1 Analytics 3 PolyBase used for 2 federated queries 3 between the DW Web Logs & ADL Store Data Lake *Performance is not Store yet optimized for Social Media Data this use case, but it

Azure SQL Azure DW works* Cortana Intelligence Suite

Azure Data Factory

Generally Available as of August 2015 Azure Data Factory Factory analogy:Factory A service for buildingandoperatingdatapipelines Materials Raw Raw Materials Acquire Acquire Raw Raw Integration and Integration Preparation Finished Goods Finished Deliver Goods Azure Data Factory ✓ ✓ Data Orchestration Purpose Sources Servesas the ‘glue’ for stitching together services data processing Automation of data ingestion, orchestration, and grouped in a Activities Pipeline Destination (Sink) Components of a Factory Pipeline Grouping of activities Dataset Tables, files, folders, Activity Dataset documents Actions performed on the data Linked Service Data source Activity Linked Service and/or compute resource for Activity execution of activity

Source Compute Environment Activity Sink Ex1 Azure SQL Database Azure SQL Database Stored Procedure Azure SQL Data Warehouse

Azure Data Factory Data Azure Ex2 Azure Data Lake Store Azure Data Lake Analytics U-SQL Script Azure Data Lake Store Azure Data Factory How Does ADF compare to Integration Services? to Integration ADF compare How Does Customtransformations Data sourceconnection Data Built IntegrationServices SQL Server Agent SQL Server Source/ destination - in transformations in ControlFlow Scheduling Data Flow Data Azure Factory Data Linked service Monitoring Scheduling Activities Pipeline Dataset Pig, C# Stored scripts Copy Hive, proc ------Azure Data Factory ✓ ✓ ✓ Data PlanningFactory Factories Multiple forOne vs. ✓ ✓ Key DifferencesfromSSIS Alignment of each factory with a Visual Studio project Studioa Visual with factory of each Alignment ADF diagram can get large An modular) combined together in ADF (not Pipeline intent and scheduling are all Copy activity Pig, C#, or SQL DB Transformation capabilities: Hive ✓ ✓ Data lineage can be one be tracked can across lineage Data volume with help can pipelines within activities Grouping ADF data data one canbeusedforonly management gateway stored procs, , ADF ✓ JSON JSON scripts JSON hand elements are Most ADF diagram - written ADF Azure Data Factory Azure Data Factory ADF Monitoring & Management App & Management Monitoring Azure Data Factory Stage Into Stage Common UseCases OperationalizeSolutions Big Data Processing ✓ ✓ ✓ ✓ support cloud data sources Certain Azure services only Scheduling Scheduling of data on & it will spin up/tear down an HDInsight cluster Provide the script you want to run (Hive, Pig, etc) HDInsight & big data stores are its strength - demand Supported Data Source Supported processing scripts Data is in the CloudData in the is ✓ ✓ scripting Comfortable with is in the cloud Sourceand/or destination Building Blocks 1 PolyBase is executed within a Hybrid Data Integration Data Factory copy activity for movement of data; Activity windows handle incremental data loads

SQL Data Line of Business 2 Data Factory copy Warehouse Systems Data Lake activity is used for 2 Analytics movement of Copy 1 Activity data; Activity PolyBase windows handle Web Logs Copy Activity incremental data Data Lake loads Store Social Media

Data Azure Data Factory Data Azure Cortana Intelligence Suite

Azure Machine Learning

Generally Available as of February 2015 Azure Machine Learning A service for buildingpredictive analytics solutions Custom R or Python processing Data pre modules - Execution of Algorithms algorithms Purpose Predictive Analytics ✓ Build predictive models using statistical techniques ✓ Learn from existing data to forecast future behaviors, outcomes & trends ✓ Minimize learning curve with predefined algorithms and drag & drop authoring environment

✓ Extensible with R and Python Azure Machine Learning Machine Azure Azure Machine Learning ✓ ✓ ✓ Descriptive Analytics ✓ ✓ Finding Anomalies Common UseCases recommendations Personalized offer customer service age group) to improve (ex: bybuying habits or Customer segmentation Analysis of returns Locating unusual or abnormal equipment readings Examining patterns for detection of fraud ✓ ✓ ✓ ✓ ✓ ✓ ✓ Predictive Analytics Student Student dropouts Hospital readmissions Machine maintenance & smart buildings Weather predictions Customer retention Product demand & revenuepredictions Credit risk Azure Machine Learning ✓ Operationalizing aSolution ✓ ✓ Started Getting MachineLearning with Common UseCases Image Image from: https://powerbi.microsoft.com/en Deploy as aweb service Extensible with custom R or Python Lowers the learning curve - us/blog/power - bi - azure - ml/ Building Blocks 1 Tested ML model is Operationalizing an ML Model published as a web service

2 Execution of the 1 ML model is invoked by calling the web 3 Machine service Learning SQL Data Line of Business Warehouse 2 Systems Data Lake 3 Integrate scored Analytics results to the data warehouse for further Web Logs analysis, using Data Lake Data Factory Store Social Media

Data Azure Machine Learning Machine Azure Cortana Intelligence Suite

Power BI

Generally Available as of July 2015 Power BI Data analysis andvisualization tools Data analysis Purpose Data Preparation, Data Modeling, Data Visualization

✓ Set of desktop, web, and mobile tools Power BI Power Power BI ✓ Third Party Reporting ✓ Front ✓ ✓ ✓ Self Common UseCases isolated isolated reporting scenarios 3 Corporate BI sources Reports and dashboards from Data visualizations Data preparation & cleansing imported & refreshed in Power BI Service Mashup of data into a small data model which is rd - Service BI Service party content packs quick start for - EndReporting via live via live queries ✓ App Embed Custom in ✓ Prototyping (separate Azure Service) Power BI Embedded and reports structure, calculations, Test out ideas for data Building Blocks 1 Connect Power BI to the data Self-Service and Corporate BI Scenarios warehouse in DirectQuery mode 1 to facilitate Power BI corporate 2 reporting 3 scenarios

Machine 2 ML scored results SQL Data Learning Line of Business Warehouse can be retured Systems Data Lake Analytics directly to Power

BI Power BI Power

Web Logs 3 Utilize Power BI to Data Lake facilitate self- Store service BI Social Media See next page Data Power BI Warehouse SQL Data SQL Flat File Bimodal IntelligenceBimodal Business Blocks Building Database On Source - Prem A Power BI Desktop Gateway B C 2 Imported Power BI Dataset Power BI Service Dataset Dataset Live Connection Power Power BI C Power Power BI Report B Power Power BI Report A 1 D D 1 B A C 2 dataset from the imported returned results are B & User runs Report imported dataset imported schedule created for refresh Data prepared mashup Data reports reports published model and Data present present report in to gateway via is requested Data A User runs Report Cortana Intelligence Suite

Azure Data Catalog

Generally Available as of April 2016 Azure Data Catalog Register, manage, search, and explore search, manage, Register, Search by: Tag, Expert Name Expert Source Type, Object Type,Object Data Preview (first (first x rows) Data Profiling Data Profile Info organizational data sources datasources organizational Azure Data Catalog Register, manage, search, and explore search, manage, Register, Tags for each column how to request access organizational data sources datasources organizational How to connect & Descriptions for each column Azure Data Catalog APIs in conjunction with custom portal Alternative to ADC interface: Purpose Data Documentation & Discovery ✓ Enterprise-wide metadata catalog for data assets ✓ Simplified data source discovery via search ✓ Enrich & understand assets with tags & annotations ✓ Collaboration between data producers & data consumers

Important to Know ✓ One Data Catalog per organization (not one per subscription) ✓ Authentication only accepts an organizational account

(cannot use a ) Azure Data Azure Catalog Azure Data Catalog Facilitate Facilitate Self Data Discovery& Provisioning DataSourcesforDocumentation Centralized Common UseCases ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Reduce duplication of effort multiple sources Assist combining data from Data dictionary Who to contact to request access Users search discover to data assets Analytic systems Data warehouse, marts Line of business systems - Service BI Service ✓ ✓ ✓ Data lake File system Reporting Services Capture ‘Tribal Capture ‘Tribal Knowledge’ ✓ ✓ Enhance understanding subject matter experts data is maintained by Documentation about the Building Blocks 1 Register data from the data Data discovery warehouse

2 Selectively Power BI register curated data from the

Catalog 1 data lake store

Machine Learning SQL Data View metadata; Line of Business 3 Warehouse Systems Data Lake view data from a Analytics table or view using Power BI or 2 Excel Web Logs Data Lake Store Social Media Data Data 3

Azure Data Azure Catalog Cortana Intelligence Suite

Azure Stream Analytics + Event Hub

Generally Available as of April 2015 Azure Stream Analytics Real - time analytics analytics for highvelocity events time streaming Purpose Stream Analytics ✓ Analytic processing engine for streaming events ✓ Internet of Things (IoT) solutions for data in motion from devices, sensors, social media, etc

Event Hub ✓ A publish-subscribe service that Simplification handles high volume & high ✓ Alternative to batch loading velocity data streams processes ✓ Allows events to be ingested into ✓ Lower bar to entry for Azure from many platforms & developers by using SQL- devices (another option: IoT Hub) like language ✓ The preferred method of event ✓ More straightforward than

ingestion for Stream Analytics an HDInsight Storm cluster Azure Stream Azure Analytics Azure Stream Analytics ✓ ✓ Identity Protection ✓ ✓ ✓ Web Logs ✓ ✓ Monitoring& Telemetry Device Common UseCases scenarios Identity theft Real Errors or degraded experience A/B testing Clickstream analysis Business continuity Level beyond acceptable threshold - time fraud alerts ✓ SocialMedia sentiment sentiment analysis Real - time ✓ Pricing Demand ✓ Traffic x minutes Bookings in past conditions Accident & traffic - ✓ Inventory Levels Based register checkouts Shelf volume vs. Building Blocks 1 Events ingested to the raw data Fraud Alerts for Unusual Activity event queue

4 1 2 2 Consume & Stream process data for a ATMs Event Hub Power BI Analytics 3 window of time

5 3 ML for Machine fraud predictions Learning SQL Data Line of Business Warehouse Systems Data Lake 4 Real-time Analytics reporting and/or

alerts Stream Analytics Stream

Web Logs 5 Persist data for Data Lake historical Store Social Media reporting & Data Data analysis Azure Azure Catalog Cortana Intelligence Suite

Cognitive Services

General Availability Dates Vary Per API Cognitive Services build intelligent systems buildintelligent Set of SDKs, andcloudservices APIs, to Purpose Give Applications a “Human Side” ✓ Designed to make applications more personalized, intelligent, and engaging ✓ Set of APIs, SDKs, and cloud services to see, hear, and interpret ✓ Vision ✓ Emotion ✓ Speech ✓ Face Recognition ✓ Language ✓ etc… ✓ Some APIs supported by Azure Machine Learning or Bing services

✓ Integration with U-SQL in Azure Data Lake Analytics Cognitive Cognitive Services Common Use Cases

Sentiment Analysis ✓ Detect key phrases & topics being discussed ✓ Identify feedback posted to website ✓ Convert speech to text ✓ Emotion recognition

Security Systems Search & Auto-Suggestions ✓ Facial detection and recognition ✓ Completion of partial search ✓ Video intelligence queries

Personalized Shopping Experience

Cognitive Cognitive Services ✓ Recommend items likely to be purchased together Building Blocks 1 Perform text analytics on Text analytics & Sentiment analysis customer e-mail records using cognitive

ATMs Stream Power BI capabilities Event Hub Analytics integrated in U- SQL

Machine 2 Perform Learning SQL Data sentiment analysis Line of Business Warehouse Systems Data Lake 3 on customer Analytics phone records

1 1 3 Integrate results 1 Cognitive Web Logs 2 in the data Services Data Lake warehouse for Store

Cognitive Cognitive Services further analysis Social Media Data Data Catalog Cortana Intelligence Suite

Bot Framework

In Public Preview Purpose Conversation Agent ✓ Automated, yet contextual & natural, interactions with users ✓ Enable applications and services to have a conversational user

interface (CUI) Bot Framework Bot Common Use Cases Messaging Bots ✓ Web chats ✓ Text/SMS conversation

Content Bots Watcher Bots for Alerting ✓ Share relevant content, ✓ Flight is delayed such as news or weather, ✓ Dental appointment reminder with you

E-Commerce Bots Bot Framework Bot ✓ Order food ✓ Book a flight ✓ Check inventory for a product Building Blocks 1 Conversation agent on website Web-based customer service to handle common customer questions via chat Stream ATMs Event Hub Power BI Analytics 2 Integrate bot with Cognitive Services to enhance Machine capabilities & Learning SQL Data Line of Business perform text Warehouse Systems Data Lake 3 analytics Analytics

3 Integrate results in the data Web Logs Cognitive

Bot Framework Bot Services warehouse for Data Lake Store 2 further analysis Social Media Data Data 1 Catalog Bot Framework Cortana Intelligence Suite

Cortana Assistant Why is it Called “Cortana Intelligence Suite” Anyway? Cortana is a fictional character in the Halo novel & video game series. Cortana is “smart AI” which can learn and adapt.

Windows Digital Assistant Cortana was the inspiration for the Cortana Intelligence Suite Windows Digital Assistant. According to Joseph Sirosh, Cortana symbolizes the Initially Cortana was a codename, but contextualized intelligence they got such a strong response that intend to achieve with the suite Microsoft kept the name. of tools. Cortana Assistant A Personal Personal Digital ssistant Cortana Assistant PowerBI integration not currently available Purpose Apply Language More Pervasively ✓ Virtual personal assistant for: ✓ Asking questions ✓ Finding things on PC ✓ Managing ✓ Task completion ✓ Monitoring & alerts ✓ Tracking packages

Integration ✓ Reminders are integrated between Windows devices ✓ Search within other apps, such as a calendar Cortana Cortana Assistant ✓ Interact with bots to make requests Cortana Assistant ✓ Work With Bots ✓ Reminders Common UseCases a meeting order or schedule bot to place an Interact with a Help with remembering appointments Building Blocks 1 Cortana for voice- based customer Voice-based customer assistance assistance, integrated with the Bot

ATMs Stream Power BI Framework Event Hub Analytics 2

2 Render a Power Machine BI report using Learning SQL Data Line of Business Cortana Warehouse Systems Data Lake Analytics

Web Logs Cognitive Services

Data Lake Cortana Assistant Cortana Store 1 Social Media Data Data Catalog Bot Framework Cortana Cortana Intelligence Suite

Wrap-Up Cortana Intelligence Components

Information Big Data Machine Learning Visualization Management Stores & Analytics

Power BI SQL Data Machine Data Data Warehouse HDInsight Factory Catalog Learning Intelligence

Cortana Bot Cognitive Data Lake Data Lake Stream Event Hub Assistant Framework Services Store Analytics Analytics

Plus other solution components Azure Azure Express IoT Hub Blob Document Azure SQL Azure R Excel Applications such as: Automation Active Route Virtual Storage DB Database Virtual Analytics Network Machine Directory Quick Reference of Cortana Intelligence Components

Enterprise Big data Event ingestion Data visualization metadata catalog storage for streaming + self-service data & data dictionary & IoT prep & modeling Power BI Data Data Lake Event Hub Catalog Store Personal digital assistant Data ingestion, Query Processing Cortana orchestration, service for engine for and batch streaming & IoT Assistant big data Stream Data processing Data Lake processing Analytics Agents which Factory Analytics automate Bot processes Framework Relational data Big data Predictive warehousing at clusters with analytics scale integrated service Extends SQL Data support for Machine applications to with big data HDInsight Warehouse Hadoop Learning Cognitive see, hear & open source Services interpret projects Cortana Intelligence Suite – Current State

Young Set of Services ✓ Many services have become generally available relatively recently ✓ Functionality is still maturing

Integration Still Evolving ✓ The ultimate goal is deep integration between many Azure services Cortana Intelligence Suite

Resources for Samples & Tutorials Team Data Science Process The ‘Process’ part of Cortana Intelligence Suite

https://azure.microsoft.com/en-us/documentation/learning-paths/data-science-process/ Cortana Intelligence Gallery

https://gallery.cortanaintelligence.com Cortana Intelligence Gallery – Solutions (1/3)

https://caqs.azure.net/ Cortana Intelligence Gallery - Solutions (2/3) Cortana Intelligence Gallery – Solutions (3/3)

Info on what has been provisioned & what steps need to be taken next Github

https://github.com/Azure/Cortana-Intelligence-Gallery-Content/tree/master/Tutorials LearnAnalytics@MS

http://learnanalytics.microsoft.com/ Azure Portal Azure Virtual Machines Azure Documentation Azure Learning Paths Visual Studio Projects Channel 9 Videos

https://channel9.msdn.com/Series/Azure-SQL-DW AppSource

https://appsource.microsoft.com/en-us/ Additional Reading

Setting up a PC for Azure Cortana Intelligence Suite Development http://www.sqlchick.com/entries/2016/6/17/setting-up-a-pc-for-azure-cortana- intelligence-suite-development

What is the Cortana Intelligence Suite? http://www.sqlchick.com/entries/2015/8/22/what-is-the-cortana-analytics-suite

Should You Use a SQL Server Marketplace Image for an Azure Virtual Machine? http://www.sqlchick.com/entries/2016/4/2/should-you-use-a-sql-server-marketplace- image-for-an-azure-virtual-machine

How to Build a Demo/Test Environment for Azure Data Catalog http://www.sqlchick.com/entries/2016/4/20/how-to-create-a-demo-test-environment- for-azure-data-catalog

Overview of Azure Data Catalog in the Cortana Intelligence Suite http://www.sqlchick.com/entries/2015/9/15/overview-of-azure-data-catalog-in-the- cortana-analytics-suite Thank You for Attending!

To download a copy of this presentation: SQLChick.com “Presentations & Downloads” page

Melissa Coates Blog: sqlchick.com Analytics Architect Twitter: @sqlchick SentryOne

Creative Commons License: Attribution-NonCommercial-NoDerivative Works 3.0