Tales from Building a SQL Server Data Warehouse in Azure Experiences & lessons learned from a migration to Azure August 21, 2017 Lead Analytics Architect, SentryOne Blog: www.sqlchick.com Microsoft Data Platform MVP Twitter: @sqlchick Melissa Coates Goals for This Session 1 Share decisions and lessons learned from a recent Azure implementation 2 Introduce key concepts for a deployment to Azure 3 Discuss items involved with building a DW environment in Azure Azure services & features change frequently. The information in the presentation is accurate as of mid-August 2017. Agenda Key Azure Concepts Deciding on Azure VM vs. Azure SQLDB vs. Azure SQLDW Additional Planning Considerations Building the Azure Environment Structuring Dev, Test, & Prod Environments Naming Conventions & Tags Out of scope: Configuration Decisions o Linux deployments Automation & Scheduling o Azure Stack & private cloud deployments Monitoring the Solution o On-premises physical & virtualized deployments (including Fast Track DW & APS/PDW) Key Takeaways & Open Q&A o Security in depth (though we touch on a few points) o Networking & connectivity o Troubleshooting, performance tuning & growth o Details on how to install and configure SQL Server Key Azure Concepts The Azure Lingo Azure Directory Subscription Subscription Resource Group Resource Group Subscription Resource Resource Resource Group Resource Resource Resource Group Subscription Azure Resource Manager (ARM) Azure Active Directory Active Directory https://azure.microsoft.com/en-us/overview/what-is-cloud-computing/ Office 365 Power BI https://azure.microsoft.com/en-us/overview/cloud-computing-dictionary/ Resource Groups Subscription Planning for resource groups is critical Resource Group Resource Group Resource Resource Resource Group Focus on: Resource Resource o Logical organization Resource Group o Permissions o Policies We have learned: ✓ Keep resource groups more narrow than broad ✓ Select the region (location) carefully Scope of ARM automation scripts (exception: Resource Explorer) Examples – Resources & Resource Groups IaaS PaaS IaaS vs. PaaS vs. SaaS Shared Public Cloud Software Highly Infrastructure as a Scalable (Lower Cost) Service Platform (SaaS) as a Service Infrastructure (Paas) Power BI, as a Service Office 365 (IaaS) Azure SQLDB & SQL Server in a Azure SQLDW Virtual Machine On-Premises Dedicated Virtual Azure Stack (Private Cloud) Infrastructure Server Physical (Higher Cost of Limits to Server Ownership) Scalability More Control Less Control (Higher Administration Effort) (Lower Administration Effort) Compute vs. Storage Compute Data Lake Analytics SQL Data Storage Warehouse HDInsight SQL Data Database SQL Data Warehouse SQL Server in a VM SQL Data Machine Database Learning Data Lake SQL Server in Store a VM Blob Storage Some resources scale, or even pause, compute separately from storage. Deciding on Azure VM vs. Azure SQLDB vs. Azure SQLDW Comparing the SQL Offerings in Azure (1/2) SQL Server in a Virtual Machine Azure SQL Database Azure SQL Data Warehouse (IaaS) (PaaS) (PaaS) Run full workload within an A relational database-as-a-service (DBaaS) An data warehouse-as-a- Azure virtual machine, service (DWaaS) optimized including SQL Server, SSIS, for performance and large- SSAS, SSRS, etc Non-Managed Managed Instance scale distributed workloads A traditional Azure Newer - closer feature MPP architecture SQLDB deployment parity to SQL Server (massively parallel (isolated DB) (instance level features) processing) Comparing the SQL Offerings in Azure (2/2) SQL Server in a Virtual Machine Azure SQL Database Azure SQL Data Warehouse (IaaS) (PaaS) (PaaS) Best for: Best for: Best for: ✓ Migrating existing solutions ✓New database solutions ✓DW with larger data volumes ✓ Running any software and/or (If a DW it should be < 4TB data size-- (bare min. of 1-4TB) sharding across DBs is not suitable for ✓Ability to scale compute all SQL Server features DW workloads) up/down, or pause (elasticity) ✓ Administering all aspects ✓OLTP with scaling & pooling ✓ Bring your own license ✓Data Lake Store integration needs (unpredictable workloads) (relational + nonrelational data) (Software Assurance) ✓Reduced administration of ✓ ✓Reduced administration Isolated dev/test environments DB, OS, HA, and DR ✓ ✓SLA: for the database SLA: for the VM ✓SLA: for the database More info: https://docs.microsoft.com/en-us/azure/sql-database/sql-database-paas-vs-sql-server-iaas Key Differences with Azure SQL Database Many features go first to Azure SQLDB (“cloud first”). However, there are some key features not available in SQLDB (PaaS): o PolyBase o R Services o Change data capture o CLR o DB snapshots o Some T-SQL syntax o Profiler Full list: o Non-primary filegroups https://docs.microsoft.com/en-us/azure/sql- database/sql-database-features Also, some features rely on Premium edition: o Columnstore indexes Key Differences with Azure SQL Data Warehouse (1/2) Control Node MPP Architecture Shared-Nothing Architecture Compute Compute Compute Node Node Node Decoupled Storage & Compute Data Storage Scale Up, Down, Pause PolyBase Key Differences with Azure SQL Data Warehouse (2/2) Not All Features Supported Control Node Different Data Loading Patterns Compute Compute Compute Distribution Keys are Critical Node Node Node Denormalized Data Model is Best Data Storage Take time to educate yourself on the key differences with the MPP architecture—it will affect the design & the data load processes Our Decisions on What to Use SQL Server in a Virtual Machine Azure SQL Database Azure SQL Data Warehouse (IaaS) (PaaS) (PaaS) We are using a VM for: We are using SQLDB for: We put SQLDW on roadmap: ✓ SQL Server DW ✓ A specific use case: public ✓ Future data growth ✓ Integration Services reporting solution via ✓ Future PolyBase ✓ Analysis Services (MD) SQLSkills Waits Library integration with multi- ✓ Master Data Services ✓ This SQLDB is loaded from structured data in Azure ✓ R Services the DW (in SQL Server) Data Lake Store Requires some refactoring of We need VMs for SSIS and SSAS the data load processes so we anyway, so we couldn’t justify are planning the move to migrating the relational DW to a SQLDW strategically PaaS solution at this time Our Current State IaaS PaaS Additional Planning Considerations Our Goals & Requirements for the Move to Azure Expand infrastructure to support future growth Ensure ability for Analytics Team to manage Support existing solutions with environment independently little to no redesign or refactoring Secure connectivity via VPN Minimize cost where practical Acceptable performance of hourly ETL jobs Think about trade-offs you’re willing to make for cost, performance, security, regulatory compliance, DR/backups/redundancy, and simplicity Initial Planning Before Provisioning Any Resources (1/3) o Licensing and Editions First Steps o Full Cloud Implementation vs. Hybrid (Partially On-Premises) o IaaS vs. PaaS decisions; feature comparison o Big 3: Storage, memory, CPU Capacity Planning o Networking & Cost Estimates o Scalability needs o Cost estimates: https://azure.microsoft.com/en-us/pricing/calculator/ High Availability & o Down time sensitivity (RPO, RTO) Disaster Recovery o SLAs from Azure Compliance & o Compliance: https://www.microsoft.com/en-us/trustcenter/compliance/default.aspx Security Initial Planning Before Provisioning Any Resources (2/3) o How administrative & owner permissions will work if decentralized Domain service o Read and/or write permissions for source or related systems accounts & o One domain service account per service, per environment credentials o Sync to Azure Active Directory for domain users & groups o Service principals for certain resources in Azure are very important o Geographic location of data o Proximity to business users Azure region: o Co-location of related resources primary location o Minimizing latency & o data redundancy Minimizing data egress charges (very inexpensive though) o Not all resources/services are available in every region Initial Planning Before Provisioning Any Resources (3/3) We went with a backup/restore approach for the Migration Method SQL Server files for the DW. The exception to this was SSISDB custom auditing Backup/ Upload & logging objects. This DDL was deployed from Restore a VHD SSDT after the SSISDB catalog was configured. Fail Over from Replication AlwaysOn Info on migration techniques: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sql/virtual- machines-windows-migrate-sql Ship a Database Azure SQL Data Warehouse migration utility: Hard Drive Migration Service https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse- migrate-migration-utility Structuring Dev, Test & Prod Environments Separation of Dev, Test, Prod Environments Most commonly environments are segregated by: 1. Resource Groups, 2. Subscription, or 3. Directory, or 4. A combination of 1 and 3, or 2 and 3 Visual Dev Test Prod Studio Environment Environment Environment Deploy Deploy Deploy from via ARM via ARM Visual Template Template Studio For large or multi-tenant implementations: be aware of Azure limits before deciding. Option: Separate By Directory Dev Directory Test Directory Prod Directory Pros: ✓ Clear boundary Subscription Subscription Subscription ✓ Offers the most scalability Cons: ✓ Resource Group: Resource Group: Resource Group: More infrastructure to Internal Rptg Internal Rptg Internal Rptg manage ✓ A lot of objects intermixed in a subscription - need Resource Group: Resource Group: Resource Group: clear resource