Our Journey to Devops at Microsoft
Total Page:16
File Type:pdf, Size:1020Kb
Azure DevOps Randy Pagels Azure Specialist - Application Development US Great Lakes Region How Microsoft Engineering uses Azure DevOps MS Engineering uses 1ES to build Azure DevOps, Windows 10, Visual Studio, Bing…etc. …using Azure DevOps …building 146,000 times per day …deploying 82,000 times per day … for millions of users + most of Microsoft. Microsoft’s DevOps Tooling – enhanced by GitHub Security Package Registry Actions The journey so far Sprint 162 Nov 2019 VSTS Preview 1ES Windows on Git Azure Pipelines Sprint 1 Sprint 29 Sprint 67 Sprint 102 Sprint 140 August 2010 June 2012 June 2014 May 2017 Sep 2018 400 Open Source Projects Microsoft on GitHub VS Code Join Linux GitHub March 2010 July 2014 April 2015 Foundation Acquired Nov 2016 Oct 2018 8.8k open source projects 25k employees contributing Before Azure DevOps Microsoft Confidential One Engineering System (”1ES”) One Engineering System with Azure DevOps Microsoft Confidential 1ES Growth 100,000 80,000 Non-engineering Users 60,000 40,000 Engineer Users 20,000 0 DevOps at Microsoft Azure DevOps is the toolchain of choice for Microsoft engineering with over 110,000 active internal users ➔ https://aka.ms/DevOpsAtMicrosoft 4.6m 28k 442k Builds per month Work items created/day Pull Requests per 155pb 500k month Build artifacts managed Work items updated/day 2.4m Private Git commits per 8.8k 24k 82,000 month Open Source Repos Employees contributing to open source Deployments per day Data: Internal Microsoft engineering system activity, November 2019 One Engineering System with Azure DevOps Azure Networking Microsoft Confidential Software Products 500GB+ 6000+ Source code Repos Microsoft Confidential On Average, Each Month 7,300 Developers making code changes 11,000 Topic branches 367,000 Commits10 commits per minute 33,000 Pull1,100 requests pull requests per day 9,700 Branch Integrations Microsoft Confidential For 1 Day of Automated Lab Testing Customer Obsessed Production First Mindset Team Autonomy + Enterprise Alignment Shift Left Quality Infrastructure as Flexible Resource Customer Obsessed Production First Mindset Team Autonomy + Enterprise Alignment Shift Left Quality Infrastructure as Flexible Resource Quantitively & Qualitatively Customer Usage Acquisition Live Site Health Things we don’t watch Engagement Satisfaction Time to Detect Original estimate Churn Time to Communicate Completed hours Feature Usage Time to Mitigate Lines of Code Which customers affected Team capacity Pipeline Velocity Incident Prevention Items Team burndown Time to Build Aging Prevention Items Team story point velocity Time to Test SLA per Customer # of bugs found Time to Deploy Customer Support Metrics % code coverage Time to Improve Failed and flaky automation Customer Obsessed Production First Mindset Team Autonomy + Enterprise Alignment Shift Left Quality Infrastructure as Flexible Resource • • • • • • • • • • • • • • • • Assume Breach Started with war games to the learn attacks and practice response vs. Initially double-blind test Shifted left to prevent top risks Over time, eliminated blue team Credential theft Our defenders need to be our defenders Secret leakage OSS vulnerabilities Customer Obsessed Production First Mindset Team Autonomy + Enterprise Alignment Shift Left Quality Infrastructure as Flexible Resource “Let’s try to give our teams three things…. Autonomy, Mastery, Purpose” Alignment Autonomy Teams Physical team rooms Cross discipline 10-12 people Self managing Clear charter and goals Intact for 12-18 months Own features in production Own deployment of features UI API Data UI API Data Typically <20% Employee choice, not change, but 100% get Cross-pollinate talent manager driven to make a choice and micro-culture Sticky Note Exercise - Self Forming Teams Leadership is responsible for driving the big picture Sprint Plan Season Strategy 3 weeks 3 sprints 6 months 12 months 1 3 6 Teams are responsible for the detail 3 weeks Spring Fall Spring Fall Customer Obsessed Production First Mindset Team Autonomy + Enterprise Alignment Shift Left Quality Infrastructure as Flexible Resource Managing the pipeline: How do you go fast and not break things? engineers on x = # your team 5 ? 10 x 5 = 50 Rule: If your bug count exceeds your bug cap… stop working on new features until you’re back under the cap. Old way of branching Composing Isolation Mechanisms Release Service Pack Fixes Hot fixes Release Integration/Main Team A Team B Team C Feature 1 Feature 2 Release Flow Using Trunk Based Development to avoid Merge Debt Your aim won’t be perfect. Control the 1 blast radius. Customer Obsessed Production First Mindset Team Autonomy + Enterprise Alignment Shift Left Quality Infrastructure as Flexible Resource One code base with multiple delivery streams Shared abstraction layer Single master branch, multiple release branches Update 2 Azure DevOps Server Update 1 Update N Azure DevOps (SaaS) Before After • 4-6 month milestones • 3-week sprints • Horizontal teams • Vertical teams • Personal offices • Team rooms • Long planning cycles • Continual Planning & Learning • PM, Dev, Test • PM & Engineering • Yearly customer engagement • Continual customer engagement • Feature branches • Everyone in master • 20+ person teams • 8-12 person teams • Secret roadmap • Publicly shared roadmap • Bug debt accumulated • Debt paid as incurred • 100 page spec documents • Mockups in PPT • Private repositories • Inner source • Deep organizational hierarchy • Flattened organization hierarchy • Success is a measure of install numbers • User satisfaction determines success • Features shipped once a year • Features shipped every sprint .