Azure DevOps
Randy Pagels Azure Specialist - Application Development US Great Lakes Region How Microsoft Engineering uses Azure DevOps
MS Engineering uses 1ES to build Azure DevOps, Windows 10, Visual Studio, Bing…etc. …using Azure DevOps …building 146,000 times per day …deploying 82,000 times per day … for millions of users + most of Microsoft. Microsoft’s DevOps Tooling – enhanced by GitHub
Security
Package Registry
Actions The journey so far Sprint 162 Nov 2019
VSTS Preview 1ES Windows on Git Azure Pipelines Sprint 1 Sprint 29 Sprint 67 Sprint 102 Sprint 140 August 2010 June 2012 June 2014 May 2017 Sep 2018
400 Open Source Projects Microsoft on GitHub VS Code Join Linux GitHub March 2010 July 2014 April 2015 Foundation Acquired Nov 2016 Oct 2018
8.8k open source projects 25k employees contributing Before Azure DevOps Microsoft Confidential One Engineering System (”1ES”) One Engineering System with Azure DevOps
Microsoft Confidential 1ES Growth 100,000
80,000
Non-engineering Users 60,000
40,000
Engineer Users 20,000
0 DevOps at Microsoft Azure DevOps is the toolchain of choice for Microsoft engineering with over 110,000 active internal users
➔ https://aka.ms/DevOpsAtMicrosoft
4.6m 28k 442k Builds per month Work items created/day Pull Requests per 155pb 500k month Build artifacts managed Work items updated/day
2.4m Private Git commits per 8.8k 24k 82,000 month Open Source Repos Employees contributing to open source Deployments per day
Data: Internal Microsoft engineering system activity, November 2019 One Engineering System with Azure DevOps Azure Networking Microsoft Confidential
Software Products
500GB+ 6000+ Source code Repos
Microsoft Confidential On Average, Each Month
7,300 Developers making code changes 11,000 Topic branches 367,000 Commits10 commits per minute 33,000 Pull1,100 requests pull requests per day 9,700 Branch Integrations
Microsoft Confidential For 1 Day of Automated Lab Testing Customer Obsessed
Production First Mindset
Team Autonomy + Enterprise Alignment
Shift Left Quality
Infrastructure as Flexible Resource Customer Obsessed
Production First Mindset
Team Autonomy + Enterprise Alignment
Shift Left Quality
Infrastructure as Flexible Resource Quantitively & Qualitatively
Customer Usage
Acquisition Live Site Health Things we don’t watch Engagement Satisfaction Time to Detect Original estimate Churn Time to Communicate Completed hours Feature Usage Time to Mitigate Lines of Code Which customers affected Team capacity Pipeline Velocity Incident Prevention Items Team burndown Time to Build Aging Prevention Items Team story point velocity Time to Test SLA per Customer # of bugs found Time to Deploy Customer Support Metrics % code coverage Time to Improve Failed and flaky automation
Customer Obsessed
Production First Mindset
Team Autonomy + Enterprise Alignment
Shift Left Quality
Infrastructure as Flexible Resource •
• •
•
• •
• • • • • • • •
•
•
Assume Breach Started with war games to the learn attacks and practice response
vs.
Initially double-blind test Shifted left to prevent top risks Over time, eliminated blue team Credential theft Our defenders need to be our defenders Secret leakage OSS vulnerabilities Customer Obsessed
Production First Mindset
Team Autonomy + Enterprise Alignment
Shift Left Quality
Infrastructure as Flexible Resource “Let’s try to give our teams three things…. Autonomy, Mastery, Purpose”
Alignment
Autonomy
Teams
Physical team rooms Cross discipline 10-12 people Self managing Clear charter and goals Intact for 12-18 months Own features in production Own deployment of features UI
API
Data UI
API
Data Typically <20% Employee choice, not change, but 100% get Cross-pollinate talent manager driven to make a choice and micro-culture
Sticky Note Exercise - Self Forming Teams Leadership is responsible for driving the big picture
Sprint Plan Season Strategy 3 weeks 3 sprints 6 months 12 months 1 3 6
Teams are responsible for the detail 3 weeks
Spring Fall Spring Fall Customer Obsessed
Production First Mindset
Team Autonomy + Enterprise Alignment
Shift Left Quality
Infrastructure as Flexible Resource Managing the pipeline:
How do you go fast and not break things? engineers on x = # your team 5 ? 10 x 5 = 50
Rule: If your bug count exceeds your bug cap… stop working on new features until you’re back under the cap. Old way of branching Composing Isolation Mechanisms
Release Service Pack Fixes Hot fixes
Release
Integration/Main
Team A
Team B
Team C
Feature 1 Feature 2 Release Flow Using Trunk Based Development to avoid Merge Debt
Your aim won’t be perfect.
Control the 1 blast radius.
Customer Obsessed
Production First Mindset
Team Autonomy + Enterprise Alignment
Shift Left Quality
Infrastructure as Flexible Resource
One code base with multiple delivery streams Shared abstraction layer Single master branch, multiple release branches
Update 2 Azure DevOps Server Update 1 Update N
Azure DevOps (SaaS) Before After • 4-6 month milestones • 3-week sprints • Horizontal teams • Vertical teams • Personal offices • Team rooms • Long planning cycles • Continual Planning & Learning • PM, Dev, Test • PM & Engineering • Yearly customer engagement • Continual customer engagement • Feature branches • Everyone in master • 20+ person teams • 8-12 person teams • Secret roadmap • Publicly shared roadmap • Bug debt accumulated • Debt paid as incurred • 100 page spec documents • Mockups in PPT • Private repositories • Inner source • Deep organizational hierarchy • Flattened organization hierarchy • Success is a measure of install numbers • User satisfaction determines success • Features shipped once a year • Features shipped every sprint