CPSC 436I Release Engineering by Dorothy Ordogh So much variability in release engineering, decisions are made by weighing different important aspects, which makes it hard to teach in school What on earth is release engineering? Anything to do with building, organizing, and delivering code. Why is release engineering important?
It’s not, unless you rely on making money off of the software you are releasing
To make money, the software you are releasing needs to:
● Always be up ● Work well Why is release engineering important?
Scaling software so that it is always up, often results in LARGE codebases
Best practices for maintaining and shipping a large code base become important
Need to know how to:
● Structure ● Manage ● Build ● Deploy ● Host Why is release engineering important?
A very large code base often means lots of people working on it
Release engineering is also important for developer productivity Structure
Monorepo vs Many repos
● Monorepo - one repo containing MANY projects that depend on each other ● Many repos - one repo containing one project depending on other repos and projects
Directory Structure
● Depends on the tool you are building with (more about this later) Structure
Benefits of a monorepo
● Removes possibility of diamond dependencies (dependency hell) ● Source dependencies makes it easier to interact with other projects in the monorepo
A A
B C B C
D D-1.0 D-1.1 Structure
Benefits of many repos
● Smaller repos ● Easier to maintain ● Tools often work better with smaller repos Structure
Examples of companies with monorepos
Examples of companies many repos
● Netflix ● Amazon
Large companies “usually” go the monorepo route Manage
Version Control
To be able to edit something without losing old work and without creating multiple copies of the files you are editing
Example version control systems:
● Git (distributed) ● Mercurial (distributed) ● Subversion (centralized) Manage
Different version control systems work differently. Knowing how they work under the hood helps you work with them better.
Git
Takes a snapshot or picture of the state of the files. If there are no changes to a file, git doesn’t store the snapshot of the file
Subversion
Stores differences of files (delta-based version control) Manage
Which version control system you use depends on what you value more
● Size of repo ● Number of contributors ● Hosting
Like anything, lots of trade-offs, and there is no one correct approach or system Build
When building something, you are compiling the code you wrote into binary (1s and 0s) to be run on any machine*
All languages have some built in way to do this
* with some caveats Build
Building and running C:
● gcc command line tool compiles the C code into an executable that you run
Building and running Java:
● javac command line tool compiles java code ● java command line tool runs the binary outputted from javac Build
Previously mentioned build tools were language specific
Language specific build tools don’t really work when we have VERY large systems we are trying to build
Scaling builds up often means finding/building polyglot build tools Build
Examples of polyglot build tools:
● Bazel/blaze (Google’s build system) ● Buck (Facebook’s build system) ● Pants (Twitter’s build system) Build
What does a build system/tool do?
● Resolves 3rd party dependencies ● Code generation ● Compiles code ● Runs tests ● Runs compiled binaries Build
How does it do it?
● Downloads 3rd party packages and makes them available to the code being compiled ● Uses a some tool to generate code ● Invokes the correct compiler for the language/languages being compiled ● Runs the binary by calling the correct language/system tool ● Runs tests by calling the correct language/system tool ie. build tools are an abstraction of language specific build tools to make developers’ lives easier Build
Why do you need a build tool in the first place?
● Speed ● Ease of use ● Reduce variability of builds between machines Deploy
Continuous Integration (CI)
Automatic testing of changes before integrating changes into master
● Consists of successfully building master, successfully running everything in staging, and passing all unit tests and integration tests ● Goal is that master is a “clean state”, so it is running well and tests are passing
This is good practice and every company should have SOME CI Deploy
Continuous Deployment (CD)
Automatic deployment of changes to hosts and users after going through CI
● Once the code has passed all necessary checks and is in master ● Automatically deploys changes to machines ● Deploys changes so they are consumable by users
Useful reading: https://puppet.com/blog/continuous-delivery-vs-continuous-deployment-what-s-diff Deploy
In practice, CI/CD is a spectrum and you do what is right for the product
● Maybe you only use CI because some manual tests need to be run before you deploy ● Maybe you don’t want users to get hundreds of updates a day (native apps) so you only deploy every two or so weeks ● Maybe you’re only deploying to servers so you can update them at anytime, so you can use CD Deploy
Shipping with zero downtime
● Have two sets of hosts. Current, and with the changes (canarying) ● Usually split traffic between those two sets to slowly roll out changes ● Eventually all hosts will have new code Deploy
Examples of CI/CD systems:
● Travis (integrated with GitHub) ● Jenkins (open sourced) ● AppVeyor ● GitLab CI ● CodeShip Host
Underlying infrastructure of your services and systems
● Cloud - usually 3rd party ● Data Center - you manage it Host
Why and when cloud?
● Third party (AWS, GCS, etc) ● Usually has elasticity and auto scales depending on traffic ● Cheaper for smaller systems because you don’t need to invest in the capital of infrastructure (building, servers, wiring, electricity, etc) Host
Why and when data centers?
● When cloud doesn’t cut it anymore (disk sharing, not enough capacity, etc) ● Have more control over your own data centers and servers ● Need $$$ ● Lots of infra needs to be built for this to be effective Release engineering
Questions??? Ask me anything!! About me About me
Currently
● Software Engineer at Twitter on the Core Systems Libraries team (formerly the Build team)
Previously
● Software Developer Co-op at Ventyx, LDRLY, & Hootsuite About me
The winding path
● Started in Physics at UBC ● Moved to BA in Psychology ● Wanted to do Actuarial Sciences ● Ended up liking programming way more ● BCS at UBC About me
How I got to Twitter
● Knew I wanted to work there but thought it was out of reach ● Went to the Grace Hopper Conference ● Saw a wonderful woman speak at GHC about scaling Twitter ● Picked up a card at their booth ● Applied online ● Did the interviews ● Got the job! ● Moved to SF ● That woman I saw speak at GHC is now my manager About me
Sometimes things are hard
● Working while in school ● Imposter syndrome ● Being female About me
At Twitter, I…
● Started on the Test Frameworks team ● Moved to the Build team to work on the Pants build tool ● Now work on core RPC libraries every service at Twitter uses ● Lead WomEng (the women in engineering group) ● Actively try to make Twitter an even better place to work
Oh yeah, we’re always hiring! https://careers.twitter.com/ Important skills
● Communication ● Customer service ● Teamwork ● Research ● Reading documentation ● Problem solving ● Critical thinking Things I wish I knew when I was you
● Don’t be afraid to try ● Take time to learn new things ● School isn’t everything ● Make sure to enjoy your time with your friends ● Invest in yourself