DOCSLIB.ORG

CPSC 436I Release Engineering

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
CPSC 436I Release Engineering CPSC 436I Release Engineering by Dorothy Ordogh So much variability in release engineering, decisions are made by weighing different important aspects, which makes it hard to teach in school What on earth is release engineering? Anything to do with building, organizing, and delivering code. Why is release engineering important? It’s not, unless you rely on making money off of the software you are releasing To make money, the software you are releasing needs to: ● Always be up ● Work well Why is release engineering important? Scaling software so that it is always up, often results in LARGE codebases Best practices for maintaining and shipping a large code base become important Need to know how to: ● Structure ● Manage ● Build ● Deploy ● Host Why is release engineering important? A very large code base often means lots of people working on it Release engineering is also important for developer productivity Structure Monorepo vs Many repos ● Monorepo - one repo containing MANY projects that depend on each other ● Many repos - one repo containing one project depending on other repos and projects Directory Structure ● Depends on the tool you are building with (more about this later) Structure Benefits of a monorepo ● Removes possibility of diamond dependencies (dependency hell) ● Source dependencies makes it easier to interact with other projects in the monorepo A A B C B C D D-1.0 D-1.1 Structure Benefits of many repos ● Smaller repos ● Easier to maintain ● Tools often work better with smaller repos Structure Examples of companies with monorepos ● Google ● Twitter ● Facebook Examples of companies many repos ● Netflix ● Amazon Large companies “usually” go the monorepo route Manage Version Control To be able to edit something without losing old work and without creating multiple copies of the files you are editing Example version control systems: ● Git (distributed) ● Mercurial (distributed) ● Subversion (centralized) Manage Different version control systems work differently. Knowing how they work under the hood helps you work with them better. Git Takes a snapshot or picture of the state of the files. If there are no changes to a file, git doesn’t store the snapshot of the file Subversion Stores differences of files (delta-based version control) Manage Which version control system you use depends on what you value more ● Size of repo ● Number of contributors ● Hosting Like anything, lots of trade-offs, and there is no one correct approach or system Build When building something, you are compiling the code you wrote into binary (1s and 0s) to be run on any machine* All languages have some built in way to do this * with some caveats Build Building and running C: ● gcc command line tool compiles the C code into an executable that you run Building and running Java: ● javac command line tool compiles java code ● java command line tool runs the binary outputted from javac Build Previously mentioned build tools were language specific Language specific build tools don’t really work when we have VERY large systems we are trying to build Scaling builds up often means finding/building polyglot build tools Build Examples of polyglot build tools: ● Bazel/blaze (Google’s build system) ● Buck (Facebook’s build system) ● Pants (Twitter’s build system) Build What does a build system/tool do? ● Resolves 3rd party dependencies ● Code generation ● Compiles code ● Runs tests ● Runs compiled binaries Build How does it do it? ● Downloads 3rd party packages and makes them available to the code being compiled ● Uses a some tool to generate code ● Invokes the correct compiler for the language/languages being compiled ● Runs the binary by calling the correct language/system tool ● Runs tests by calling the correct language/system tool ie. build tools are an abstraction of language specific build tools to make developers’ lives easier Build Why do you need a build tool in the first place? ● Speed ● Ease of use ● Reduce variability of builds between machines Deploy Continuous Integration (CI) Automatic testing of changes before integrating changes into master ● Consists of successfully building master, successfully running everything in staging, and passing all unit tests and integration tests ● Goal is that master is a “clean state”, so it is running well and tests are passing This is good practice and every company should have SOME CI Deploy Continuous Deployment (CD) Automatic deployment of changes to hosts and users after going through CI ● Once the code has passed all necessary checks and is in master ● Automatically deploys changes to machines ● Deploys changes so they are consumable by users Useful reading: https://puppet.com/blog/continuous-delivery-vs-continuous-deployment-what-s-diff Deploy In practice, CI/CD is a spectrum and you do what is right for the product ● Maybe you only use CI because some manual tests need to be run before you deploy ● Maybe you don’t want users to get hundreds of updates a day (native apps) so you only deploy every two or so weeks ● Maybe you’re only deploying to servers so you can update them at anytime, so you can use CD Deploy Shipping with zero downtime ● Have two sets of hosts. Current, and with the changes (canarying) ● Usually split traffic between those two sets to slowly roll out changes ● Eventually all hosts will have new code Deploy Examples of CI/CD systems: ● Travis (integrated with GitHub) ● Jenkins (open sourced) ● AppVeyor ● GitLab CI ● CodeShip Host Underlying infrastructure of your services and systems ● Cloud - usually 3rd party ● Data Center - you manage it Host Why and when cloud? ● Third party (AWS, GCS, etc) ● Usually has elasticity and auto scales depending on traffic ● Cheaper for smaller systems because you don’t need to invest in the capital of infrastructure (building, servers, wiring, electricity, etc) Host Why and when data centers? ● When cloud doesn’t cut it anymore (disk sharing, not enough capacity, etc) ● Have more control over your own data centers and servers ● Need $$$ ● Lots of infra needs to be built for this to be effective Release engineering Questions??? Ask me anything!! About me About me Currently ● Software Engineer at Twitter on the Core Systems Libraries team (formerly the Build team) Previously ● Software Developer Co-op at Ventyx, LDRLY, & Hootsuite About me The winding path ● Started in Physics at UBC ● Moved to BA in Psychology ● Wanted to do Actuarial Sciences ● Ended up liking programming way more ● BCS at UBC About me How I got to Twitter ● Knew I wanted to work there but thought it was out of reach ● Went to the Grace Hopper Conference ● Saw a wonderful woman speak at GHC about scaling Twitter ● Picked up a card at their booth ● Applied online ● Did the interviews ● Got the job! ● Moved to SF ● That woman I saw speak at GHC is now my manager About me Sometimes things are hard ● Working while in school ● Imposter syndrome ● Being female About me At Twitter, I… ● Started on the Test Frameworks team ● Moved to the Build team to work on the Pants build tool ● Now work on core RPC libraries every service at Twitter uses ● Lead WomEng (the women in engineering group) ● Actively try to make Twitter an even better place to work Oh yeah, we’re always hiring! https://careers.twitter.com/ Important skills ● Communication ● Customer service ● Teamwork ● Research ● Reading documentation ● Problem solving ● Critical thinking Things I wish I knew when I was you ● Don’t be afraid to try ● Take time to learn new things ● School isn’t everything ● Make sure to enjoy your time with your friends ● Invest in yourself.
Load more

Recommended publications
  • I Don T Handle Protocol Git Clone Https
    I Don T Handle Protocol Git Clone Https Ne'er-do-well Rollins curtails or show-card some herbarium invectively, however combinative Tremain sibilate vernacularly or exposing. Neogene and defrayable Noach skiatron some Nepal so labially! Niall still dribble centripetally while earless Urbano tack that self-praise. Ie i clone feature branch to the central master and send anything related to the book No idiot that possibility of. You git protocol to gits fault of http server certificate to! Please use git clone name to handle merges that http revision for software stack overflow for most time? Security and cloning is. The issues I authorize are declare just reset and pull. Tight coupling of git protocol improvements to handle extremely picky and cloning the difference: understanding the internet line option only get help! Called at night after the origin had a mechanism when someone to the turnover rate of something that many contributors. Each breast should highlight a living purpose you should completely implement general purpose. DVCS _is_ a similar system update a VCS due procedure the Distributed features. It clasp the exact same error. Because Git documentation tends to focus entirely on the relationship between actually working directory but your local repository. No, I start not associated with the product in mind way. There are now world metrics that scholarship be applied to vehicle how useable a particular interface is. This batter be concern if those use although other fund the windows explorer or home you get problems with the context menu displaying incorrectly. Open source developer who created because git protocol you to handle those like best practices that http proxy by cloning from the https can be handled that? Collaboration in git protocol.
    [Show full text]
  • State of Microservices 2020 Authors Table of Patryk Mamczur Editor in Chief
    THE NEW STANDARD FOR BUILDING SCALABLE SOFTWARE? State of Microservices 2020 Authors _Table of Patryk Mamczur Editor in Chief Tomasz Czechowski contents Mateusz Mól Mariusz Nowak Design of online version Design of print version Developers 01 669 microservice experts from around the world 06 Maturity 02 Great architecture for solving scalability issues 12 Programming languages 03 JavaScript and TypeScripts rule all 20 Deployment and serverless 04 Microservice developers choose AWS 26 Repositories 05 Monorepo or multirepo? 32 Experts Varia Marek Gajda Adam Polak CTO of The Software House Head of Node.js Team 06 Communication, authorisation, message brokers 38 at The Software House Yan Cui Peter Cooper Continuous integration Host of Real World The Founder of Cooperpress 07 Microservices + CI = <3 42 Serverless Podcast Ewelina Wilkosz Thomas Boltze Debugging IT Consultant at Praqma CTO of Asto Digital 08 Are logs enough for your project? 48 Sarup Banskota Luca Mezzalira Head of Growth at ZEIT VP of Architecture at DAZN Consumption of APIs 09 Is static the future? 54 Richard Rodger CEO of Voxgig Micro frontends 10 Time for the microservice revolution on frontend? 60 Future Published in 2020 Ver. 1.0 11 Microservices – the new backend architecture standard 66 How many IT experts filled in the survey? Total answers: 669 252 163 Western Europe Eastern Europe 96 49 North America South and East Asia 23 24 Central and Middle East South America 40 22 Other Australia and 4 New Zealand 5 01 _develo- 01 pers 669 microservice experts from around the world 6 7 Microservice architecture – an architectural style where your application is based on a collection of fine-grained, interconnected services – is definitely a hot thing right now.
    [Show full text]
  • Microservices and Monorepos
    Microservices and Monorepos Match made in heaven? Sven Erik Knop, Perforce Software Overview . Microservices refresher . Microservices and versioning . What are Monorepos and why use them? . These two concepts seem to contradict – why mix them together? . The magic of narrow cloning . A match made in heaven! 2 Why Microservices? . Monolithic approach: App 3 Database Microservices approach . Individual Services 4 DB DB Database Versioning Microservices . Code . Executables and Containers . Configuration . Natural choice: individual repositories for each service Git . But: • Security • Visibility • Refactoring • Single change id to rule them all? 5 Monorepo . Why would you use a monorepo? . Who is using monorepos? . How would you use a monorepo? 6 Monorepos: Why would you do this? . Single Source of Truth for all projects . Simplified security . Configuration and Refactoring across entire application . Single change id across all projects . Examples: • Google, Facebook, Twitter, Salesforce, ... 7 Single change across projects change 314156 8 Monorepos: Antipatterns User workspace User workspace 9 Monorepos – view mapping User workspace . Map one or more services . Users only access files they need . Simplified pushing of changes 10 What does this have to do with Git? . Git does not support Monorepos • Limitations on number and size of files, history, contributing users • Companies have tried and failed . Android source spread over a thousand Git repositories • Requires repo and gerrit to work with 11 How can we square this circle? https://en.wikipedia.org/wiki/Squaring_the_circle 12 Narrow cloning! . Clone individual projects/services . Clone a group of projects into a single repo 13 Working with narrowly cloned repos . Users work normally in Git . Fetch and push changes from and to monorepo .
    [Show full text]
  • Josh Bloch Charlie Garrod
    Principles of Software Construction: Objects, Design, and Concurrency Part 4: et cetera Toward SE in practice: DevOps and branch management Josh Bloch Charlie Garrod 17-214 1 Administrivia • Homework 6 available – Checkpoint deadline tonight – Due next Wednesday, April 29th 17-214 2 Key concepts from last Thursday • SE empirical methods: Test-driven development case study • Version and release management 17-214 3 Today: Software engineering in practice • Release management, introduction to DevOps • Choose your own adventure… • Monolithic repositories 17-214 4 Consider: timelines of traditional software development e.g., the Microsoft* OS development history Source: By Paulire - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=46634740 17-214 5 Modern Facebook release cycle (1000+ diffs / day) 17-214 6 Aside: Canary testing 17-214 7 Aside: Dark launches and A/B testing • Focuses on user response to frontend changes rather than performance of backend • Measure user response via metrics: engagement, adoption 17-214 8 Version management using feature flags https://martinfowler.com/articles/feature-toggles.html https://docs.microsoft.com/en-us/azure/devops/migrate/phase-features-with-feature-flags?view=azure-devops 17-214 9 Warning! Feature flags can be dangerous Knight Capital Group realized a $460 million loss in 45-minutes, going from being the largest trader in US equities to bankruptcy. https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/ 17-214 10 Configuration management in the modern world Deployment
    [Show full text]
  • Experten-Dossier 2019
    Experten-Dossier 2019 Über 80 Seiten mit praxisorientiertem Wissen für .NET-Entwickler rund um .NET Core, Azure DevOps, TypeScript, Cosmos DB, ML.NET, Git und Azure! bastacon www.basta.net Inhalt Agile & DevOps Die DevOps-Challenge 4 DevOps-Fallstricke und wie man ihnen entkommen kann von Kevin Gerndt Microservices & APIs Warum einfach? Es geht auch komplex! 10 Entwicklung von Microservices mit Microsoft .NET von Dr. Felix Nendzig Go Git! 16 Git erobert die Entwicklerwelt von Uwe Baumann Des Kaisers neue Kleider 21 Aus VSTS wird Azure DevOps – mehr als nur ein neuer Name? von Nico Orschel und Thomas Rümmler .NET Framework & C# R. I. P .NET „Core“ 29 .NET Framework, .NET Core und Mono sind tot – lang lebe .NET 5.0! von Dr. Holger Schwichtenberg Machine Learning für die Zukunft 33 Hintergrund und Einstieg in ML mit .NET von Kevin Gerndt Architektur Kolumne: Stropek as a Service 40 Zehn Hausaufgaben für die Cloud-Architektur – Eine gute Softwarearchitektur setzt klare Ziele voraus von Rainer Stropek Große Business-Apps mit Angular meistern 43 Nachhaltige Angular-Architekturen mit Nx und Strategic Design von Manfred Steyer Inhalt Sicherheit Du kommst hier nicht rein 48 API Authorization in ASP.NET Core 3.0 mit IdentityServer von Sebastian Gingter Wasm – Ist das sicher oder kann das weg? 53 Neue Besen kehren gut, sagt man. Aber sind sie auch sicher? von Carsten Eilers HTML5 & JavaScript Das Beste aus zwei Welten 59 Mit ASP.NET Core und Angular eine Webanwendung erstellen von Fabian Gosebrink Injections für echte TypeScript-Junkies 70 Dependency
    [Show full text]
  • Git and Github
    Git and GitHub Working Dir/Staging, Local/Remote, Clone, Push, Pull, Branch/Merge, Monorepo, GitHub Desktop Source code is by far the most important asset any Even if not using GitHub for their own source, app software company owns. It is more valuable than developers still need to get familiar with it as most of buildings, brand names, computer hardware, furniture today’s popular open source projects are using it and or anything else a software company has. Source code app developers will invariably need to use these. needs to be valued and treated like the very important company asset that it is. Hence the need for a robust This course covers both and helps developers gain source code management system. hands-on experience in how to incorporate both into their development workflow. Many Git-related terms Git is the most popular source code management have entered the developer lexicon – push, pull request, system; GitHub.com is the most popular Git cloud cloning, forking, promoting, repo – and this course hosting solution. Either Git alone or Git and GitHub can helps attendees understand each concept and mentally be used to comprehensively manage and protect source. tie everything together to see how they work in unison. Contents of One-Day Training Course Distributed Version Control Command Line Tooling Using what you might already know Porcelain vs. plumbing Adding distributed influence Beyond the basics - more complete look at Organizing teams via Git advanced command line tools for Git Strategies for managing source trees Managing
    [Show full text]
  • AWS Amplify Console User Guide
    AWS Amplify Console User Guide AWS Amplify: Console User Guide Copyright © Amazon Web Services, Inc. and/or its affiliates. All rights reserved. AWS Amplify Console User Guide Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. AWS Amplify Console User Guide Table of Contents What is the AWS Amplify Console? ...................................................................................................... 1 Amplify Console features ............................................................................................................ 1 Getting started .......................................................................................................................... 2 Getting started with hosting ............................................................................................... 2 Getting started with the Admin UI ....................................................................................... 2 Modern SPA web applications ...................................................................................................... 2 Getting started .................................................................................................................................
    [Show full text]
  • Google's Single Repository
    Google’s Single Repository Zachary Crews, Inhan Park, Ayush Ganotra, Gabbie Velazquez-Pergola What a Single Repository is: - Also known as Monorepo - A means of storing assets and maintaining a codebase. - All projects created would depend and be built on top of a single codebase. - Unified Versioning - Features include: useful, enormous libraries, tons of code for reuse, and custom versioning and control. - - How does it work : - Piper / Citc source system - Allow to access from any machine - Work on local copied file - Keep receive updates from server - Update to server after receive review - Trunk based development - Prevent long-lived branch - Make changes visible immediately Why Google chose this way (background) : - Unified Version of Code - Extensive Code Sharing and Reuse - Collaboration Across Teams - Flexible Team Boundaries - Code Visibility Costs and Tradeoffs - Tooling needs to be very scalable - Cost of systems and developer tools - Increased dependencies - Less APIs - Less documentation Alternatives: Mercurial - Google is currently investigating using Mercurial, a version control system - Facebook monorepo with Mercurial - hgwatchman file monitor extension - File status - remotefilelog extension - File history and updates - Only download commit metadata - On demand file download Discussion Questions - As a developer, would you prefer working with a monorepo or multiple repos? - Can a monorepo be continued if more codes are added? - Can a Single repo be changed into a multi repo? Sources https://dl.acm.org/doi/pdf/10.1145/2854146 https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-line s-of-code-in-a-single-repository/fulltext#:~:text=A%20single%20repository%20prov ides%20unified,can%20depend%20on%20it%20directly https://www.youtube.com/watch?v=W71BTkUbdqE https://medium.com/@maoberlehner/monorepos-in-the-wild-33c6eb246cb9 https://engineering.fb.com/core-data/scaling-mercurial-at-facebook/.
    [Show full text]
  • Applying Microservice Architecture with Modern Grpc API to Scale up Large and Complex Application
    Hoang Vo Applying microservice architecture with modern gRPC API to scale up large and complex application Metropolia University of Applied Sciences Bachelor of Engineering Information Technology Bachelor’s Thesis 28th April, 2021 Abstract Author Hoang Vo Title Applying microservice architecture with modern gRPC API to scale up large and complex application NumBer of Pages 46 pages Date 28th April, 2021 Degree Bachelor of Engineering Degree Programme Information Technology Professional Major Software Engineering Instructors Janne Salonen, Head of School of ICT Department The main oBjective of this thesis is to explain how microservices and gRPC works and discuss its implementations, limitations and usages. Reliable online articles were used to support the demonstration of a microservice with gRPC servers. The complexity of modern applications has significantly increased in the past decades. Traditional applications have limitations with the monolith server architecture and the traditional HTTP/1 transportation layer. Using this information, two small projects were designed based on the microservices methodology and gRPC. The first project utilized NodeJS and ReactJS to show the usage Event Bus in micro- service communication. The second project used gRPC on top of HTTP/2 with different programming languages. Data communication and latency were two key things to con- sider in both of the projects. In conclusion, the projects demonstrates the usage of microservices and gRPC in a software application and its effectiveness in software scalability.
    [Show full text]
  • Azure Devops Sample Pipeline
    Azure Devops Sample Pipeline Fusile Elmer embalms cooingly while Sam always assibilate his garrottes imperialises remonstratingly, he ozonized so fetchingly. Light Cobb batted no self-hatred evanesces clean after Adair wending groggily, quite prescientific. Dickey still coronate nohow while unwell Rice pulsating that forger. The account in a build steps outlined above, which is having the azure devops pipeline is to the solution, professional living in Team gets more than what you run all the azure devops sample pipeline! Validate the sample markdown from the steps are other tasks for how google cloud native architecture and azure devops sample pipeline will not. Here's a short example of earth to clamp your first pipeline. First to steal gift cards using azure devops sample pipeline deploys it allows you can use it a sample stage column that enter an output on your organization? We pierce our Packer build to assure every week to greet the latest and greatest updates for the OS and whatever components we want the update. With the new name, divine was interested enough to give makeup a try. Dev branch with infrastructure solutions for this up once the experience that builds right from this will show when prompted for the. Edit it tried deploying and azure devops sample pipeline to trigger at the sample applications have our container passed to do have the. As carthage or azure devops sample pipeline menu or travis instances and modernizing legacy apps and web. Variable groups are a handy feature allowing you to share variables across pipeline and link secrets to Azure key vaults.
    [Show full text]
  • Various Projects in the Assistments Foundation
    Various Projects in the ASSISTments Foundation A Major Qualifying Project Submitted to the Faculty of WORCESTER POLYTECHNIC INSTITUTE in partial fulfillment of the requirements for the Degree of Bachelor of Science by Adam Goldsmith Date: August 9, 2019 Submitted to: Professor Neil Heffernan Worcester Polytechnic Institute Abstract This Major Qualifying Project (MQP) consisted of several sub-projects, all relating to the ASSISTments Foundation and associated projects. The first sub-project created a way to inject JavaScript code into ASSISTments’ web tutor application via the PeerAssist system for proving hints using stored Cross Site Scripting techniques to allow performing a study. The second sub-project consisted of improvements to the mobile app written in C term of 2019 during my IQP, including an unsuccessful attempt to add support for the new version of ASSISTments. The third sub-project was the successful migration of the entire codebase between version control systems, from Subversion to Git, and to new hosting on Github. i Acknowledgments I would like to thank Professor Neil Heffernan for offering the opportunity to continue work- ing on projects in the ASSISTments Foundation. Thank you to Anthony Botelho for his continued assistance with the mobile app, Cristina Heffernan for her UI advice on the mobile app, and March Patikorn for his mentoring on PeerAssist and the ASSISTments develop- ment environment. Thank you to Chris Donnelly and David Magid for their development knowledge and willingness to work through the Git migration with me. I would also like to thank all the contributors to ASSISTments for providing an environ- ment to build upon.
    [Show full text]
  • Brno University of Technology Mmo Plugin For
    BRNO UNIVERSITY OF TECHNOLOGY VYSOKÉ UČENÍ TECHNICKÉ V BRNĚ FACULTY OF INFORMATION TECHNOLOGY FAKULTA INFORMAČNÍCH TECHNOLOGIÍ DEPARTMENT OF COMPUTER GRAPHICS AND MULTIMEDIA ÚSTAV POČÍTAČOVÉ GRAFIKY A MULTIMÉDIÍ MMO PLUGIN FOR DEPLOYMENT OF MICROSERVICES INTO THE CLUSTER MODUL MMO PRO NASAZENÍ MIKROSLUŽEB DO CLUSTERU BACHELOR’S THESIS BAKALÁŘSKÁ PRÁCE AUTHOR JAKUB KULICH AUTOR PRÁCE SUPERVISOR PRof. Ing. ADAM HEROUT, PhD. VEDOUCÍ PRÁCE BRNO 2018 Abstract Orchestration of applications with microservice architecture is difficult. Available tools do not allow deployment of an application to the user without deep knowledge of target platform. Another problem is that many mistakes are done when these tools are used. MMO – Monorepo Microservice Orchestrator is a tool that makes the development of applications with microservice architecture easier. MMO can be extended with parts that can help the user to avoid problems mentioned above. One part is used for generation of the configurations for deployment of the application to Kubernetes and second part is used for deployment of the application the Kubernetes cluster. A result of using MMO extension is an ability of beginner users to create deployment configurations, faster creation of deployment configurations by advanced users and decreased number of mistakes done when the application is deployed manually by Kubernetes users. Abstrakt Aplikácie s architektúrou typu „mikroslužby“ je náročné orchestrovať. Dostupné nástroje neumožňujú užívateľom nasadenie aplikácie bez veľkej znalosti cieľovej platformy. Ďalším problémom je, že užívatelia pri nepozornosti robia chyby pri používaní týchto nástrojov. Vývoj aplikácií s architektúrou typu „mikroslužby“ zjednodušuje open-source nástroj MMO – Monorepo Microservice Orchestrator. Rozšírením tohto nástroja o určité časti nám umožňuje vyhnúť sa spomínaným problémom.
    [Show full text]