How to Speed up Your Development Cycle Using Modern Application Practices and Solutions

How to speed up your development cycle using modern application practices and solutions

Deploy faster on AWS Ibotta’s Digital Transformation

Jay Yeras, Partner Solutions Architect, DevOps, AWS Justin Hart, Distinguished Engineer, Ibotta October 2019

• Introductions • Building the Amazon Way • AWS Developer Tools as services • Meet Ibotta • Who we are • Our challenges, solution, and results • Lessons learned and best practices • Concluding remarks • Questions & Answers

“I am a Partner Solutions Architect at AWS. I bring over 20 years of experience in various technical roles and I am a domain expert in building highly technical cloud architectures. I am passionate about leading strategic initiatives to help customers embrace a DevOps culture.”

– Jay Yeras, Partner Solutions Architect, AWS

“I am a Distinguished Engineer at Ibotta. I am the company’s first platform developer and have seen Ibotta grow from its first lines of code to an evolving set of microservices, serving millions of users daily and worked on by over a hundred technologists. I hold associate certificates for AWS Developer and Solutions Architect. Prior to Ibotta, I worked in the internet platform space for both Photobucket and Webroot.”

– Justin Hart, Distinguished Engineer, Ibotta

Organization: Autonomous organization design that is self- sufficient

Architecture: Decoupled Culture: service- Leadership oriented (Innovation) = (O *A )^(M *C ) where principles 푓 architecture and APIs

Mechanisms: Processes, service levels, monitoring, scale, iterate

Cultural Philosophy Practices & Patterns Tools

Customer Obsession Think Big

Ownership Bias for Action

Invent and Simplify Frugality

Are Right, A Lot Earn Trust

Learn and Be Curious Dive Deep

Hire and Develop the Best Disagree and Commit

Insist on the Highest Standards Deliver Results

Developers Delivery pipeline Monolith Build 3 Build

Build Build 2 Build

Release Test 1 Build

Transformation

Before After

Examine Services APIs

Break-up activities into services

Every team will expose data only through interfaces for their services

No team will have means to communicate directly to other teams data

Use any technology you wish, but only accessible through interfaces

Design interfaces to allow for externalization

Two-Pizza Team

Builds Uses

Dev Team Microservice Customer

Dedicated product teams

ProjectManagers Product Development Product Go Product Managers Product Developers Testers Implementers Operations Backlog A • Story 1 • Story 2 • Story 3

• Story 4

- to to Market

Backlog B • Story 1 • Story 2 • Story 3 • Story 4

Product Team A Backlog C • Story 1 Product Team B • Story 2 • Story 3 Product Team C • Story 4

Development Operations

Database Security

AWS Cloud9

AWS CodeCommit

AWS CodePipeline

AWS CodeBuild

AWS CodeDeploy

Workshop workflow

Amazon EC2 Container AWS Cloud9 AWS CodeCommit Registry

Image AWS CLI

AWS CodePipeline

AWS CloudFormation

Stack AWS CodeBuild Amazon Elastic Container Service for Kubernetes

https://modernize.awsworkshop.io https://bitlylink.com/aws-modernization

● Ibotta started in 2012 in Denver, CO in the basement of a historic firehouse in Lower Downtown Denver

● We pay rewards to our users for the purchases they make of brands we promote and at retailers we support

● The app has grown to over 30 million users, paying out well over $600 million in shopping rewards from purchases at hundreds of retailers

● Working alongside the other two founding members and our iOS engineer, two of us (myself and the CTO) got to work building the platform

● From the beginning we wanted to move as fast as possible to market + ● To start quickly but grow later, we chose AWS for our launch

● We want to focus on our business and our customers, not how we get there

● We were a small team: a single service and mobile app were a quick answer for us

● We leveraged Amazon Relational Database Service (RDS) and Amazon ElastiCache to minimize ops management

● Server code managed by a script: it could launch instances and deploy

● Unit testing & continuous integration culture from the beginning

AWS Cloud ● We opted for the lightest tools that US-EAST-1 let us build our business at our API Security Backend SG DB SG scale Group ● 5 months from the first lines of code to our first real users

● 1 month later: Hurricane Sandy - would US-EAST-1 be affected? We US-WEST-2 need some Disaster Recovery API SG Backend SG DB SG ● Ibotta continued to scale up, paying out $1 million in under 1 year to our users

● The weekly shopping cycle means high traffic weekends

● “Auto” scaling used our deploy tool to launch more servers on Saturday, and we would watch CPU and request rate metrics all weekend

● When the servers are overloaded, users can’t get rewards, and may not return

● We can measure how this hurts the business with data analysis

● Along with the user base, the number of engineers, lines of code, and features continue to grow.

● Deploys become brittle, scary for new engineers. A lot of time is spent managing deployments.

● We had hit a threshold for our approach

Region ● We migrate to Chef and AWS CloudFormation to define servers and deploys, set up Autoscaling

AZ - A AZ - B AZ - C ● Allows us to scale all aspects again, and in 2016 we had paid out over $100 million to users

● We continue to add more features to the app and infrastructure, like data analytics using Amazon Redshift,

Chef Amazon EMR, Amazon Simple Master Storage Service (S3)

● Deploy has now become complex, and not as many engineers know the deploy or AWS infrastructure, so it becomes daunting. More production errors.

● We began to add microservices, a few using AWS Elastic Beanstalk to learn very fast

● Those services enabled us to move quick, but also fragmented the teams with unique infrastructure and deploy

● We hit new scale thresholds: a monolith app and adding too much unique process. There are too many team dependencies, and feature work slows. © 2019, AWS Marketplace, Amazon Web Services, Inc. or its Affiliates. Ibotta: Infrastructure Standardization and Automation

● With our new experience, we moved full speed onto Kubernetes for new services

● We built new infrastructure tools in Terraform with reusable modules

● Originally, this happened in one repository, with “apply” by hand

● Trained a group of engineers across teams as infrastructure specialists

● Set up an automation pipeline using standard CI tools to apply changes

● Hit a threshold of a ‘monolith’ of infrastructure state, slowing down changes and introducing lots of blocking errors

● Implement Atlantis, spreading the state across repositories and automating with git

● Kubernetes (and now Amazon Elastic Monolith Account Kubernetes Service EKS) standardize how microservices run API group Backend group ● Harness.io standardizes and automates how we deploy

● We build artifacts automatically from CI, and Harness performs and verifies the deployment K8S Account

● We have built a set of standard Ingress group Node group libraries for common concerns such as logging, metrics, authentication

● This “paved road” allows teams to build without needing to invent how

In 2012: Today: • 2 platform developers • 130+ engineers • 1 service • 50+ services • 1-2 deployments per day • 20+ deployments per day • 5+ months to new service • 18 minutes to deployable code • Many manual steps, commands to • 1 command to provision repository, provision and deploy infrastructure and deploy

● Every team, service, repository is single-point of autonomy for infrastructure, code, and deploy. We want them to be able to own the whole stack

● This includes machine learning leveraging Amazon Sagemaker and Data Engineering using AWS Glue. Every part of the business

● We’re working on the stack in-flight, and the monolith is still around. So treat it like a (big) service. Same deploy package, different entry points for different cases, and work towards extracting and refactoring.

● Gain trust in the tools and process so every team is autonomous and using continuous automation, to move faster

• Innovation is born of cultural and organizational transformation • Speed & ability to quickly deliver new features is critical to your business • The Amazon Way as a model for transforming your organization • Leveraging AWS services to empower and enable your teams • How Ibotta leveraged AWS services as well as AWS partner offerings from Chef and Harness.io to modernize their platform

Quick Pay-as-you-go Verified Get the software you need in Only pay for what you use through All software in AWS Marketplace minutes with just a few clicks or use various payment options and are continuously scanned to ensure the 1-Click deployment option. receive discounts on longer or reliability. custom terms. Software in AWS Marketplace are SaaS applications run on AWS and All charges from AWS Marketplace are built to scale. ready-to-run on AWS. are consolidated into one bill from AWS.

Find Buy Deploy

From a breadth Through flexible With multiple of categories: pricing options: deployment options:

Machine Learning Free trial Amazon Elastic Container Services (ECS) Containers Pay-as-you-go Amazon Elastic Container Services for Kubernetes (EKS) Networking Hourly Amazon Machine Image (AMI) Security Monthly Application Program Interface (API) Storage Annual and Multi-Year Amazon SageMaker DevOps Bring Your Own License (BYOL) AWS Fargate Database Seller Private Offers CloudFormation Template Operating Systems Consulting Partner Private Offers SaaS BI & Big Data

Agile App Infrastructure Management AppDev Server/Stacks CI/CD Testing/ITSM as Code SecOps Monitoring/AIOps v v V v v v

Custom Reports Overall Visibility, Dashboards & Reporting

Cloud Stacks Infrastructure Change Release Verification Rollback Build & Test Artifacts Provisioning Management Strategy

Basic Rolling

Code Artifact Custom Custom Custom Custom Manual Manual Scripts Scripts Scripts Scripts

Canary Blue/Green

Continuous Integration Environment Variables Custom Custom Secrets, Auditing & Pipeline Management Scripts Scripts & Compliance

Overall Visibility,Continuous Dashboards Insights & Reporting

Cloud Stacks Infrastructure Change Release Verification Rollback Build & Test Artifacts Provisioning Management Strategy

Basic Rolling

Code Artifact Smart Automation Continuous Verification

Canary Blue/Green Continuous Integration Env Variables & Secrets, Auditing Continuous Security Pipeline Mgmt & Compliance

Continuous Delivery As-A-Service

AWS Marketplace DevOps Solutions 1 https://aws.amazon.com/marketplace/b/2649274011

AWS Quick Starts 2 https://aws.amazon.com/quickstart

Modern Application Development 3 https://aws.amazon.com/modern-apps/

Modernize Workshop 4 https://modernize.awsworkshop.io/