Testing Chrome for Android

Presented by Armand Navabi on Feb. 2, 2017 to Purdue University CS 408 Testing Chrome for Android

Infrastructure Team: implement tools for developers

Chrome Test Team: Chrome on Android Design and implement Developers: implement test strategy features Infrastructure Testing Chrome on Android

Presented by Armand Navabi on Feb. 6, 2015 to Purdue University CS 40800 Lots of Developers

● > 4000 unique committers in chromium’s src.git

● ~ 1 commit every few minutes during the busy time

● If one change breaks the build, it has a huge effect on developer productivity

● How do they do it?

Google Confidential and Proprietary Outline

● The Chromium Developer’s Workflow

● Supporting Chromium Development

● Testing Chrome on Android

● Questions and Discussions

● Quiz

Google Confidential and Proprietary Chromium Developer’s Workflow

Get the Code Write Code Review Code Submit Code

Build Code Test Code Chromium Developer’s Workflow

Get the Code Write Code Review Code Submit Code

Build Code Test Code So you think you can code?

● Code reviews: First line of defense

● Continuous Integration Testing: Second (and best) line of defense ○ Pre-commit tests (i.e. commit queue, CQ) to test code that has been reviewed before it is commited ○ Post-commit tests to run a greater suite of tests on every committed change (i.e. CL) ○ We use Buildbot an open-source framework for automating build, test and release processes (also used by WebKit, Mozilla and others)

Google Confidential and Proprietary Continuous Integration Testing: Why don’t I just show you...

● The waterfall: build.chromium.org

● Tools for monitoring the build. E.g. https://sheriff-o-matic.appspot.com/

● Example Code Review and CQ: https://codereview.chromium.org/799673003/ Tests for Chrome on Android

● Main waterfall: ○ Make sure Chrome on Android builds ○ Build Chrome for Android with Clang, a compiler with useful warning and error messages: source code analysis tool to find bugs ○ Unit tests

● ASan (Address Sanitizer) ○ Fast memory error detector ○ Finds use-after-free and {heap,stack,global}-buffer overflow bugs

● Performance tests ○ Integration tests ○ Tracks performance regression ○ sunspider: JavaScript benchmark

Google Confidential and Proprietary Testing on Android: Devices

● ~ 400 bots supporting Android

● ~ 1000 devices currently in the lab (and growing fast)

● Dizzying array of device configurations ○ Tablets/phones ○ High end/mid range/low end (matters for perf) ○ Different Android versions (ICS, JB, K, L) ○ Different architectures: arm, x86, arm64, etc.

Google Confidential and Proprietary Google Confidential and Proprietary Testing Chrome on Android: Bot

Host

USB Testing Chrome on Android: Bot

● Host ○ Check out code Host ○ Build code ○ Push apks and test data (via adb) ○ Start tests on device (via adb) ○ Collect output logs (adb logcat)

● Devices ○ Provision and setup devices USB ○ Run tests Testing Chrome on Android: Bot

● Challenges: ○ adb over usb is flaky Host ○ devices are fundamentally flaky ○ stress testing devices like never before ○ pushing data to devices slow ○ limited, expensive resources

● Goals ○ Device stability (power, connectivity) USB ○ Speed (availability, cycle time) ○ Efficient resource utilization Testing Chrome on Android: Bot

● Multiple devices per host Host ○ Fault tolerance ○ Sharding: run tests in parallel across devices USB

● Sharding on perf bots ○ Same exact device types can demonstrate USB

different performance characteristics USB ○ Test affinity: pin a test to a particular device USB Resource utilization: Split builders and testers Tester Builder (Nexus4) debug-build USB

USB

USB gsutil cp USB Tester (Nexus7) gsutil upload

USB USB Google gsutil cp Storage USB USB Resource utilization: Split builders and testers

VM Mountain View Lab Tester (Nexus4) Builder USB debug-build USB USB gsutil cp USB Tester gsutil upload (Nexus7)

USB USB Google gsutil cp Storage USB USB Resource utilization: Split builders

and testers Nexus4 dbg, rel VM’s

debug-

build Google gsutil Storage gsutil cp upload Nexu7 dbg, rel release -build Questions and Discussions Quiz

1. How many developers work on chromium? a. hundreds b. thousands c. millions

2. What is Buildbot? a. a meta-checkout tool for managing source code b. a test for Chrome on Android c. a continuation integration framework used for testing

3. Why do we have multiple devices per buildbot host? a. fault tolerance, in case some devices lose adb connectivity b. to run tests faster by running different tests in parallel across devices c. both of the above

Google Confidential and Proprietary Supporting Chromium Developer’s Workflow: Get the Code

● Chromium is a project consisting of multiple repositories from different version control systems

● gclient: Meta-checkout tool managing both subversion and git checkouts ○ python script to manage workspace of modular dependencies that are each checked out independently from different subversion and git repositories ○ dependencies can be specified on a per-OS basis (target_os) ○ variables can be used to abstract concepts ○ hooks can be specified to run after checkout

● A DEPS file specifies dependencies of a project

Supporting Chromium Developer’s Workflow: Development

● No feature branch! ○ Most of the time feature doesn’t work well on the branch ○ It never works well when its merged back to master ○ Disruptive to merge features back to master

● Most of our features are developed on the master branch ○ Release branch is then forked and unworthy features are disabled ○ Major release once every 6 weeks (features that are not ready, do not go in)

● 4 release channels in order of freshness: stable, beta, developer and canary

Google Confidential and Proprietary Release Process

Stable

Beta

Developer

Canary