DOCSLIB.ORG

Testing Chrome for Android

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Testing Chrome for Android Testing Chrome for Android Presented by Armand Navabi on Feb. 2, 2017 to Purdue University CS 408 Testing Chrome for Android Infrastructure Team: implement tools for developers Chrome Test Team: Chrome on Android Design and implement Developers: implement test strategy features Infrastructure Testing Chrome on Android Presented by Armand Navabi on Feb. 6, 2015 to Purdue University CS 40800 Lots of Developers ● > 4000 unique committers in chromium’s src.git ● ~ 1 commit every few minutes during the busy time ● If one change breaks the build, it has a huge effect on developer productivity ● How do they do it? Google Confidential and Proprietary Outline ● The Chromium Developer’s Workflow ● Supporting Chromium Development ● Testing Chrome on Android ● Questions and Discussions ● Quiz Google Confidential and Proprietary Chromium Developer’s Workflow Get the Code Write Code Review Code Submit Code Build Code Test Code Chromium Developer’s Workflow Get the Code Write Code Review Code Submit Code Build Code Test Code So you think you can code? ● Code reviews: First line of defense ● Continuous Integration Testing: Second (and best) line of defense ○ Pre-commit tests (i.e. commit queue, CQ) to test code that has been reviewed before it is commited ○ Post-commit tests to run a greater suite of tests on every committed change (i.e. CL) ○ We use Buildbot an open-source framework for automating build, test and release processes (also used by WebKit, Mozilla and others) Google Confidential and Proprietary Continuous Integration Testing: Why don’t I just show you... ● The waterfall: build.chromium.org ● Tools for monitoring the build. E.g. https://sheriff-o-matic.appspot.com/ ● Example Code Review and CQ: https://codereview.chromium.org/799673003/ Tests for Chrome on Android ● Main waterfall: ○ Make sure Chrome on Android builds ○ Build Chrome for Android with Clang, a compiler with useful warning and error messages: source code analysis tool to find bugs ○ Unit tests ● ASan (Address Sanitizer) ○ Fast memory error detector ○ Finds use-after-free and {heap,stack,global}-buffer overflow bugs ● Performance tests ○ Integration tests ○ Tracks performance regression ○ sunspider: JavaScript benchmark Google Confidential and Proprietary Testing on Android: Devices ● ~ 400 bots supporting Android ● ~ 1000 devices currently in the lab (and growing fast) ● Dizzying array of device configurations ○ Tablets/phones ○ High end/mid range/low end (matters for perf) ○ Different Android versions (ICS, JB, K, L) ○ Different architectures: arm, x86, arm64, etc. Google Confidential and Proprietary Google Confidential and Proprietary Testing Chrome on Android: Bot Host USB Testing Chrome on Android: Bot ● Host ○ Check out code Host ○ Build code ○ Push apks and test data (via adb) ○ Start tests on device (via adb) ○ Collect output logs (adb logcat) ● Devices ○ Provision and setup devices USB ○ Run tests Testing Chrome on Android: Bot ● Challenges: ○ adb over usb is flaky Host ○ devices are fundamentally flaky ○ stress testing devices like never before ○ pushing data to devices slow ○ limited, expensive resources ● Goals ○ Device stability (power, connectivity) USB ○ Speed (availability, cycle time) ○ Efficient resource utilization Testing Chrome on Android: Bot ● Multiple devices per host Host ○ Fault tolerance ○ Sharding: run tests in parallel across devices USB ● Sharding on perf bots ○ Same exact device types can demonstrate USB different performance characteristics USB ○ Test affinity: pin a test to a particular device USB Resource utilization: Split builders and testers Tester Builder (Nexus4) debug-build USB USB USB gsutil cp USB Tester (Nexus7) gsutil upload USB USB Google gsutil cp USB Storage USB Resource utilization: Split builders and testers VM Mountain View Lab Tester (Nexus4) Builder USB debug-build USB USB gsutil cp USB Tester gsutil upload (Nexus7) USB USB Google gsutil cp Storage USB USB Resource utilization: Split builders and testers Nexus4 dbg, rel VM’s debug- build Google gsutil Storage gsutil cp upload Nexu7 dbg, rel release -build Questions and Discussions Quiz 1. How many developers work on chromium? a. hundreds b. thousands c. millions 2. What is Buildbot? a. a meta-checkout tool for managing source code b. a test for Chrome on Android c. a continuation integration framework used for testing 3. Why do we have multiple devices per buildbot host? a. fault tolerance, in case some devices lose adb connectivity b. to run tests faster by running different tests in parallel across devices c. both of the above Google Confidential and Proprietary Supporting Chromium Developer’s Workflow: Get the Code ● Chromium is a project consisting of multiple repositories from different version control systems ● gclient: Meta-checkout tool managing both subversion and git checkouts ○ python script to manage workspace of modular dependencies that are each checked out independently from different subversion and git repositories ○ dependencies can be specified on a per-OS basis (target_os) ○ variables can be used to abstract concepts ○ hooks can be specified to run after checkout ● A DEPS file specifies dependencies of a project Supporting Chromium Developer’s Workflow: Development ● No feature branch! ○ Most of the time feature doesn’t work well on the branch ○ It never works well when its merged back to master ○ Disruptive to merge features back to master ● Most of our features are developed on the master branch ○ Release branch is then forked and unworthy features are disabled ○ Major release once every 6 weeks (features that are not ready, do not go in) ● 4 release channels in order of freshness: stable, beta, developer and canary Google Confidential and Proprietary Release Process Stable Beta Developer Canary.
Load more

Recommended publications
  • Buildbot: a Continuous Integration System
    Buildbot: A continuous integration system Krzysztof Voss January, 2013 Outline • Testing and Continuous Integration • Introduction to Buildbot • BuildMaster • BuildMaster components • BuildSlave • Installation and Usage 1 Testing and continuous integration Tests: • the best specification • safety-net for refactoring • bug identification Tests are the most effective if we: • run them often • run them on different machines/environments • can easily see their results 2 The most straightforward approach would entail: • logging in to different machines • fetching the newest source code • running tests • analyzing their output In case we want to test a few environments, repeating the above steps is tedious. Developers do not focus on the code, instead they run tests. A continuous integration system performs all of these steps for us, so developers can focus on their code. 3 Introduction to Buildbot: Features • run builds on a variety of BuildSlave platforms • arbitrary build process: handles projects using C, Python, . • minimal host requirements: python and Twisted • BuildSlave can be behind a firewall if they can still do checkout • status delivery through web page, email, IRC, other protocols • track builds in progress, provide estimated completion time • flexible configuration by subclassing generic build process classes 4 • debug tools to force a new build, submit fake Changes, query BuildSlave status • released under the GPL source: http://buildbot.net/buildbot/docs/current/manual/introduction.html 5 Introduction to Buildbot: Overview system overview source: http://buildbot.net/buildbot/docs/0.8.1/full.html 6 BuildMaster BuildMaster components source: http://buildbot.net/buildbot/docs/0.8.1/full.html 7 BuildMaster BuildMaster: • holds the configuration of the entire system.
    [Show full text]
  • Analysis of Devops Tools to Predict an Optimized Pipeline by Adding Weightage for Parameters
    International Journal of Computer Applications (0975 – 8887) Volume 181 – No. 33, December 2018 Analysis of DevOps Tools to Predict an Optimized Pipeline by Adding Weightage for Parameters R. Vaasanthi V. Prasanna Kumari, PhD S. Philip Kingston Research Scholar, HOD, MCA Project Manager SCSVMV University Rajalakshmi Engineering Infosys, Mahindra City, Kanchipuram College, Chennai Chennai ABSTRACT cloud. Now-a-days more than ever, DevOps [Development + Operations] has gained a tremendous amount of attention in 2. SCM software industry. Selecting the tools for building the DevOps Source code management (SCM) is a software tool used for pipeline is not a trivial exercise as there are plethora’s of tools development, versioning and enables team working in available in market. It requires thought, planning, and multiple locations to work together more effectively. This preferably enough time to investigate and consult other plays a vital role in increasing team’s productivity. Some of people. Unfortunately, there isn’t enough time in the day to the SCM tools, considered for this study are GIT, SVN, CVS, dig for top-rated DevOps tools and its compatibility with ClearCase, Mercurial, TFS, Monotone, Bitkeeper, Code co- other tools. Each tool has its own pros/cons and compatibility op, Darcs, Endevor, Fossil, Perforce, Rational Synergy, of integrating with other tools. The objective of this paper is Source Safe, and GNU Bazaar. Table1 consists of SCM tools to propose an approach by adding weightage to each with weightage. parameter for the curated list of the DevOps tools. 3. BUILD Keywords Build is a process that enables source code to be automatically DevOps, SCM, dependencies, compatibility and pipeline compiled into binaries including code level unit testing to ensure individual pieces of code behave as expected [4].
    [Show full text]
  • Studying Routing Issues in Vanets by Using NS-3 Bachelor Thesis on Informatics
    Studying Routing Issues in VANETs by Using NS-3 Bachelor Thesis on Informatics by Christos Profentzas Thesis supervisor Dr. Periklis Chatzimisios Alexander Technological Educational Institute of Thessaloniki Department of Informatics A.T.E.I. of Thessaloniki P.O. Box 141 GR -547 00 Thessaloniki, Macedonia, Greece November 2012 i Acknowledgements This research project would not have been possible without the sup- port of many people. The author wishes to express his gratitude to his supervisor, Assistant Professor Periklis Chatzimisios (Alexan- der TEI of Thessaloniki, Greece) and Assistant Professor Gennaro Boggia (Politecnico di Bari, Italy) who was abundantly helpful and offered invaluable assistance, support and guidance. Deepest grati- tude are also due to the members of the supervisory committee, Assis- tant Professor Luigi Alfredo Grieco and Ph.D Student Giuseppe Piro without whose knowledge and assistance this study would not have been successful. Special thanks also to all group members of Telematics Lab at the Electrical & Electronics Engineering Depart- ment of Politecnico di Bari, for sharing the literature, invaluable assis- tance and laboratory facilities. The author would also like to convey thanks to the Office of Erasmus Program and Faculty of Alexander Technological Educational Institution of Thessaloniki for providing the financial means. Abstract A Vehicular Ad-hoc Network (VANET) is a system of nodes (vehi- cles) that are being connected with each other by wireless technolo- gies. Usually the nodes are moving with very high speeds and, thus, the topology is unpredictable and frequently changing. Such networks can be stand alone and making paths along vehicles or may be con- nected by an infrastructure internet.
    [Show full text]
  • This Book Doesn't Tell You How to Write Faster Code, Or How to Write Code with Fewer Memory Leaks, Or Even How to Debug Code at All
    Practical Development Environments By Matthew B. Doar ............................................... Publisher: O'Reilly Pub Date: September 2005 ISBN: 0-596-00796-5 Pages: 328 Table of Contents | Index This book doesn't tell you how to write faster code, or how to write code with fewer memory leaks, or even how to debug code at all. What it does tell you is how to build your product in better ways, how to keep track of the code that you write, and how to track the bugs in your code. Plus some more things you'll wish you had known before starting a project. Practical Development Environments is a guide, a collection of advice about real development environments for small to medium-sized projects and groups. Each of the chapters considers a different kind of tool - tools for tracking versions of files, build tools, testing tools, bug-tracking tools, tools for creating documentation, and tools for creating packaged releases. Each chapter discusses what you should look for in that kind of tool and what to avoid, and also describes some good ideas, bad ideas, and annoying experiences for each area. Specific instances of each type of tool are described in enough detail so that you can decide which ones you want to investigate further. Developers want to write code, not maintain makefiles. Writers want to write content instead of manage templates. IT provides machines, but doesn't have time to maintain all the different tools. Managers want the product to move smoothly from development to release, and are interested in tools to help this happen more often.
    [Show full text]
  • Expert Python Programming Third Edition
    Expert Python Programming Third Edition Become a master in Python by learning coding best practices and advanced programming concepts in Python 3.7 Michał Jaworski Tarek Ziadé BIRMINGHAM - MUMBAI Expert Python Programming Third Edition Copyright © 2019 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Kunal Chaudhari Acquisition Editor: Chaitanya Nair Content Development Editor: Zeeyan Pinheiro Technical Editor: Ketan Kamble Copy Editor: Safis Editing Project Coordinator: Vaidehi Sawant Proofreader: Safis Editing Indexer: Priyanka Dhadke Graphics: Alishon Mendonsa Production Coordinator: Shraddha Falebhai First published: September 2008 Second edition: May 2016 Third edition: April 2019 Production reference: 1270419 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78980-889-6 www.packtpub.com To my beloved wife, Oliwia, for her love, inspiration, and her endless patience.
    [Show full text]
  • Design and Implementation of a Continuous Integration System
    Prakash Acharya DESIGN AND IMPLEMENTATION OF A CONTINUOUS INTEGRATION SYSTEM DESIGN AND IMPLEMENTATION OF A CONTINUOUS INTEGRATION SYSTEM Prakash Acharya Bachelor’s Thesis Spring 2019 Information Technology Oulu University of Applied Sciences ABSTRACT Oulu University of Applied Sciences Information Technology, Internet Services Author: Prakash Acharya Title of the bachelor’s thesis: Design and Implementation of a Continuous Integration System Supervisor: Lasse Haverinen Term and year of completion: Spring 2019 Number of pages: 43 Not having an automated test system in the software development process leaves space for errors to go unnoticed in code review, which can break a codebase if integrated. The aim of the Bachelor’s thesis was to design and implement a Continuous Integration system where tests and build tasks could be automated. The Buildbot framework was used as a Continuous Integration and automation framework. The Ansible playbook was used for automating the deployment of Buildbot configuration. The Buildbot configuration was created in such a way that it allows Buildbot instances to be created easily and in a configurable manner. In addition, automation of the Python static code analysis tool, Prospector, was added to the Continuous Integration system. The result of having a configurable Buildbot system was that the developers could start adding automation tasks and test and verify their change without affecting the production instance. Having an automated static code analysis run on changes brings into attention potential problems and error before they are integrated. Even though the Buildbot system is not taken into use, it was tested in a test environment and it is shown to work.
    [Show full text]
  • Driving and Virtualizing Control Systems: the Open Source Approach Used in Whiterabbit
    Driving and virtualizing control systems: the Open Source approach used in WhiteRabbit Javier Muñoz Mellid & Samuel Iglesias Gonsalvez Academia-Industry Matching event on Technology of Controls for Accelerators and Detectors November 2013 Agenda Knowing Igalia The value of the Open Source approach Open Source approach in WhiteRabbit A walk on technical details and demo Driving and virtualizing control systems: the Open Source approach used in WhiteRabbit Javier Muñoz Mellid & Samuel Iglesias Gonsalvez What is Igalia? Open Source Company 40 engineers and hackers working around the globe Hacking upstream in different technologies and communities kernel (Linux) virtualization (QEMU/KVM) browsers/multimedia (WebKit, Blink, GStreamer...) compilers (V8, JavaScriptCore...) UI (GTK+,...) documents (Evince, LibreOffice...) distros (Debian, Tizen...) automotive/IVI 13 years old now! www.igalia.com Driving and virtualizing control systems: the Open Source approach used in WhiteRabbit Javier Muñoz Mellid & Samuel Iglesias Gonsalvez Partnering Linux Foundation www.linuxfoundation.org/news- media/announcements/2011/04/igalia-joins-linux- foundation W3C www.igalia.com/nc/igalia-247/news/item/igalia-joins- the-world-wide-web-consortium Tizen Association www.igalia.com/nc/igalia-247/news/item/igalia-joins- the-tizen-association-partner-program/ ... Driving and virtualizing control systems: the Open Source approach used in WhiteRabbit Javier Muñoz Mellid & Samuel Iglesias Gonsalvez What services and solutions provides Igalia? We are experts in Open Source
    [Show full text]
  • Splash Documentation Release 3.5
    Splash Documentation Release 3.5 Scrapinghub Jun 16, 2020 Contents 1 Documentation 3 1.1 Installation................................................3 1.2 Splash HTTP API............................................6 1.3 Splash Scripts Tutorial.......................................... 18 1.4 Splash Lua API Overview........................................ 24 1.5 Splash Scripts Reference......................................... 26 1.6 Response Object............................................. 67 1.7 Request Object.............................................. 69 1.8 Element Object.............................................. 71 1.9 Working with Binary Data........................................ 87 1.10 Available Lua Libraries......................................... 88 1.11 Splash and Jupyter............................................ 95 1.12 FAQ.................................................... 97 1.13 Contributing to Splash.......................................... 102 1.14 Implementation Details......................................... 103 1.15 Changes................................................. 105 i ii Splash Documentation, Release 3.5 Splash is a javascript rendering service. It’s a lightweight web browser with an HTTP API, implemented in Python 3 using Twisted and QT5. The (twisted) QT reactor is used to make the service fully asynchronous allowing to take advantage of webkit concurrency via QT main loop. Some of Splash features: • process multiple webpages in parallel; • get HTML results and/or take screenshots; • turn
    [Show full text]
  • Buildbot Documentation Release 0.9.15
    Buildbot Documentation Release 0.9.15 Brian Warner Jan 02, 2018 Contents 1 Buildbot Tutorial 3 1.1 First Run.................................................3 1.2 First Buildbot run with Docker......................................6 1.3 A Quick Tour...............................................9 1.4 Further Reading............................................. 17 2 Buildbot Manual 23 2.1 Introduction............................................... 23 2.2 Installation................................................ 29 2.3 Concepts................................................. 45 2.4 Secret Management........................................... 56 2.5 Configuration............................................... 58 2.6 Transition to “worker” terminology................................... 245 2.7 Customization.............................................. 250 2.8 New-Style Build Steps.......................................... 275 2.9 Command-line Tool........................................... 278 2.10 Resources................................................. 289 2.11 Optimization............................................... 289 2.12 Plugin Infrastructure in Buildbot..................................... 289 2.13 Deployment............................................... 290 3 Buildbot Development 293 3.1 General Documents........................................... 293 3.2 APIs................................................... 377 3.3 Python3 compatibility.......................................... 470 3.4 Classes.................................................
    [Show full text]
  • Automating Build, Test, and Release with Buildbot
    Automating Build, Test, and Release with Buildbot Dustin J. Mitchell Mozilla, Inc. Email: [email protected] Tom Prince Email: [email protected] Abstract—Buildbot is a mature framework for building con- to support a variety of deployments. Schedulers react to new tinuous integration systems which supports parallel execution of data from change sources, external events, or triggers based jobs across multiple platforms, flexible integration with version- on clock time (e.g., for a nightly build), and add new build control systems, extensive status reporting, and more. Beyond the capabilities it shares with similar tools, Buildbot has a number requests to the queue. of unique strengths and capabilities, some of them particu- Each build request comes with a set of source stamps larly geared toward release processes. This paper summarizes identifying the code from each codebase used for a build. A Buildbot’s structure, contrasts the framework with related tools, source stamp represents a particular revision and branch in a describes some advanced configurations, and highlights ongoing source code repository. Build requests also specify a builder improvements to the framework.1 that should perform the task and a set of properties, arbitrary I. INTRODUCTION key-value pairs giving further detail about the requested build. Buildbot is a continuous integration (CI) framework which Each builder defines the steps to take in executing a particular has been in use for well over 10 years[13]. While it began type of build. Once the request is selected for execution, the as a simple build-and-test tool, it has grown into an advanced steps are performed sequentially.
    [Show full text]
  • Mining Software Repositories for Release Engineers - Empirical Studies on Integration and Infrastructure-As-Code
    UNIVERSITÉ DE MONTRÉAL MINING SOFTWARE REPOSITORIES FOR RELEASE ENGINEERS - EMPIRICAL STUDIES ON INTEGRATION AND INFRASTRUCTURE-AS-CODE YUJUAN JIANG DÉPARTEMENT DE GÉNIE INFORMATIQUE ET GÉNIE LOGICIEL ÉCOLE POLYTECHNIQUE DE MONTRÉAL THÈSE PRÉSENTÉE EN VUE DE L’OBTENTION DU DIPLÔME DE PHILOSOPHIÆ DOCTOR (GÉNIE INFORMATIQUE) AOÛT 2016 c Yujuan Jiang, 2016. UNIVERSITÉ DE MONTRÉAL ÉCOLE POLYTECHNIQUE DE MONTRÉAL Cette thèse intitulée: MINING SOFTWARE REPOSITORIES FOR RELEASE ENGINEERS - EMPIRICAL STUDIES ON INTEGRATION AND INFRASTRUCTURE-AS-CODE présentée par: JIANG Yujuan en vue de l’obtention du diplôme de: Philosophiæ Doctor a été dûment acceptée par le jury d’examen constitué de: M. GAGNON Michel, Ph. D., président M. ADAMS Bram, Doctorat, membre et directeur de recherche M. GUÉHÉNEUC Yann-Gaël, Doctorat, membre Mme BAYSAL Olga, Ph. D., membre externe iii DEDICATION To my grandpa Who has become a star in the heaven iv ACKNOWLEDGMENTS Many people I met, many things happened, are hidden as part of this thesis. First of all, I want to thank my parents, for their endless and unconditional love. You respect my every single choice and impractical dream. You always hold the faith in me, even when I myself had lost it. Exclusive thanks are dedicated to my supervisor, Dr.Bram Adams. Humble, enthusiastic and hard-working, for work and for life, you show me what should be like as a researcher and an individual person. You lead me through all the way up here, with enormous patience and kindness. In past four years, what I gained was not only professional skills, but also the will to face new challenges, the courage of learning from failures and the belief in hard working.
    [Show full text]
  • Buildbot This Is the Buildbot Manual
    BuildBot This is the BuildBot manual. Copyright (C) 2005, 2006, 2009, 2010 Brian Warner Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. i Table of Contents 1 Introduction..................................... 1 1.1 History and Philosophy ........................................ 1 1.2 System Architecture............................................ 2 1.2.1 BuildSlave Connections.................................... 3 1.2.2 Buildmaster Architecture .................................. 4 1.2.3 Status Delivery Architecture............................... 6 1.3 Control Flow ................................................... 6 2 Installation ...................................... 8 2.1 Requirements .................................................. 8 2.2 Installing the code ............................................. 8 2.3 Creating a buildmaster ......................................... 9 2.4 Upgrading an Existing Buildmaster............................ 10 2.5 Creating a buildslave.......................................... 10 2.5.1 Buildslave Options ....................................... 12 2.6 Launching the daemons ....................................... 13 2.7 Logfiles ................................................... .... 14 2.8 Shutdown ................................................... 14 2.9 Maintenance .................................................. 15 2.10 Troubleshooting.............................................
    [Show full text]