wa Documentation Release 3.3.1.dev1 Arm Limited Aug 25, 2021 Contents 1 What’s New 3 2 User Information 17 3 Developer Information 75 4 Plugin Reference 129 5 API 241 6 Glossary 255 7 FAQ 257 Python Module Index 261 Index 263 i ii wa Documentation, Release 3.3.1.dev1 Workload Automation (WA) is a framework for executing workloads and collecting measurements on Android and Linux devices. WA includes automation for nearly 40 workloads and supports some common instrumentation (ftrace, hwmon) along with a number of output formats. WA is designed primarily as a developer tool/framework to facilitate data driven development by providing a method of collecting measurements from a device in a repeatable way. WA is highly extensible. Most of the concrete functionality is implemented via plug-ins, and it is easy to write new plug-ins to support new device types, workloads, instruments or output processing. Note: To see the documentation of individual plugins please see the Plugin Reference. Contents • Welcome to Documentation for Workload Automation – What’s New – User Information – Developer Information – Plugin Reference – API – Glossary – FAQ Contents 1 wa Documentation, Release 3.3.1.dev1 2 Contents CHAPTER 1 What’s New 1.1 What’s New in Workload Automation 1.1.1 Version 3.3 New Features: Commands: • Add report command to provide a summary of a run. Instruments: • Add proc_stat instrument to monitor CPU load using data from /proc/stat. Framework: • Add support for simulating atomic writes to prevent race conditions when running current instances of WA. • Add support file transfer for SSH connections via SFTP and falling back to using SCP implementation. • Support detection of logcat buffer overflow and present a warning if this occurs. • Allow skipping all remaining jobs if a job had exhausted all of its retires. • Add polling mechanism for file transfers rather than relying on timeouts. • Add run_completed reboot policy to enable rebooting a target after a run has been completed. 3 wa Documentation, Release 3.3.1.dev1 Android Devices: • Enable configuration of whether to keep the screen on while the device is plugged in. Output Processors: • Enable the use of cascading deletion in Postgres databases to clean up after deletion of a run entry. Fixes/Improvements Framework: • Improvements to the process command to correctly handle skipped and in process jobs. • Add support for deprecated parameters allowing for a warning to be raised when providing a parameter that will no longer have an effect. • Switch implementation of SSH connections to use Paramiko for greater stability. • By default use sftp for file transfers with SSH connections, allow falling back to scp by setting use_scp. • Fix callbacks not being disconnected correctly when requested. • ApkInfo objects are now cached to reduce re-parsing of APK files. • Speed up discovery of wa output directories. • Fix merge handling of parameters from multiple files. Dockerfile: • Install additional instruments for use in the docker environment. • Fix environment variables not being defined in non interactive environments. Instruments: • trace_cmd additional fixes for python 3 support. Output Processors: • postgres: Fixed SQL command when creating a new event. Workloads: • aitutu: Improve reliability of results extraction. • androbench: Enabling dismissing of additional popups on some devices. • antutu: Now supports major version 8 in additional to version 7.X. • exoplayer: Add support for Android 10. • googlephotos: Support newer apk version. 4 Chapter 1. What’s New wa Documentation, Release 3.3.1.dev1 • gfxbench: Allow user configuration for which tests should be ran. • gfxbench: Improved score detection for a wider range of devices. • gfxbench: Moved results extraction out of run stage. • jankbench: Support newer versions of Pandas for processing. • pcmark: Add support for handling additional popups and installation flows. • pcmark: No longer clear and re-download test data before each execution. • speedometer: Enable the workload to run offline and drops requirement for UiAutomator. To support this root access is now required to run the workload. • youtube: Update to support later versions of the apk. Other: • cpustates: Improved name handling for unknown idle states. 1.1.2 Version 3.2 Warning: This release only supports Python 3.5+. Python 2 support has now been dropped. Fixes/Improvements Framework: • TargetInfo now tracks installed modules and will ensure the cache is also updated on module change. • Migrated the build scripts for uiauto based workloads to Python 3. • Uiauto applications now target SDK version 28 to prevent PlayProtect blocking the installation of the automation apks on some devices. • The workload metadata now includes the apk package name if applicable. Instruments: • energy_instruments will now have their teardown method called correctly. • energy_instruments: Added a keep_raw parameter to control whether raw files generated during exe- cution should be deleted upon teardown. • Update relevant instruments to make use of the new devlib collector interface, for more information please see the devlib documentation. Output Processors: • postgres: If initialisation fails then the output processor will no longer attempt to reconnect at a later point during the run. 1.1. What’s New in Workload Automation 5 wa Documentation, Release 3.3.1.dev1 • postgres: Will now ensure that the connection to the database is re-established if it is dropped e.g. due to a long expecting workload. • postgres: Change the type of the hostid field to Bigint to allow a larger range of ids. • postgres: Bump schema version to 1.5. • perf: Added support for the simpleperf profiling tool for android devices. • perf: Added support for the perf record command. • cpustates: Improve handling of situations where cpufreq and/or cpuinfo data is unavailable. Workloads: • adodereader: Now support apk version 19.7.1.10709. • antutu: Supports dismissing of popup asking to create a shortcut on the homescreen. • gmail: Now supports apk version 2019.05.26.252424914. • googlemaps: Now supports apk version 10.19.1. • googlephotos: Now supports apk version 4.28.0. • geekbench: Added support for versions 4.3.4, 4.4.0 and 4.4.2. • geekbench-corporate: Added support for versions 5.0.1 and 5.0.3. • pcmark: Now locks device orientation to portrait to increase compatibility. • pcmark: Supports dismissing new Android 10 permission warnings. Other: • Improve documentation to help debugging module installation errors. 1.1.3 Version 3.1.4 Warning: This is the last release that supports Python 2. Subsequent versions will be support Python 3.5+ only. New Features: Framework: • ApkWorkload: Allow specifying A maximum and minimum version of an APK instead of requiring a specific version. • TestPackageHandler: Added to support running android applications that are invoked via am instrument. • Directories can now be added as Artifacts. 6 Chapter 1. What’s New wa Documentation, Release 3.3.1.dev1 Workloads: • aitutu: Executes the Aitutu Image Speed/Accuracy and Object Speed/Accuracy tests. • uibench: Run a configurable activity of the UIBench workload suite. • uibenchjanktests: Run an automated and instrument version of the UIBench JankTests. • motionmark: Run a browser graphical benchmark. Other: • Added requirements.txt as a reference for known working package versions. Fixes/Improvements Framework: • JobOuput: Added an augmentation attribute to allow listing of enabled augmentations for individual jobs. • Better error handling for misconfiguration job selection. • All Workload classes now have an uninstall parameter to control whether any binaries installed to the target should be uninstalled again once the run has completed. • The cleanup_assets parameter is now more consistently utilized across workloads. • ApkWorkload: Added an activity attribute to allow for overriding the automatically detected version from the APK. • ApkWorkload Added support for providing an implicit activity path. • Fixed retrieving job level artifacts from a database backend. Output Processors: • SysfsExtractor: Ensure that the extracted directories are added as Artifacts. • InterruptStatsInstrument: Ensure that the output files are added as Artifacts. • Postgres: Fix missing system_id field from TargetInfo. • Postgres: Support uploading directory Artifacts. • Postgres: Bump the schema version to v1.3. Workloads: • geekbench: Improved apk version handling. • geekbench: Now supports apk version 4.3.2. 1.1. What’s New in Workload Automation 7 wa Documentation, Release 3.3.1.dev1 Other: • Dockerfile: Now installs all optional extras for use with WA. • Fixed support for YAML anchors. • Fixed building of documentation with Python 3. • Changed shorthand of installing all of WA extras to all as per the documentation. • Upgraded the Dockerfile to use Ubuntu 18.10 and Python 3. • Restricted maximum versions of numpy and pandas for Python 2.7. 1.1.4 Version 3.1.3 Fixes/Improvements Other: • Security update for PyYAML to attempt prevention of arbitrary code execution during parsing. 1.1.5 Version 3.1.2 Fixes/Improvements Framework: • Implement an explicit check for Devlib versions to ensure that versions are kept in sync with each other. • Added a View parameter to ApkWorkloads for use with certain instruments for example fps. • Added "supported_versions" attribute to workloads to allow specifying a list of supported version for a particular workload. • Change default behaviour to run any available version of a workload if a specific version is not specified. Output Processors: