Workload Automation Documentation Release 2.7.0
Total Page:16
File Type:pdf, Size:1020Kb
Workload Automation Documentation Release 2.7.0 WA Mailing List <[email protected]>,Sergei Trofimov <sergei.trofi[email protected]>, Vasilis Flouris <vasilis.fl[email protected]>, Mohammed Binsabbar <[email protected]> Jul 06, 2018 Contents 1 What’s New 3 2 Usage 23 3 Extensions 55 4 In-depth 163 5 API Reference 199 6 Indices and tables 385 Python Module Index 387 i ii Workload Automation Documentation, Release 2.7.0 Workload Automation (WA) is a framework for running workloads on real hardware devices. WA supports a number of output formats as well as additional instrumentation (such as Streamline traces). A number of workloads are included with the framework. Contents • Welcome to Documentation for Workload Automation – What’s New – Usage – Extensions – In-depth – API Reference – Indices and tables Contents 1 Workload Automation Documentation, Release 2.7.0 2 Contents CHAPTER 1 What’s New 1.1 What’s New in Workload Automation 1.1.1 Version 2.6.0 Note: Users who are currently using the GitHub master version of WA should uninstall the existing version before upgrading to avoid potential issues. Additions: Workloads • AdobeReader: A workload that carries out following typical productivity tasks. These include opening a file, performing various gestures and zooms on screen and searching for a predefined set of strings. • octaned8: A workload to run the binary (non-browser) version of the JS benchmark Octane. • GooglePlayBooks: A workload to perform standard productivity tasks with Google Play Books. This workload performs various tasks, such as searching for a book title online, browsing through a book, adding and removing notes, word searching, and querying information about the book. • GooglePhotos: A workload to perform standard productivity tasks with Google Photos. Carries out various tasks, such as browsing images, performing zooms, and post-processing the image. • GoogleSlides: Carries out various tasks, such as creating a new presentation, adding text, images, and shapes, as well as basic editing and playing a slideshow. • Youtube: The workload plays a video, determined by the video_source parameter. While the video is playing, some common actions such as video seeking, pausing playback and navigating the comments section are performed. 3 Workload Automation Documentation, Release 2.7.0 • Skype: Replacement for the skypevideo workload. Logs into Skype and initiates a voice or video call with a contact. Framework • AndroidUxPerfWorkload: Added a new workload class to encapsulate functionality common to all uxperf workloads. • UxPerfUiAutomation: Added class which contains methods specific to UX performance testing. • get-assets: Added new script and command to retrieve external assets for workloads Results Processors • uxperf: Parses device logcat for UX_PERF markers to produce performance metrics for workload actions using specified instrumentation. Other • State Detection: Added feature to use visual state detection to verify the state of a workload after setup and run. Fixes/Improvements: Documentation • Revent: Added file structure to the documentation. • Clarified documentation regarding binary dependencies. • Updated documentation with create and get-assets commands. Instruments • sysfs_extractor: Fixed error when tar.gz file already existed on device, now overwrites. • cpufreq: Fixed error when tar.gz file already existed on device, now overwrites. • file-poller: – Improved csv output. – Added error checking and reporting. – Changed files to be a mandatory parameter. • fps: – Added a new parameter to fps instrument to specify the time period between calls to dumpsys SurfaceFlinger --latency when collecting frame data. – Added gfxinfo methods to obtain fps stats. Auto detects and uses appropriate method via android version of device. – Fixed issue with regex. 4 Chapter 1. What’s New Workload Automation Documentation, Release 2.7.0 – Now handles empty frames correctly. • energy_model: Ensures that the ui runtime parameter is only set for ChromeOS devices. • ftrace: Added support to handle traces collected by both WA and devlib. • Perf: Updated 32bit binary file for little endian devices. Resource Getters • http_getter: Now used to try and find executables files from a provided remove_assets_url. Result Processors • cpu_states: Fixes using stand-alone script with timeline option. Workloads • antutu: Fixed setting permissions of FINE_LOCATION on some devices. • bbench Fixed handling of missing results. • camerarecord: – Added frame stats collection through dumpsys gfxinfo. – Added possibility to select slow_motion recording mode. • Geekbench: – Fixed output file listing causing pull failure. – Added support for Geekbench 4. • recentfling: – Fixed issue when binaries were not uninstalled correctly. – Scripts are now deployed via install() to ensure they are executable. – Fixed handling of when a PID file is deleted before reaching processing results stage. – Added parameter to not start any apps before flinging. • rt-app: Added camera recorder simulation. • sysbench: Added arm64 binary. • Vellamo: Fixed capitalization in part of UIAutomation to prevent potential issues. • Spec2000: Now uses WA deployed version of busybox. • NetStat: Updated to support new default logcat format in Android 6. • Dex2oat: Now uses root if available. 1.1. What’s New in Workload Automation 5 Workload Automation Documentation, Release 2.7.0 Framework • adb_shell: – Fixed issue when using single quoted command with adb_shell. – Correctly forward stderror to the caller for newer version of adb. • revent – Added -S argument to “record” command to automatically record a screen capture after a recording is completed. – Fixed issue with multiple iterations of a revent workload. – Added -s option to executable to allow waiting on stdin. – Removed timeout in command as -s is specified. – Revent recordings can now be parsed and used within WA. – Fixed issue when some recordings wouldn’t be retrieved correctly. – Timeout is now based on recording duration. – Added magic and file version to revent files. Revent files should now start with REVENT followed by the file format version. – Added support for gamepad recording. This type of recording contains only the events from a gamepad device (which is automatically identified). – A mode field has been added to the recording format to help distinguish between the normal and gamepad recording types. – Added -g option to record command to expose the gamepad recording mode. – The structure of revent code has undergone a major overhaul to improve maintainability and robust- ness. – More detailed info command output. – Updated Makefile to support debug/production builds. • Android API: Upgraded Android API level from 17 to 18. • uiautomator: The window hierarchy is now dumped to a file when WA fails on android devices. • AndroidDevice: – Added support for downgrading when installing an APK. – Added a broadcast_media_mounted method to force a re-index of the mediaserver cache for a specified directory. – Now correctly handles None output for get_pids_of() when there are no running processes with the specified name. – Renamed the capture method from capture_view_hierachy to capture_ui_hierarchy. – Changed the file extension of the capture file to .uix – Added -rf to delete_files to be consistent with LinuxDevice. • LinuxDevice: Now ensures output from both stdout and etderr is propagated in the event of a DeviceError. • APKWorkload: – Now ensure APKs are replaced properly when reinstalling. 6 Chapter 1. What’s New Workload Automation Documentation, Release 2.7.0 – Now checks APK version and ABI when installing. – Fixed error on some devices when trying to grant permissions that were already granted. – Fixed some permissions not being granted. – Now allows disabling the main activity launch in setup (required for some apps). – Added parameter to clear data on reset (default behaviour unchanged). – Ignores exception for non-fatal permission grant failure. – Fixed issue of multiple versions of the same workload failing to find their APK. – Added method to ensure a valid apk version is used within a workload. – Updated how APK resolution is performed to maximise likelihood of a workload running. – When check_apk is True will prefer host APK and if no suitable APK is found, will use target APK if the correct version is present. When False will prefer target apk if it is a valid version otherwise will fallback to host APK. • RunConfiguration: Fixed disabling of instruments in workload specs. • Devices: – Added network connectivity check for devices. – Subclasses can now set requires_network to True and network connectivity check will be performed during setup(). • Workloads: – Added network check methods. – Fixed versions to be backwards compatible. – Updated workload versions to match APK files. – Fixed issues with calling super. • Assets: Added script to retrieve external assets for workloads. • Execution: Added a clean_up global config option to delete WA files from devices. • Runner: No longer takes a screenshot or dump of UI hierarchy for some errors when unnecessary, e.g. host errors. • core: Constraints and allowed values are now checked when set instead of when validating. • FpsProcessor: – Added requirement on filtered_vsyncs_to_compose for total_vsync metric. – Removed misleading comment in class description. • BaseUiAutomation: Added new Marker API so workloads generate start and end markers with a string name. • AndroidUiAutoBenchmark: Automatically checks for known package versions that don’t work well with AndroidUiAutoBenchmark