DOCSLIB.ORG

Improving Random GUI Testing with Image-Based Widget Detection

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Improving Random GUI Testing with Image-Based Widget Detection This is a repository copy of Improving random GUI testing with image-based widget detection. White Rose Research Online URL for this paper: http://eprints.whiterose.ac.uk/145601/ Version: Accepted Version Proceedings Paper: White, T., Fraser, G. and Brown, G. orcid.org/0000-0001-8565-5476 (2019) Improving random GUI testing with image-based widget detection. In: ISSTA 2019 Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. ISSTA 2019 - 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, 15-19 Jul 2019, Beijing, China. ACM Digital Library , pp. 307-317. ISBN 9781450362245 https://doi.org/10.1145/3293882.3330551 © 2019 ACM. This is an author-produced version of a paper subsequently published by the ACM Digital Library. Uploaded in accordance with the publisher's self-archiving policy. Reuse Items deposited in White Rose Research Online are protected by copyright, with all rights reserved unless indicated otherwise. They may be downloaded and/or printed for private study, or other acts as permitted by national copyright laws. The publisher or other rights holders may allow further reproduction and re-use of the full text version. This is indicated by the licence information on the White Rose Research Online record for the item. Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing [email protected] including the URL of the record and the reason for the withdrawal request. [email protected] https://eprints.whiterose.ac.uk/ Improving Random GUI Testing with Image-Based Widget Detection Thomas D. White Gordon Fraser Guy J. Brown [email protected] [email protected] [email protected] Department of Computer Science, Chair of Software Engineering II, Department of Computer Science, The University of Sheield University of Passau The University of Sheield Sheield, United Kingdom Passau, Germany Sheield, United Kingdom ABSTRACT can be written using many diferent GUI libraries and widget sets, Graphical User Interfaces (GUIs) are one of the most common user each providing a diferent API to access widget information. Al- interfaces, enabling interactions with applications through mouse though this can be circumvented by accessibility APIs, these difer movements and key presses. Tools for automated testing of GUIs between operating systems, and updates to an Operating System exist, however they usually rely on operating system speciic or can remove or replace parts of the API. Furthermore, some appli- framework speciic knowledge in order to interact with the appli- cations may not even be supported by such APIs, such as those cation under test. Because of frequent operating system updates which draw directly to the screen, e.g., web canvasses [1]. These and a large variety of GUI frameworks, such tools are made obso- challenges make it diicult to produce and to maintain testing tools lete by time. For an automated GUI test generation tool, supporting that rely on GUI information. Without knowledge of GUI widgets, many frameworks and operating systems is impractical; new oper- test generation tools have to resort to blindly interacting with ran- ating system updates can remove required information and each dom screen locations. diferent GUI framework uses unique underlying data structures. To relieve GUI testing tools of the dependency on GUI and acces- We propose a technique for improving random GUI testing by au- sibility APIs, in this paper we explore the use of machine learning tomatically identifying GUI widgets in screen shots using machine techniques in order to identify GUI widgets. A machine learning learning techniques. This information provides guidance to GUI model is trained to detect the widget types and positions on the testing tools in environments not currently supported by deriv- screen, and this information is fed to a test generator which can ing GUI widget information from screen shots only. In our exper- then make more informed choices about how to interact with a pro- iments, we found that identifying GUI widgets from screen shots gram under test. However, generating a widget prediction model is and using this information to guide random testing achieved a sig- non-trivial: Diferent GUI libraries and operating systems use dif- niicantly higher branch coverage in 18 of 20 applications, with an ferent visual appearance of widgets. Even worse, GUIs can often be average increase of 42.5% compared to conventional random test- customized with user-deined themse, or assistive techniques such ing. as a high/low contrast graphical mode. In order to overcome this challenge, we randomly generate Java Swing GUIs, which can be KEYWORDS annotated automatically, as training data. The inal machine learn- ing model uses only visual data and can identify widgets in real GUI testing, object detection, black box testing, software engineer- application GUIs without needing additional information from an ing, data generation, convolutional neural networks operating system or API. In detail, the contributions of this paper are as follows: 1 INTRODUCTION • We describe a technique to automatically generate GUIs in A Graphical User Interface (GUI) enables events to be triggered in large quantities, in order to serve as training data for GUI an application through visual entities called widgets (e.g., buttons). widget prediction. Using keyboard and mouse, users interact with the widgets on a • We describe a technique based on deep learning that adapts GUI to ire events in the application. Automated GUI test genera- machine learning object detection algorithms to the prob- tion tools (e.g., AutoBlackTest [10], Sapienz [9], or GUITAR [11]) lem of GUI widget detection. simulate users by interacting with the widgets of a GUI, and they • We propose an improve random GUI testing approach that are increasingly applied by companies to test mobile and desktop relies on no external GUI APIs, and instead selects GUI in- applications. The efectiveness of these GUI test generation tools teractions based on a widget prediction model. depends the information they have available. A naïve GUI test gen- • We empircally investigate the efects of using GUI widget erator simply clicks on random screen positions. However, if a GUI prediction on random GUI testing. test generator knows the locations and types of widgets on the cur- rent application screen, then it can make better informed choices In our experiments, for 18 out of 20 Java open source applica- about where to target interactions with the program under test. tions tested, a random tester guided using predicted widget loca- GUI test generation tools tend to retrieve the information about tions achieved a signiicantly higher branch coverage than a ran- available GUI widgets through the APIs of the GUI library of the dom tester without guidance, with an average coverage increase target application, or the accessibility API of the operating sys- of 42.5%. Although our experiments demonstrate that the use of an tem. However, relying on these APIs has drawbacks: Applications API that provides the true widget details can lead to even higher coverage, such APIs are not always available. In contrast, our wid- With the exception of randomized testing, all the current ap- get prediction library requires nothing but a screenshot of the ap- proaches rely on an automated method of extracting widget infor- plication, and even works across diferent operating systems. mation from a GUI. There exists applications and application sce- narios where widget information cannot be automatically derived, 2 BACKGROUND and image labelling may be able to help with this. Interacting with applications through a GUI involves triggering 3 PREDICTING GUI WIDGETS FOR TEST events in the application with mouse clicks or key presses. Lo et al. [8] deine three types of widgets in a GUI: GENERATION In order to improve random GUI testing, we aim to identify wid- • Static widgets in a GUI are generally labels or tooltips. gets in screen shots using machine learning techniques. A chal- • Action widgets ire internal events in an application when lenge lies in retrieving a suiciently large labelled training dataset interacted with (e.g. buttons). to enable modern object recognition approaches to be applied. We • Data widgets are used to store data (e.g., text ields). produce this data by (1) generating random Java Swing GUIs, and In this paper, we focus on identifying action and data widgets. (2) labelling screenshots of these applications with widget data re- The simplest approach to generating GUI tests is through click- trieved through GUI ripping based on the Java Swing API. The ing on random places in the GUI window [6], hoping to hit wid- trained network can then predict the location and dimensions of gets by chance. This form of testing (“monkey testing”) is efective widgets from screen shots during test generation, and thus inlu- at inding crashes in applications and is cheap to run; no infor- ence where and how the GUI tester interacts with the application. mation is needed (although knowing the position and dimensions of the application on the screen is helpful). Monkey is now also 3.1 Identifying GUI Widgets commonly used to test mobile applications through tools like the Environmental factors such as operating system, user-deined theme, Android Monkeyrunner [5]. or application designer choice efect the appearance of widgets. GUI test generation tools can be made more eicient by provid- Each application can use a unique widget palette. When widget ing them with information about the available widgets and events. information cannot be extracted through use of external tools, e.g., This information can be retrieved using the GUI libraries underly- an accessibility API, then this diversity of widgets presents a prob- ing the widgets used in an application, or through the operating lem for GUI testing tools.
Load more

Recommended publications
  • Swing: Components for Graphical User Interfaces
    Swing: Components for Graphical User Interfaces Computer Science and Engineering College of Engineering The Ohio State University Lecture 22 GUI Computer Science and Engineering The Ohio State University GUI: A Hierarchy of Nested Widgets Computer Science and Engineering The Ohio State University Visual (Containment) Hierarchy Computer Science and Engineering The Ohio State University Top-level widgets: outermost window (a container) Frame, applet, dialog Intermediate widgets: allow nesting (a container) General purpose Panel, scroll pane, tabbed pane, tool bar Special purpose Layered pane Atomic widgets: nothing nested inside Basic controls Button, list, slider, text field Uneditable information displays Label, progress bar, tool tip Interactive displays of highly formatted information Color chooser, file chooser, tree For a visual (“look & feel”) of widgets see: http://java.sun.com/docs/books/tutorial/uiswing/components Vocabulary: Widgets usually referred to as “GUI components” or simply “components” History Computer Science and Engineering The Ohio State University Java 1.0: AWT (Abstract Window Toolkit) Platform-dependent implementations of widgets Java 1.2: Swing Most widgets written entirely in Java More portable Main Swing package: javax.swing Defines various GUI widgets Extensions of classes in AWT Many class names start with “J” Includes 16 nested subpackages javax.swing.event, javax.swing.table, javax.swing.text… Basic GUI widgets include JFrame, JDialog JPanel, JScrollPane, JTabbedPane,
    [Show full text]
  • Proveos User Manual
    User Manual Contents Welcome .............................................................................................. 5 How proVEOS Works ...........................................................................................7 Installing the proVEOS Software for Windows ..................................... 9 Configuring Firewall Software .............................................................................10 Installing the Windows proVEOS Software .........................................................14 Signing in to proVEOS ........................................................................................24 Installing the proVEOS Software for Mac........................................... 29 Configuring the Firewall ......................................................................................30 Installing the Mac proVEOS Software.................................................................32 Signing in to proVEOS ........................................................................................34 Presenting with proVEOS ................................................................... 37 Starting a Presentation .......................................................................................40 Managing Participants ........................................................................................40 Chatting with Participants ...................................................................................45 Playing Music and Movies ................................................................
    [Show full text]
  • Java Programming
    CHAPTER 15 Advanced GUI Topics In this chapter, you will: Use content panes Use color Learn more about layout managers Use JPanels to increase layout options Create JScrollPanes Understand events and event handling more thoroughly Use the AWTEvent class methods Handle mouse events Use menus Unless noted otherwise, all images are © 2014 Cengage Learning Copyright 2013 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s). Editorial review has deemed that any suppressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions require it. CHAPTER 15 Advanced GUI Topics Understanding the Content Pane The JFrame class is a top-level container Java Swing class. (The other two top-level container classes are JDialog and JApplet.) Every GUI component that appears on the screen must be part of a containment hierarchy. A containment hierarchy is a tree of components that has a 802 top-level container as its root (that is, at its uppermost level). Every top-level container has a content pane that contains all the visible components in the container’s user interface. The content pane can directly contain components like JButtons, or it can hold other containers, like JPanels, that in turn contain such components. A top-level container can contain a menu bar. A menu bar is a horizontal strip that conventionally is placed at the top of a container and contains user options.
    [Show full text]
  • Software Manual IDEA – the Software
    Software Manual IDEA – The Software Version 1.1 Open Technologies Srl WWW.SCANINABOX.COM Copyright © 2016 Open Technologies Srl First printing, June 2016 Contents I Part One 1 IDEA - the application for Scan in a Box .........................7 1.1 Application Interface7 1.1.1 Project management panel..........................................8 1.1.2 3D view panel....................................................9 1.1.3 Toolbar......................................................... 10 1.1.3.1 Project management (orange)........................................... 10 1.1.3.2 Acquisition (bordeaux)................................................ 10 1.1.3.3 Alignment (light blue)................................................. 11 1.1.3.4 Selection (turquoise).................................................. 11 1.1.3.5 Rendering (pink).................................................... 12 1.1.3.6 General (purple).................................................... 12 1.1.3.7 Features specific to range images (blue)..................................... 13 1.1.3.8 Features specific to triangle meshes (light green)................................ 13 2 Using IDEA ................................................... 17 2.1 Optical set-up 17 2.1.1 Optical set-up mode................................................ 19 2.1.2 Calibration of the optical head........................................ 22 2.2 Capture and alignment 25 2.2.1 Free mode acquisition............................................... 26 2.2.2 Turn Table acquisition mode..........................................
    [Show full text]
  • LWUIT Developer's Guide
    Lightweight UI Toolkit Developer’s Guide Part No. 07-08-10 July 2010 Copyright © 2008, 2010 Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this is software or related software documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500 Oracle Parkway, Redwood City, CA 94065.
    [Show full text]
  • COMP1006/1406 Notes 2
    COMP1406/1006 - Design and Implementation of Computer Applications Winter 2006 2 Applications and LayoutManagers What's in This Set of Notes? We look here at how to create a graphical user interface (i.e., with windows) in JAVA. Creating GUIs in JAVA requires adding components onto windows. We will find out how to do this as well as look at an interesting JAVA feature called a "LayoutManager" that automatically arranges components on the window. This allows us to create simple windows without having to worry about resizing issues. Here are the individual topics found in this set of notes (click on one to go there): • 2.1 Creating a Basic GUI Application • 2.2 Components and Containers • 2.3 Layout Managers o 2.3.1 NullLayout o 2.3.2 FlowLayout o 2.3.3 BorderLayout o 2.3.4 CardLayout o 2.3.5 GridLayout o 2.3.6 GridBagLayout o 2.3.7 BoxLayout 2.1 Creating a Basic GUI Application Recall that a Graphical User Interface (GUI) is a user interface that has one or more windows. A frame: • is the JAVA terminology for a window (i.e., its a window frame) • represented by the Frame and JFrame classes in JAVA • used to show information and handle user interaction • has no security restrictions ... can modify files, perform I/O, open network connections to other computers, etc.. The following code creates a basic window frame in JAVA and shows it on the screen: javax.swing.JFrame frame = new javax.swing.JFrame("This appears at the top of the window"); frame.setSize(300, 100); frame.setVisible(true); Here is what it looks like: Although this code for bringing up a new window can appear anywhere, we typically designate a whole class to represent the window (that is, the JAVA application).
    [Show full text]
  • Release Notes for Fedora 15
    Fedora 15 Release Notes Release Notes for Fedora 15 Edited by The Fedora Docs Team Copyright © 2011 Red Hat, Inc. and others. The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. The original authors of this document, and Red Hat, designate the Fedora Project as the "Attribution Party" for purposes of CC-BY-SA. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. For guidelines on the permitted uses of the Fedora trademarks, refer to https:// fedoraproject.org/wiki/Legal:Trademark_guidelines. Linux® is the registered trademark of Linus Torvalds in the United States and other countries. Java® is a registered trademark of Oracle and/or its affiliates. XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries. MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries. All other trademarks are the property of their respective owners.
    [Show full text]
  • Maplogic Layout Manager User's Manual
    MAPLOGIC CORPORATION GIS Software Solutions MapLogic Layout Manager User’s Manual MapLogic Layout Manager User’s Manual © 2009 MapLogic Corporation All Rights Reserved 330 West Canton Ave., Winter Park, Florida 32789 Phone (407) 657-1250 • Fax (407) 657-7008 MapLogic Layout Manager is a trademark of MapLogic Corporation. ArcGIS, ArcMap and ArcView are registered trademarks of ESRI. Table of Contents Introduction _____________________________________________________________________ 1 What Is MapLogic Layout Manager? ____________________________________________ 1 MapLogic Layout Manager User Interface ____________________________________________ 2 The Layouts Tab ____________________________________________________________ 2 The MapLogic Layout Manager Toolbar _________________________________________ 3 Element Properties __________________________________________________________ 4 Before You Get Started ____________________________________________________________ 6 How The MapLogic Layout Manager Changes The ArcMap Document_________________ 6 Different Licenses For The MapLogic Layout Manager _____________________________ 6 Basic Concepts ___________________________________________________________________ 8 Multiple Layouts Within The ArcMap Document __________________________________ 8 The Map Series Layout _______________________________________________________ 8 The Map Book Layout _______________________________________________________ 9 New Elements You Can Add To Map Series And Book Layouts _____________________ 10 Two-Sided Map
    [Show full text]
  • R16 B.TECH CSE IV Year Syllabus
    R16 B.TECH CSE. JAWAHARLAL NEHRU TECHNOLOGICAL UNIVERSITY HYDERABAD B.TECH. COMPUTER SCIENCE AND ENGINEERING IV YEAR COURSE STRUCTURE & SYLLABUS (R16) Applicable From 2016-17 Admitted Batch IV YEAR I SEMESTER S. No Course Code Course Title L T P Credits 1 CS701PC Data Mining 4 0 0 4 2 CS702PC Principles of Programming Languages 4 0 0 4 3 Professional Elective – II 3 0 0 3 4 Professional Elective – III 3 0 0 3 5 Professional Elective – IV 3 0 0 3 6 CS703PC Data Mining Lab 0 0 3 2 7 PE-II Lab # 0 0 3 2 CS751PC Python Programming Lab CS752PC Android Application Development Lab CS753PC Linux programming Lab CS754PC R Programming Lab CS755PC Internet of Things Lab 8 CS705PC Industry Oriented Mini Project 0 0 3 2 9 CS706PC Seminar 0 0 2 1 Total Credits 17 0 11 24 # Courses in PE - II and PE - II Lab must be in 1-1 correspondence. IV YEAR II SEMESTER Course S. No Course Title L T P Credits Code 1 Open Elective – III 3 0 0 3 2 Professional Elective – V 3 0 0 3 3 Professional Elective – VI 3 0 0 3 4 CS801PC Major Project 0 0 30 15 Total Credits 9 0 30 24 Professional Elective – I CS611PE Mobile Computing CS612PE Design Patterns CS613PE Artificial Intelligence CS614PE Information Security Management (Security Analyst - I) CS615PE Introduction to Analytics (Associate Analytics - I) R16 B.TECH CSE. Professional Elective – II CS721PE Python Programming CS722PE Android Application Development CS723PE Linux Programming CS724PE R Programming CS725PE Internet of Things Professional Elective - III CS731PE Distributed Systems CS732PE Machine Learning CS733PE
    [Show full text]
  • AGL HMI Framework Architecture Document
    AGL HMI Framework Architecture Document Version Date 0.9 2019/x/x AGL HMI Framework Architecture Document 1. HMI Framework overview ........................................................................................ 4 1.1. Oerview .............................................................................................................. 4 1.1.1. HMI-FW Components ................................................................................. 5 1.1.2. Related components ..................................................................................... 6 2. GUI-library ................................................................................................................ 7 2.1. Overview ............................................................................................................ 7 2.1.1. Related external components ....................................................................... 7 2.1.2. Internal Components ................................................................................... 8 2.2. Graphics functions .............................................................................................. 9 2.2.1. Procedure necessary for HMI-Apps ............................................................ 9 2.2.2. Software configuration of GUI-lib ............................................................ 10 2.3. Sound functions ................................................................................................ 12 2.4. Input functions .................................................................................................
    [Show full text]
  • Java Lecture 5 AWT
    Java Lecture 5 AWT Abstract Windows Toolkit: A Java window is a set of nested components, starting from the outermost window down to the smallest user interface (UI) component. Peers: Peers are native GUI components (native to the platform). When a Java program creates and displays a Java AWT component, that component actually creates and displays a native component (called a peer). Thus, under Windows 95, a windows 95 button, or dialog, etc. is generated. Under Solaris a Motif peer and under Macintosh, a Macintosh peer. Components include seeable things such as: windows menu bars buttons text fields and non-seeable things like containers which can include other components. Containers have layout managers that deal with positioning and shaping of components. The nesting of components within containers within other components creates a hierarchy for painting and event passing. Major components: Containers: Containers are generic AWT components that can contain other components, including other containers. The most common form of container is the panel, which represents a container that can be displayed on screen. Applets are a form of panel (in fact, the "Applet" class is a subclass of the "Panel" class). Canvases: A canvas is a simple drawing surface. Although you can draw on panels, (as you've been doing all along), canvases are good for painting images or other graphics operations. UI components: These can include buttons, lists, simple popup menus, checkboxes test fields, and other typical elements of a user interface. Window construction components: These include windows, frames, menu bars, and dialogs. These are listed separately from the other UI components because you'll use these less often--particularly in applets.
    [Show full text]
  • Manual – Remote Control Page 2 of 111 7.2 Authenticating on the Remote Computer Using a Windows User Account 28
    TeamViewer 10 Manual Remote Control Rev 10.3-201506 TeamViewer GmbH • Jahnstraße 30 D-73037 Göppingen www.teamviewer.com Table of contents 1 About TeamViewer 5 1.1 About the software 5 1.2 About the manual 5 2 Basics 7 2.1 How TeamViewer works 7 2.2 Description of the main TeamViewer window 7 3 Establishing a connection with TeamViewer 10 4 The Remote Control connection mode 11 4.1 Remote Control window options 11 4.2 Remote computer options in the TeamViewer Panel 18 5 The File Transfer connection mode 21 5.1 Options in File Transfer connection mode 21 5.2 Transferring files using drag and drop 23 5.3 File transfer via the Windows context menu 24 6 The VPN connection mode 25 6.1 Sample uses of TeamViewer VPN 25 6.2 Requirements for using TeamViewer VPN 25 6.3 VPN dialog box options 26 7 Other connection options 27 7.1 Establishing a LAN connection using the IP address 27 TeamViewer 10 Manual – Remote Control www.teamviewer.com Page 2 of 111 7.2 Authenticating on the Remote Computer using a Windows user account 28 7.3 Establish a connection via a Windows shortcut 29 7.4 Connections to your own computers without a password. 30 8 Computers & Contacts – Managing Contacts 32 8.1 The TeamViewer account 33 8.2 Computers & Contacts options 37 8.3 Service cases 53 8.4 Integrated system health checks 56 9 Multimedia functions 58 9.1 Options within the Session list 58 9.2 Transmitting webcam video to your partner 58 9.3 Talk to your partner via Voice over IP or conference call 59 9.4 Chatting with your partner during a TeamViewer session 61 9.5
    [Show full text]