Keynote Talk

ICST 2018 11th IEEE Conference on Software Testing, Validation and Verification

Västerås, Sweden • April 9-13, 2018

Content Program at a glance...... 3 Sponsors...... 4 Message from the General Chair...... 5 Message from the Program Chairs...... 8 Organization...... 10 Keynote Talks...... 12 Venue...... 15 Detailed program ICST...... 36 Workshops, overview...... 17

Workshops, detailed program...... 36 A-MOST...... 47 InSTA...... 45 ITEQS...... 36 IWCT...... 45 MUTATION...... 39 NEXTA...... 38 SAST, Swedish Association for Software Testing...... 18 TAIC PART...... 46 USE...... 37 VVIoT...... 37

2 Program at a glance

Monday Tuesday Wednesday Thursday Friday April 9th April 10th April 11th April 12th April 13th

09.00 SAST meeting Intro & Welcome MIP #1 workshops Auditorium 09.30 Keynote 1: Keynote 2: Keynote 3: Infer: Static Analysis But is it fun? Safety Verification for A-MOST Magnus Nordin/David 10.00 at Scale Deep Neural Networks Room 302 workshops Dulma Churchill King/Stefan Posthuma Marta Kwiatkowska 10.30 Coffee InSTA ITEQS Room 301 11.00 Room 107 Research 1 Research 6 Research 9 Industry Track 1 Research 7 Research 10 IWCT 11.30 Tools/Demos MUTATION Doctoral symp 1 Room 109 Lightning Talks 12.00 Room 109 TESTOMAT meeting TAIC PART 12.30 NEXTA Room 107 Room 301 Lunch 13.00 13.30 USE Room 101 14.00 Research 2 Research 8 Open Steering VVIoT Research 3 Tool 2 Committee & Closing 14.30 Room 106 Tool 1 Doctoral Symp 2 Room 109 15.00 15.30 Coffee

16.00 Research 4 Demo Research 5 Industry Track 2 16.30 Journal First Doctoral symp 3

17.00 Panel Discussion

17.30 17:30 Burgers & activities 18.00 at Strike, 18:00 18:00 Torggatan 1, Västerås Welcome reception at Bus from the 18.30 the Plaza lobby bar, Congress Center Hotel Plaza to the conference 19.00 19:00 banquet at the PC dinner at Munktell museum, restaurant Pråmen, Eskilstuna Västerås

ROOMS

Key notes and panel: Congress hall

Sessions: TESTOMAT meeting: Tuesday, see page 12 Room 108 Wednesday, see page 13 Thursday, see page 14

3 Conference Sponsors

Platinum Sponsors

Gold Sponsors

Silver Sponsors

Testing of critical system characteristics

Bronze Sponsors

INFOTIV

4

Message from the ICST 2018 General Chair

Hans Hansson Mälardalen University & RISE SICS Västerås, Sweden [email protected]

Welcome!

I welcome you all to Västerås, Sweden, for the 11th IEEE International Confer- ence on Software Testing, Verification and Validation (ICST 2018). Yes, ICST is approaching adoles- cence! Indications of this maturity in- clude that the conference now is rated A according to CORE20181 and that this year’s conference features awards to and presentations by authors of the two most influential ICST papers during the first ten years of ICST. We are proud that Västerås and pants may want to spend a day or two tools. For the success of this I’m much Sweden was selected the venue for before or after the conference to ex- in debt to Robert Feldt and Shin Yoo, this year’s conference. Västerås with plore Stockholm, the capital of Sweden Chairs of the Main Technical Track Pro- 150.000 inhabitants is a medium size that has a lot to offer, less than an hour gram Committee for putting together city by Swedish standards. Still, it can away from Västerås by train. the technical program, and also for be considered the engineering capital In addition to the comprehensive supporting me in general issues related of Sweden, with a number of major conference programme, which includes to the organization of the conference. engineering companies, such as ABB, a regular programme and nine co-located In addition, I wish to thank Daniel Bombardier Transportation, Westing- workshops, there are a number of events Sundmark, Stig Larsson and Richard house and Alstom. The size of the city and meetings co-located with ICST 2018, Torkar for Chairing the Industry Papers makes Västerås ideal for a conference most notably a meeting of the Swedish Track and encouraging practitioners to such as ICST since all hotels and loca- Association for Software Testing (SAST), showcase their work on the applica- tions for social events are within con- which is expected to include some 250 tion of recently developed techniques venient walking distance from the Aros software practitioners keen to interact to real industrial software systems. Congress Center where we will meet. with the ICST community. Daniel Sundmark deserves an additional The only exception is the conference The aim is to have this meeting con- recognition for taking responsibility for banquet which will take place in the tinue the long ICST tradition of foster- the very successful sponsoring efforts, Munktell museum in the neighbouring ing the integration of software testing as well as for supporting me in many city of Eskilstuna, the home of Volvo and verification research and practice, other ways. Thanks also to the sponsors Construction Equipment. which is why we have worked very hard for their very generous contributions! Although Västerås has a lot to offer, to develop a strong technical content The workshops chairs, Gregory Gay and the more touristically inclined partici- that combines research, practice, and Mike Papadakis, have made a great

1 http://www.core.edu.au/conference-portal (Rank A - excellent conference, and highly respected in a discipline area.)

5 effort in coordinating the co-located Mehrdad Saadatmand and Leo Hatvani, workshops. Thanks also to the Test- with Leo also excellently handling the ing tools, demonstrations and artifacts conference web. track chairs Wasif Afzal, Francisco Behind any successful event there Gomes and René Just, and the PhD are a number of persons that do not al- symposium chairs David Binkley and ways get the recognition they deserve: Arpad Beszédes for excellent organiza- the team of local heroes that makes tion of their respective tracks. One of sure that everything runs smoothly and the conference high-lights is a panel seamlessly. For the organization of ICST on the eternal and important theme of 2018, I would in particular like to thank whether software testing research con- the local chair Malin Rosqvist that with tributions are real contributions, organ- great enthusiasm have coordinated ized by Sigrid Eldh and Elaine Weyuker. the local team, also including Daniel If you are attending ICST for the first Sundmark, Karin Engström, Susanne time, it is most likely due to the tireless Fronnå, Eduard Enoiu, Leo Hatvani, and efforts of the publicity chairs Eduard Predrag Filipovikj, as well as a number Enoiu, Xiaoyuan Xie and Tingting Yu. of student helpers. Equally important is the responsibility to Last, but not least, I would like to handle the production of the proceed- thank the ICST Steering Committee, ings shared by the publicity chairs chaired by Gordon Fraser, for showing

Photographs: Linda Heplinger, VKL, Leon Grimaldi, Clifford Shirley, Kristoffer Hasselberg

6 confidence and providing support & RSV six years ago, RSV and MDH have guidance. been working closely in a large number ICST 2018 is co-organised by the of research and education projects. Embedded Systems (ES) environment at There is also a number of persons with Mälardalen University (MDH) and the double affiliation, including myself. It is research institute RISE SICS Västerås thus natural that we join forces in the (RSV). ES is a large academic research organisation of ICST 2018. environment with approximately 200 It takes a diverse and hard-working researchers and PhD-students, cover- team to put a conference together. ing a broad set of topics related to Working with this team has been an software embedded in products and amazing experience. Working together, production systems; most notably we hope that we have created an event related to quality assurance of such that is long remembered by the com- software, including testing, verification munity, both enjoyable and educational. and validation. RSV includes some 40 Once again, I welcome you to ICST 2018! researchers targeting a broad set of topics related to the digitalization of the manufacturing industry in a broad sense, including expertise in software General Chair of ICST 2018 testing. Already since the inception of Hans Hansson

7 Message from the ICST 2018 Program Chairs

We are very pleased to present the In addition, nine workshops have been increasingly important artefact submis- proceedings of the 11th International collocated with ICST 2018. Finally, this sions. Gregory Gay and Mike Papadakis Conference on Software Testing, Veri- year’s ICST took place in conjunction organised the collocated workshops, fication & Validation on behalf of the with the quarterly meeting of Swedish continuing the technical programme entire Programme Committee. This Association for Software Testing (SAST), would not have been what it is without year’s edition of ICST is in Västeras, putting a strong emphasis on the close their effort, for which we are deeply one of the oldest cities in Sweden, and collaboration between academia and grateful. We also want to express our a vibrant industrial city. The location industry. gratitude to the ICST Steering Commit- provides a fitting background for ICST, We would like to thank all members tee for supporting us through the pro- which continues to bring together re- of the ICST 2018 Program Committee, cess and for helping out with backup, searchers and practitioners to exchange whose support and contribution was lastminute reviews and in questions ideas and celebrate latest progress. absolutely critical to the high quality of about conference scope. The programme committee is happy the technical programme of ICST 2018. ICST has a strong tradition of to report that ICST 2018 received 195 The general chair, Hans Hansson, led an having the world leading researchers number of submissions to the four excellent organisation team, streamlin- and practitioners as its keynote speak- tracks. After filtering out desk rejects ing every process as much as possible. ers, and this year is no exception. We and abstract only submissions, the pro- Industry chairs, Stig Larsson, Daniel heard from Dulma Churchill, whose gramme committee has reviewed 119 Sundmark, and Richard Torkar worked work is about providing industry scale submissions for the Full Research Track, closely with the industry partners and static analysis at Facebook. We had the 9 for the Industry Track, 21 for the Tools, sponsors to ensure that ICST 2018 is team of Magnus Nordin, David King, Demos & Artefacts Track, and 14 for both relevant and accessible to industry and Stefan Posthuma from DICE and the PhD Symposium. Participation was practitioners. Arpad Beszédes and Dave Electronic Arts, all of whom work hard from 43 different countries, reflecting Binkley worked hard not only to elicit to test the highly complex systems that the international nature of our com- and review PhD Symposium submis- are modern video games. Finally, we munity. After carefully reviewing the sions but also to make sure students had the honour of hearing the keynote submissions, we have accepted 30 Full are sufficiently supported. Wasif talk from Marta Kwiatkowska, profes- Research Track papers, 5 Industry Track Afzal, Francisco Gomez, and Rene sor at Oxford University, whose work papers, 9 Tools, Demo & Artefacts Track Just oversaw the process of handling on verification of deep neural networks papers, and 7 PhD Symposium papers. tool papers, demo proposals, and the cannot be more timely and relevant.

Photograph: Leon Grimaldi

8 This year, we implemented the Program Chairs of ICST 2018 Double Blind Review process to the Full Research Track for the first time in ICST. Based on many empirical evidence, our belief is that the double blind review- ing is the best way to enable as fair a review process as possible, although Robert Feldt not without its own inconveniences Chalmers University of Technology, Sweden and flaws. We want to thank both the Programme Committee and the submitting authors for their patience with a new and unfamiliar process. Fair and just peer review is vital to the health and progress of our community; we would like to encourage all who participated to continue to exchange ideas and perspectives on this impor- Shin Yoo tant issue. Korea Advanced Institute of Science and Technology, We would like to thank all the Republic of Korea authors who submitted to ICST 2018 regardless of the outcome, and every- one who attended the conference. We also hope that these proceedings can help anyone who were not in Västeras have the opportunity to experience the open and vibrant ICST community as well as its excellent research outcomes.

Photographs: Jennifer Gosch, Clifford Shirley

9 Organization

General Chair: Hans Hansson, Mälardalen University / RISE SICS Västerås, Sweden Program Chairs: Robert Feldt, Chalmers University of Technology, Sweden Shin Yoo, Korea Advanced Institute of Science and Technology, Republic of Korea Workshop Chairs: Gregory Gay, University of South California, USA Mike Papadakis, University of Luxembourg, Luxembourg Doctoral Symposium Chairs: David Binkley, Loyola University Baltimore, USA Arpad Beszédes, University of Szeged, Hungary Industry Track Chairs: Daniel Sundmark, Mälardalen University, Sweden Stig Larsson RISE SICS Västerås, Sweden Richard Torkar Chalmers/University of Gothenburg, Sweden Testing Tools, Demonstrations Francisco Gomes, Chalmers University of Technology, and Artifacts Track Chairs: and University of Gothenburg, Sweden René Just, University of Massachusetts Amherst, USA Wasif Afzal Mälardalen University, Sweden Publicity Chairs: Eduard Enoiu, Mälardalen University, Sweden Xiaoyuan Xie, Wuhan University, China Tingting Yu, University of Kentucky, USA Publication Chairs: Mehrdad Saadatmand, Rise SICS Västerås, Sweden Leo Hatvani, Mälardalen University, Sweden Local Arrangements Chair: Malin Rosqvist, RISE SICS Västerås, Sweden Web Chair: Leo Hatvani, Mälardalen University, Sweden

Hans Hansson Robert Feldt Shin Yoo Gregory Gay Mike Papadakis David Binkley Arpad Beszédes General Chair Program Chair Program Chair Workshop Chair Workshop Chair Doctoral Symposium Doctoral Symposium Chair Chair

Daniel Sundmark Stig Larsson Richard Torkar Francisco Gomes René Just Wasif Afzal Eduard Enoiu Industry Track Chair Industry Track Chair Industry Track Chair Testing Tools, Testing Tools, Testing Tools, Publicity Chair Demonstrations and Demonstrations and Demonstrations and Artifacts Track Chair Artifacts Track Chair Artifacts Track Chair

Xiaoyuan Xie Tingting Yu Mehrdad Malin Rosqvist Leo Hatvani Publicity Chair Publicity Chair Saadatmand Local Arrangements Publication Chair Publication Chair Chair Web Chair

10 Steering Committee Zijiang Yang, Western Michigan University, USA Lionel Briand, Université du Luxembourg, Luxembourg Hiroaki Yoshida, Fujitsu Laboratories of America, USA Myra Cohen, University of Nebraska - Lincoln, USA Tingting Yu, University of Kentucky, USA Gordon Fraser (Chair), University of Sheffield, UK Andy Zaidman, Delft University of Technology, The Netherlands Mark Grechanik, University of Illinois at Chicago, USA Darko Marinov, University of Illinois at Urbana-Champaign, USA Program Committee: Industry Track Atif Memon, University of Maryland, USA Emil Alégroth, Blekinge Institute of Technology, Sweden John Penix, Google, USA Anneliese Andrews, University of Denver, USA Franz Wotawa, Graz University of Technology, Austria Jeffrey Carver, University of Alabama, USA Shin Yoo, Korea Advanced Institute of Science and Technology, Republic of Korea Sigrid Eldh, Ericsson AB, Sweden Thomas Gustafsson, Scania CV AB, Sweden Program Committee: Research Track Gregory Kapfhammer, Allegheny College, USA Bram Adams, Polytechnique Montréal, Canada Atif Memon, University of Maryland, USA Emil Alégroth, Blekinge Institute of Technology, Sweden John Micco, Google, USA Shaukat Ali, Simula Research Laboratory, Norway Adithya Nagarajan, Apple, USA Anneliese Andrews, University of Denver, USA Bao Nguyen, Google, USA Emily Bache, Bache Consulting, Sweden Manuel Oriol, ABB Corporate Research, Switzerland Antonia Bertolino, ISTI-CNR, Italy Murat Ozturk, Google, UK Jeffrey Carver, University of Alabama, USA Corina Pasareanu, CMU/NASA Ames Research Center, USA Ana Cavalli, Telecom Sudparis, France Saurabh Sinha, IBM Research, USA T.Y. Chen, Swinburne University of Technology, Australia Rachel Tzoref-Brill, IBM Research - Haifa, Israel Byoungju Choi, Ewha Womans University, Republic of Korea Tanja Vos, Open Universiteit and Universitat Politecnica de Valencia, Spain Eun-Hye Choi, National Institute of Advanced Industrial Science and Technology, Japan Kristian Wiklund, Ericsson AB and Mälardalen University, Sweden James Clause, University of Delaware, USA Tingting Yu, University of Kentucky, USA Marcelo d’Amorim, Federal University of Pernambuco, Brazil Hyunsook Do, University of North Texas, USA Program Committee: Testing Tools Papers Track Alastair Donaldson, Imperial College London, UK Mariano Ceccato, Fondazione Bruno Kessler, Italy Lydie Du Bousquet, Laboratoire d’Informatique de Grenoble (LIG), France Eun-Hye Choi, National Institute of Advanced Industrial Science and Sigrid Eldh, Ericsson AB, Sweden Technology (AIST), Japan Michael Felderer, University of Innsbruck, Austria Damiano Distante, Unitelma Sapienza University, Italy Jaco Geldenhuys, Stellenbosch University, South Africa Takashi Ishio, Nara Institute of Science and Technology, Japan Alessandra Gorla, IMDEA Software Institute, Spain Yang Liu, Nanyang Technological University, Singapore Hadi Hemmati, University of Calgary, Canada Karthik Pattabiraman, The University of British Columbia, Canada Rob Hierons, Brunel University, UK Goran Petrovic, Google Switzerland GmbH, Switzerland Shin Hong, Handong Global University, Republic of Korea Alexander Pretschner, Technische Universität München, Germany John Hughes, Chalmers University of Technology, Sweden Xiao Qu, ABB Corporate Research, USA Gregory Kapfhammer, Allegheny College, USA José Miguel Rojas, The University of Sheffield, UK Fitsum Meshesha Kifetew, Fondazione Bruno Kessler, Italy Stefano Scala, Fiat Chrysler Automobiles, Italy Yunho Kim, Korea Advanced Institute of Science and Technology, Republic of Korea Hideo Tanida, Fujitsu Laboratories Ltd., USA Nan Li, Medidata Solutions, USA Porfirio Tramontana, Università degli Studi di Napoli Federico II, Italy Xuandong Li, Nanjing University, China Zijiang Yang, Western Michigan University, USA Francesca Lonetti, ISTI-CNR, Italy Ana Paiva, University of Porto, Portugal Patricia Machado, Federal University of Campina Grande, Brazil Additional Reviewers: Research Track Mika Mäntylä, University of Oulu, Finland Myra Cohen, University of Nebraska-Lincoln, USA Eda Marchetti, ISTI-CNR, Italy Darko Marinov, University of Illinois at Urbana-Champaign, USA Jean-Yves Marion, Université de Lorraine, LORIA, France Gordon Fraser, The University of Sheffield, UK Atif Memon, University of Maryland, USA James Miller, University of Alberta, USA Subreviewers Razieh Nokhbeh Zaeem, The University of Texas at Austin, USA César Andrés, Universidad Complutense de Madrid, Spain Manuel Oriol, ABB Corporate Research, Switzerland Pamela Carvallo, Telecom SudParis, France Mike Papadakis, University of Luxembourg, Luxembourg Maayan Goldstein, Bell Labs, Israel Corina Pasareanu, CMU/NASA Ames Research Center, USA Florian Hauer, Technische Universität München, Germany Andy Podgurski, Case Western Reserve University, USA Takashi Kitamura, National Institute of Advanced Industrial Adam Porter, University of Maryland, USA Science and Technology, Japan Simon Poulding, Blekinge Institute of Technology, Sweden Alexander Knüppel, Technische Universität Braunschweig, Germany Ina Schaefer, Technische Universität Braunschweig, Germany Sabrina Lischke, Technische Universität Braunschweig, Germany Itai Segall, Bell Labs, Israel Sascha Lity, Technische Universität Braunschweig, Germany Saurabh Sinha, IBM Research, USA Lars Luthmann, Technische Universität Darmstadt, Germany Mary Lou Soffa, University of Virginia, USA Osamu Mizuno, Kyoto Institute of Technology, Japan Sara Sprenkle, Washington and Lee University, USA Huu Nghia Nguyen, Montimage, France Alin Stefanescu, University of Bucharest, Romania Michael Nieke, Technische Universität Braunschweig, Germany Rachel Tzoref-Brill, IBM Research - Haifa, Israel Yannic Noller, Humboldt Universität, Germany Tanja E. J. Vos, Open Universiteit and Universitat Politecnica de Valencia, Spain Saahil Ognawala, Technische Universität München, Germany Helene Waeselynck, LAAS-CNRS, France Georges Ouffoué, Université Paris Sud, France Neil Walkinshaw, The University of Leicester, UK Khalifa Toumi, Telecom SudParis, France 11 th ICSTKeynote conference Talk Tuesday, program April 10 Tuesday, April 10th

09:00 Intro & Welcome 09:30 Track 1 Keynote 1: Infer: Static Analysis at Scale Dulma Churchill (Congress Hall) 10:00 10:30 Track 1 Coffee Break Congress Hall Auditorium (Hörsalen) Room 107

11:00 Research 1 Industry Track 1 Tools/Demos Lightning Talks Testing & Debugging 1 11:30 Session Chair: John Micco Session Chair: Wasif Afzal Session Chair: Andy Podgurski 12:00 12:30 Lunch

14:00 Research 2 Research 3 Tool 1 Regression and Testing & Debugging 2 14:30 Continuous Integration Session Chair: Rene Just Session Chair: Franz Wotawa 15:00 Session Chair: Robert Feldt 15:30 Coffee Break 16:00 Research 4 Research 5 Journal First Mobile and Web Program Repair and Synthesis 16:30 Application Testing Session Chair: Neil Walkinshaw Session Chair: Jeff Offutt 17:00 Session Chair: Emil Alégroth

TESTOMAT meeting: Room 108

Keynote 1: Infer: Static Analysis at Scale Tuesday, April 10th • 09:30 • Congess Hall

Dulma Churchill / Facebook UK Infer: Static Analysis at Scale Infer is a static analysis tool being developed at Facebook. It helps developers find issues in their code early, at code review time. It finds issues such as Null Dereferences, Memory Leaks, and many more, in a range of languages such as Java, C, C++ and Objective-C. In this talk we will cover how Infer is deployed at Facebook and the challenges to handle the scale of Facebook’s code, as well as the differences between static analysis and testing. We will also cover the basics of Infer’s architecture, and the range of checkers it consists of. Originally it was based on separation logic. Now, additionally, there are various other checkers based on abstract interpretation such as buffer overflow and lock consist- ency checkers.

Biography Dulma Churchill is a Software Engineer at Facebook, London. She has been developing Infer for over five years and worked on many areas of the tool: frontend, backend, build system integrations, etc. Before, she did her PhD at the University of Munich in resource analysis for Java using type systems.

12 ICST conference program Wednesday, April 11th

09:00 MIP #1 09:25 Track 1 Keynote 2: But is it fun? Magnus Nordin/David King/Stefan Posthuma (Congress Hall)

10:20 Track 1 Coffee Break Congress Hall Room 107 Room 108

10:50 Research 6 Research 7 Doctoral symposium 1 Visual Testing Search Based Approaches 11:20 Session Chair: Atif Memon Session Chair: Richard Torkar 11:50 12:20 Lunch 14:00 Research 8 Tool 2 Doctoral Symposium 2 Testing & Debugging 3 14:30 Session Chair: Francisco Gomes Session Chair: Rene Just 15:00 15:30 Coffee Break 16:00 Demo Industry Track 2 Doctoral Symposium 3 16:30 Session Chair: John Hughes 17:00 Panel Discussion: When are Software Testing Research Contributions, Real Contributions? Prof. Lionel Briand (Univ Luxembourg) Dr. Markus Borg (RISE SICS) Dr. Kristian Wiklund (Ericsson)

Session chairs: Sigrid Eldh, Elaine Weyuker

Keynote 2: But is it fun? Wednesday, April 11th • 09:25 • Congess Hall

But is it fun? Modern video game development is very complex with teams of hundreds of people in locations across the globe collaborating on a product that will get released to a demanding and vocal audience of millions of gamers. Quality Verification for game projects includes testing performance and stability of game clients on consoles, load balancing large back-end service systems and the collection and analysis of social media content to gauge the sentiment of our players. At Electronic Arts we spend thousands of hours testing our software to ensure our players have the best possible experience Magnus Nordin / when playing our games. Ultimately, we strive to answer the most elusive question of all: “is it fun?” Electronic Arts Biographies Magnus Nordin is a Technical Director of the AI group with the SEED R&D team at Electronic Arts. Magnus has spent 25 years doing computer science and software engineering in a large number of projects and companies. He is currently heading up the deep learning and AI research team of SEED, an EA R&D division. David King is a Technical Director of the Quality Engineering group at DICE. David is responsible for software driven testing of the DICE game franchises, most notably Star Wars Battlefront and Battle- David King / DICE field. He has worked in the field of software testing at Electronic Arts for the last 9 years. Stefan Posthuma is a Senior Director of Quality Engineering for EA’s European Studios. Ever since publishing his first game back in 1985, Stefan has worked on (and played!) countless of video games. He joined EA in 1998 and has worked with EA’s global Quality Verification group for the last 4 years. Magnus, David and Stefan are all located at EA’s studio in Stockholm, Sweden.

Stefan Posthuma / Electronic Arts

13 KeynoteICST conference Talk program Thursday, April 12th

09:30 Track 1 Keynote 3: Safety Verification for Deep Neural Networks Marta Kwiatkowska (Congress Hall) 10:00 10:30 Track 1 Coffee Break Congress Hall Auditorium

11:00 Research 9 Research 10 Defect Analysis Mutation Analysis 11:30 Session Chair: Michael Felderer Session Chair: José Miguel Rojas 12:00 12:30 Lunch

14:00 Open Steering Commitee & Closing Room 109 14:30 15:00

Keynote 3: Safety Verification for Deep Neural Networks Thursday, April 12th • 09.30 • Congress Hall

Prof. Marta Kwiatkowska / University of Oxford Safety Verification for Deep Neural Networks Deep neural networks have achieved impressive experimental results in image classification, but can surprisingly be unstable with respect to adversarial perturbations, that is, minimal changes to the input image that cause the network to misclassify it. With potential applications including perception modules and end-to-end controllers for self-driving cars, this raises concerns about their safety. This lecture will describe progress with developing a novel automated verification framework for deep neural networks to ensure safety of their classification decisions with respect to image manipulations, for example scratches or changes to camera angle or lighting conditions, that should not affect the classification. The techniques work directly with the network code and, in contrast to existing methods, can offer guarantees that adversarial examples are found if they exist. We implement the techniques using Z3 and evaluate them on state-of-the-art networks, including regularised and deep learning networks. We also compare against existing techniques to search for adversarial examples.

Biography Marta Kwiatkowska is Professor of Computing Systems and Fellow of Trinity College, University of Oxford. Prior to this she was Professor in the School of Computer Science at the University of Birmingham, Lecturer at the University of Leicester and Assistant Professor at the Jagiellonian University in Cracow, Poland. Kwiatkowska has made fundamental contributions to the theory and practice of model checking for probabilistic systems, focusing on automated techniques for verification and synthesis from quan- titative specifications. She led the development of the PRISM model checker (www.prismmodelchecker.org), the leading software tool in the area and winner of the HVC Award 2016. Probabilistic model checking has been adopted in many diverse fields, includ- ing distributed computing, wireless networks, security, robotics, game theory, systems biology, DNA computing and nanotech- nology, with genuine flaws found and corrected in real-world protocols. Kwiatkowska awarded an honorary doctorate from KTH Royal Institute of Technology in Stockholm in 2014 and the Royal Society Milner Medal in 2018. Her recent work was supported by the ERC Advanced Grant VERIWARE “From software verification to ‘everyware’ verification” and the EPSRC Programme Grant on Mobile Autonomy. She is a Fellow of ACM and Member of Academia Europea.

14 Keynote Talk

Venue

Room 115 Room 114

Room 113 Auditorium “Hörsalen” Room 112 Room 111 Room 117

Room 118

Elevator Small Lecture Hall “Lilla Salen“ Delivery Lift

70 m2 Elevator

Toilets Room 110 Toilets

Toilets

Room 109 Room 101 Exhibition Area

Room 102 Room 108 Conference Registration Room 103 Reception Room 107 Prof. Marta Kwiatkowska / University of Oxford Elevator VIP Room “Misteln“ Room 106

Room 105

Skyway to Parking House, 1 500 spaces

Delivery Lift

15 Keynote Talk

Delivery Lift

Room 212 Toilets Room 211 Room 210 Toilets Room 209 Congress Hall, Stalls Room 208 Room 207 Room 206 Room 205 Exhibition Area

Room 204

Elevator Room 203 (can be divided) Room 202 (can be divided) Room 201

2

Delivery Lift

Toilets Room 306

Room 305 Toilets

Room 304

Room 303

Room 302 Elevator

Room 301

16 SASTKeynote workshop Talk Monday, April 9th

SAST, Swedish Association for Software Testing Quarterly meeting: Research and Future...... 18

Workshops Monday, April 9th

ITEQS Message from the General Chairs...... 20 Organization...... 21 Detailed Program...... 36 MUTATION Message from the General Chairs...... 22 Organization...... 23 Detailed Program...... 39 NEXTA Message from the General Chairs...... 24 Organization...... 26 Detailed Program...... 38 USE Message from the General Chairs...... 37 VVIoT Message from the General Chairs...... 27 Organization...... 27 Detailed Program...... 37

2

Delivery Lift Workshops Friday, April 13th Toilets Room 306 A-MOST Room 305 Toilets Message from the General Chairs...... 28

Room 304 Organization...... 29 Detailed Program...... 47 Room 303 InSTA Message from the General Chairs...... 30 Organization...... 30 Detailed Program...... 45 IWCT Room 302 Elevator Message from the General Chairs...... 31 Organization...... 33 Room 301 Detailed Program...... 45 TAIC PART Message from the General Chairs...... 34 Organization...... 35 Detailed Program...... 46

17 th KeynoteSAST Research Talk and Future Monday, April 9 • Room: Auditorium (Hörsalen)

Swedish Association for Software Testing SAST was formed the 1995, by Anna Eriksson and Sigrid Eldh. The association was formed to create a forum where the members can exchange experiences and knowledge within software testing. The mission is to mediate knowledge within software testing as well as the neighbor disciplines. SAST is an association for anyone interested in the testing skill set and the increasing interest for the area has made the member count increasing as well. Four times a year SAST is arranging full day seminars for the members. Presenters from all over the world are invited to share their knowledge. SAST is a non-profit organization and all board members work without payment. The prize for the meetings equals the cost of arranging the meeting and the presenters are paid by the meeting sponsors. www.sast.se

Testing of critical system characteristics

09:15 Welcome and Introduction

09:30 A brief background of ICST, Robert Feldt

09:45 Test Automation - Beyond Automation of Test Execution, Per Erik Strandberg 10:30 Coffee

11:00 How Do Automatically Generated Unit Tests Influence Software Maintenance? José Miguel Rojas

11:30 I know where you’re hiding, Elaine Weyuker 12:00 Walk and talk

12:30 Lunch

14:00 End-to-End Web Testing: Current Practice and Future Technologies, Paolo Tonella and Filippo Ricca 14:30 Crashing simulated planes is cheap? Can simulation detect robotics bugs early? Chris Timperley

15:00 It Is Great That We Automate Our Tests, But Why Are They So Bad? Jeff Offutt 15:30 Coffee

16:00 Detecting and Debugging Flaky Tests, Jonathan Bell 16:30 Summary of the day and mingle

17:30- SAST/ICST joint social event Cue sports, bowling, sport simulators, 21:00 mingle and informal dinner (burger) at restaurant Strike

Per Erik Strandberg Test Automation - Beyond Automation of Test Execution Westermo develops robust networking solutions for industrial systems, and has for many years been working on automating different parts of the build and test process. Today, in addition to test execution, Westermo has also automated test selection for regression testing and mapping of test cases to physical test systems. Per Erik Strand- berg will tell you about Westermo’s test automation, focusing on how to use graph theory to avoid identical tests to be repeated over and over again. Per has a long experience in software development and testing and is currently an industrial PhD student at Westermo R&D and Mälardalen University.

18 th SAST Research and Future Monday, April 9 • Room: Auditorium (Hörsalen)

José Miguel Rojas How Do Automatically Generated Unit Tests Influence Software Maintenance? Automatically generated test cases save time and can lead to better test coverage. However, these test cases are often more difficult to understand and maintain. José Miguel Rojas will present results from a study comparing the maintenance effort required for manually written versus automatically generated test cases. José obtained his PhD from the Technical University of Madrid, worked as a Research Associate at the University of Sheffield. He is currently an Assistant Professor at the University of Leicester, where he focuses on test automation, gamification of testing and empirical studies of software engineering.

Elaine Weyuker I know where you are hiding When validating a software system, wouldn’t it be valuable to know in advance which files are most likely to contain the largest numbers of bugs? Elaine Weyuker has been working on fault prediction in large telecom systems for many years at AT&T. In this work, she has developed statistical methods that can accurately predict the most error- prone files in major software systems. Elaine is now a Guest Professor at Mälardalen University in Västerås and at the University of Central Florida. She has previously also worked as a professor at New York University.

Paolo Tonella and Filippo Ricca End-to-End Web Testing: Current Practice and Future Technologies End-to-End web testing aims to test fully integrated web applications in a production-like environ- ment. In this talk, Paolo Tonella and Filippo Ricca will describe results from their experimental re- search in this area, and how these results have led to new web testing tools. Filippo is a lecturer

Paolo Tonella at the University of Genoa and Paolo is a professor at the Fondazione Bruno Kessler in Trento, Filippo Ricca both in Italy. Their work in web application testing has been awarded the highly prestigious ICSE Most Influential Paper award, retrospectively selected 10 years after the research has been published.

Chris Timperley Crashing simulated planes is cheap? Can simulation detect robotics bugs early? As robotics systems become ubiquitous, the need for methods to ensure the reliability of these systems increases. In this presentation, Chris Timperley will present the results from a study on software failures in a major robotics project, where the main conclusion is that many errors can be relatively easily recreated in a simulated environ- ment. Chris received his PhD at the University of York, and now works as a postdoctoral researcher with Professor Claire Le Goues at Carnegie Mellon University. His research focuses on methods for automatically detecting, detecting and debugging software failures in large-scale, real-world software systems.

Jeff Offutt It’s Great That We Automate Our Tests, But Why Are They So Bad? TOCSYC is a five-year Swedish research project focusing on model-based testing. This presentation briefly describes the project, and then discusses challenges with test automation in the context of model-based testing. Jeff Offutt is a Professor of Software Engineering at George Mason University, a pioneer in model and mutation-based testing, and one of two authors of the book Introduction to Software Testing - one of the most widely used textbooks in test education. Jeff is also one of the founders of the ICST conference.

Jonathan Bell Detecting and Debugging Flaky Tests Flaky tests are test cases whose results are not consistent over time, even when the software under test is not changed. These test cases can be very frustrating, partly because developers cannot assume that a falling test case depends on the latest changes made to the software. Jonathan Bell will discuss several of his recent projects in flaky test detection, including a tool for automatic detection of flaky tests. Jonathan received his PhD from Columbia University in 2016 and currently serves as an Assistant Professor in Software Engineering and Software Systems at George Mason University.

19 Workshops

Message from the Workshop Chairs ITEQS 2018

Mehrdad Saadatmand Birgitta Lindström Bernhard K. Aichernig RISE SICS University of Skövde Graz University of Västerås, Sweden Skövde, Sweden Technology [email protected] [email protected] Graz, Austria [email protected]

Testing a system with respect to extra-functional properties poses specific challenges as traditional methods and approaches for testing the system’s functional correctness may not apply. A few examples of such challenges are: fault localization, the need to have appropriate techniques for different types of extra- functional properties, the role and impact of the environment when testing, observability and testability issues. ITEQS provides a well-focused forum with the goal of bringing together researchers and practitioners to share ideas, identify challenges, propose solutions and techniques, and in general expand the state of the art in testing EFPs and quality characteristics of software systems and services. ITEQS 2018 attracted 8 submissions (1 withdrawn). Each paper underwent a careful review and discussion process by at least three members of the program committee. Based on the reviews, 5 papers were selected for publication and presentation at the workshop. ITEQS 2018 program includes keynotes by Konstantinos Sagonas and Brian Nielsen, who both kindly accepted our invitation. Konstantinos gives a keynote on testing for concurrency bugs and Brian gives a keynote where he discusses how we may achieve a methodology for compositional testing of real-time systems. Finally, we would like to thank our general sponsors, the publicity chairs, the program committee, the external reviewers, the keynote speakers, the authors and participants. You all contribute to the quality and success of this workshop. Welcome to the workshop! We hope you will enjoy it.

20 Workshops

ITEQS 2018ITEQS Committees 2018 Committees

Program CommitteeProgram Committee Antonia Bertolino,Antonia CNR Bertolino, (National CNR Research (National Council), Research Italy Council), Italy Brian Nielsen,Brian Aalborg Nielsen, University, Aalborg Denmark University, Denmark Lionel Briand,Lionel University Briand, of UniversityLuxembourg, of Luxembourg, Luxembourg Luxembourg Mohammad Mousavi,Mohammad University Mousavi, of UniversityLeicester, UKof Leicester, UK Vahid Garousi,Vahid University Garousi, of UniversityLuxembourg, of Luxembourg, Luxembourg Luxembourg Jeff Offutt, GeorgeJeff Offutt, Mason George University, Mason USA University, USA Yvan Labiche,Yvan Carleton Labiche, University, Carleton Canada University, Canada Vittorio Cortellessa,Vittorio University Cortellessa, of UniversityL’Aquila, Italyof L’Aquila, Italy Markus Bohlin,Markus RISE Bohlin, SICS Västerås, RISE SICS Sweden Västerås, Sweden Björn Lisper, BjörnMälardalen Lisper, University, Mälardalen Sweden University, Sweden Mika Mäntylä,Mika University Mäntylä, of UniversityOulu, Finland of Oulu, Finland Hadi Hemmati,Hadi University Hemmati, of UniversityCalgary, Canada of Calgary, Canada Bestoun S. Ahmed,Bestoun Czech S. Ahmed, Technical Czech Un Technicaliversity in UnPrague,iversity Czech in Prague, Republic Czech Republic James H. Hill,James Indiana H. University-PurdueHill, Indiana University-Purdue University Indianapolis, University Indianapolis,USA USA Wasif Afzal, WasifMälardalen Afzal, University, Mälardalen Sweden University, Sweden Pasqualina Potena,Pasqualina RISE Potena, SICS Västerås, RISE SICS Sweden Västerås, Sweden Heinz Schmidt,Heinz RMIT Schmidt, University, RMIT Australia University, Australia

Publicity Publicity Nils Müllner,Nils Mälardalen Müllner, University, Mälardalen Sweden University, Sweden Sahar Tahvili,Sahar RISE Tahvili, SICS, Västerås, RISE SICS, Sweden Västerås, Sweden

General SponsorsGeneral Sponsors TOCSYC: TestingTOCSYC: of Critical Testing System of Critical Characteristics System Characteristics Mälardalen UniversityMälardalen University RISE SICS VästeråsRISE SICS Västerås

13 13 21

Workshops

Message from the General Chairs Mutation 2018

It is our pleasure to welcome you to the 13th edition of the International Workshop on Mutation Analysis (Mutation 2018), collocated with the 11th IEEE International Conference on Software Testing, Verification and Validation (ICST 2018) in Västerås, Sweden. Since its first edition, the Mutation workshop has provided a forum to bring together researchers and practitioners, enabling them to exchange ideas, address fundamental challenges in mutation analysis and testing and discuss new applications of mutation.

It is our pleasure to accept seven, high-quality papers, after the corresponding reviewing and discussion process among the program committee members (2 full papers, 1 industry paper, and 4 short papers). The topics of the papers range from new mutation tools to theoretical and practical advances in the field.

Mutation 2018 features two keynotes focused on both academia and practice. Our first keynote speaker is Prof. John Clark from the University of Sheffield who is going to discuss about the past and future of mutation, elaborating on challenges and future areas that mutation can be leveraged and applied. Our second keynote speaker is Goran Petrovic from Google Zurich who is going to present his experiences in implementing and deploying mutation at scale, leveraging Google’s software development infrastructure.

Finally, we would like to thank the program committee and the external reviewers for their contributions, and the authors and participants for assuring the quality and success of this workshop.

Enjoy the workshop!

10 22

Workshops

MutationMutation 2018 Committees 2018 Committees Program Chairs Program Chairs Marinos Kintis, University of Luxembourg Marinos Kintis, University of Luxembourg Nan Li, Medidata Solutions Nan Li, Medidata Solutions José Miguel Rojas, University of Leicester José Miguel Rojas, University of Leicester

Program Committee Program Committee Birgitta Lindström, University of Skövde (Sweden) Birgitta Lindström, University of Skövde (Sweden) Goran Petrovic, Google Switzerland GmbH (Switzerland) Goran Petrovic, Google Switzerland GmbH (Switzerland) Gordon Fraser, Universität Passau (Germany) Gordon Fraser, Universität Passau (Germany) Gregory Kapfhammer, Allegheny College (USA) Gregory Kapfhammer, Allegheny College (USA) Jens Krinke, University College London (UK) Jens Krinke, University College London (UK) Jie Zhang, Peking University (China) Jie Zhang, Peking University (China) José Miguel Rojas, University of Leicester (UK) José Miguel Rojas, University of Leicester (UK) Lech Madeyski, Wroclaw University of Technology (Poland) Lech Madeyski, Wroclaw University of Technology (Poland) Lin Deng, Towson University (USA) Lin Deng, Towson University (USA) Marinos Kintis, University of Luxembourg (SnT) (Luxembourg) Marinos Kintis, University of Luxembourg (SnT) (Luxembourg) Mike Papadakis, University of Luxembourg (Luxembourg) Mike Papadakis, University of Luxembourg (Luxembourg) Milos Gligoric, The University of Texas at Austin (USA) Milos Gligoric, The University of Texas at Austin (USA) Nan Li, Medidata Solutions (USA) Nan Li, Medidata Solutions (USA) Nicos Malevris, Athens University of Economics and Business (Greece) Nicos Malevris, Athens University of Economics and Business (Greece) Paul Ammann, George Mason Unversity (USA) Paul Ammann, George Mason Unversity (USA) Rahul Gopinath, Saarland University ( Germany) Rahul Gopinath, Saarland University ( Germany) Rene Just, University of Massachusetts Amherst (USA) Rene Just, University of Massachusetts Amherst (USA) Sina Shamshiri, The University of Sheffield (UK) Sina Shamshiri, The University of Sheffield (UK) Sudipto Ghosh, Colorado State University (USA) Sudipto Ghosh, Colorado State University (USA) Vinicius Durelli, University of São Paulo (Brazil) Vinicius Durelli, University of São Paulo (Brazil) Yue Jia, University College London and Facebook (UK) Yue Jia, University College London and Facebook (UK) Yves Le Traon, University of Luxembourg (Luxembourg) Yves Le Traon, University of Luxembourg (Luxembourg)

Subreviewers Subreviewers Thierry Titcheu Chekam, University of Luxembourg Thierry Titcheu Chekam, University of Luxembourg

11 11 23

Workshops

The 1st IEEE Workshop on the Next Level of Test Automation (NEXTA 2018) - From the Program Chairs

Sigrid Eldh1, Serge Demeyer2, Markus Borg3, and Adnan Causevic4

I. WELCOME FROM THE CHAIRS background. All papers received four reviews and based on that the workshop organizers accepted eleven papers, Welcome to the 1st IEEE Workshop on the Next Level resulting in an acceptance ratio of 11/15 = 73%. The key to of Test Automation (NEXTA 2018), co-located with the a successful workshop is to focus on quality papers, not on 11th IEEE Conference on Software Testing, Verification and the acceptance number – to yield a good discussion. Authors Validation (ICST 2018) in Vaster¨ as,˚ Sweden. were encouraged to incorporate the reviews into a second Test automation has been an acknowledged software engi- iteration of their papers, which resulted in the papers you neering best practice for years. However, the topic involves are reading in this volume. In addition, we made the format more than the repeated execution of test cases that often open, suggesting both tool papers and video submission in comes first to mind. Simply running test cases using a unit addition to experience reports and research papers. testing framework is no longer enough for test automation We would like to express our gratitude to the program to keep up with the ever-shorter release cycles driven by committee members for their efforts in providing very thor- continuous deployment and technological innovations such ough evaluations of the submitted papers. We also wish as microservices and DevOps pipelines. Now test automation to thank all contributors and participants for sharing their needs to rise to the next level by going beyond mere test ex- practice and real-world experiences. Finally, we owe special ecution. The NEXTA workshop will explore how to advance thanks to the local organization committee of the ICST test automation to further contribute to software quality in conference for their support. In addition – we would like the context of tomorrow’s rapid release cycles. Take-aways to thank the EUREKA ITEA3 and Vinnova for the funding for industry practitioners and academic researchers will en- of the TESTOMAT Project that allowed us to make this compass test case generation, automated test result analysis, workshop happen. test suite assessment and maintenance, and infrastructure for the future of test automation. II. BACKGROUND –THE TESTOMAT PROJECT As the topic is highly relevant for both research and The TESTOMAT Project is an EUREKA ITEA3 Project industry, the aim of the workshop is to attract both academic consisting of 6 countries and 35 Partners. The goal of the researchers and industry practitioners. Test automation has project is to help software teams to strike the right balance by been a hot topic in industry for years, and after the initial increasing the development speed without sacrificing quality hype many organizations have realized the limitations. We in the context of test automation in an agile context. As believe the time is right for a workshop on how to bring the TESTOMAT Project contains experts from the field of test automation to the next level. Thus, we particularly want test automation and is a mix of practitioners, academic to attract industry participants by focusing discussions on researchers and a series of solution providers, such as consul- problems and solutions relevant to industry practice. Our tants and tool vendors in the field, the industry drive is in the strategy is for the workshop to co-locate with major con- context of agile processes. This means the project is actively ference in the field, to reach a broad audience and to attract opening up the community – to challenge contemporary ways and disseminate among the best academic researchers and of working. In theory, everything can be automated in the test practitioners. This year, the decision was to co-locate with process, but still, the know-how of how far we can push the the 11th IEEE Conference on Software Testing, Verification limits is yet unsolved – and smart, cost-effective solutions and Validation (ICST 2018) in Vaster¨ as,˚ Sweden in April are still to be invented. 2018. The TESTOMAT Project is organized into four technical The workshop organizers broadcasted a call for papers work packages: 1) test effectiveness, 2) test prioritization, on various fora and received fifteen papers by January 19, 3) testing for quality standards, and 4) test automation 2018. These papers were reviewed by a program committee improvement model. Next, we briefly introduce the work consisting of twenty-one test experts with a heterogeneous packages to set the scene for NEXTA 2018.

1 S. Eldh is with Ericsson AB, Sweden. A. Test effectiveness [email protected] 2S. Demeyer is with University of Antwerp, Belgium. Industry acknowledges software testing as a critical but [email protected] expensive software engineering activity, thus companies are 3M. Borg is with RISE SICS AB, Sweden [email protected] 4A. Causevic is with Malardalen¨ University, Sweden. aware of the importance of test effectiveness — including [email protected] the effectiveness of automated testing. However, while being

24 Workshops

considered important in industry, the approaches to assess will be published as an eBook to ensure easy access for and improve the effectiveness of automated testing are often the target audience. Inspired by similar improvement models ad-hoc or based on simplistic measures. To improve state-of- from the past (e.g., CMMI and TMMI) TAIM will define practice test effectiveness TESTOMAT will make contribu- key improvement areas in test automation, with the focus on tions in five complementary themes: i) test maintenance, ii) measurable improvement steps [Eldh2014]. In that sense, we test result analytics, iii) code coverage, iv) mutation testing, explicitly avoid human interpretation via external auditors and v) GUI testing. assessing the current test automation level by means of interviews of key persons in the testing team. Instead the B. Test prioritization model will exploit the presence of continuous integration Due to limited test resources, it is often necessary to tools and tap into the logs to get quantitative data about prioritize and select test cases for a given system under the effectiveness of the tests. The actual best practices to test. In addition to existing prioritization techniques, the improve the test automation level will be documented in a TESTOMAT Project seeks to develop new prioritization pattern catalog. goals. Prioritization sources are links to artifacts (e.g., re- quirements), metrics per test cases (e.g., cost, time, and risk), III. WORKSHOP THEME AND TENTATIVE PROGRAM metrics from system under test code (e.g., coverage, results Partly driven by agile development, partly driven by tech- from static/dynamic analysis, and version control commits), nological innovations, software is being deployed in ever- the test-run-feedback loop (e.g., recent and past failures), and shorter release cycles. Software teams all over the world tend even increased test coverage over time (e.g., many pairwise to embrace faster release cycles as “the” way to incorporate runs eventually lead to 3-wise coverage). Prioritization shall customer feedback into product development processes. A then be used for test selection. This can be post-selection (in much faster release cycle implies rethinking the typical view terms of re-sorting existing test suites), guided generation of on software quality. prioritized test suites, selection and ordering at test execution High-quality software has long been synonymous with time, and even be used in conjunction with mutation testing. software “without bugs”. Today, however, quality software has come to mean “easy to adapt” because of the constant C. Testing for quality standards pressure to change. Consequently, modern software teams Software often needs to be compliant with quality stan- seek for a delicate balance between two opposing forces: dards. These standards include generic quality standards striving for reliability and striving for agility. The old view (e.g., ISO/IEC 25010 and 25023) and domain specific does not match well with the challenges trusted on today’s safety standards (e.g., RTCA/DO-178C and ISO 26262). software testing teams. Traditional quality assurance was The TESTOMAT project will identify and provide a set of based on strict processes, comprehensive documentation, metrics and test techniques to assess the functional and non- and rigorous planning; valuable assets yet corresponding functional properties and quality characteristics of software- to a development process optimized for perfection. Agile intensive systems. Based on such quality metrics, the goal software development, on the other hand, forwards a series is to determine if and to what extent different standards of other values: customer value, iterative and incremental are fulfilled, and also evaluate the quality characteristics development, continuous integration, hence optimizes for of the system in terms of performance, safety, reliability, ease of change. maintainability, etc. To enable such evaluations, appropriate The next level of test automation must facilitate future mechanisms such as traceability between test cases, require- software teams’ strive to increase the development agility ments and code artifacts will also be studied. Moreover, without sacrificing quality. To achieve this, test automation the TESTOMAT Project will also look into process aspects tools and frameworks must seamlessly integrate with the related to testing non-functional properties in a continuous rapid delivery pipelines of the future – test automation must integration environment, including the efficiency of test gen- advance to the next level to keep up. eration and execution after design refinements. The focus Automating testing is a challenge. Practitioners often think will be to implement measurements that can automatically of test execution automation and researchers in academia identify a non-functional quality through test case analysis tend to focus on automating the test case generation – the verdicts, and to automatically analyze them over time. test design. NEXTA 2018 is a workshop that focuses on the next level – taking test automation focus beyond the current D. Test automation improvement model scope. The DevOps context brings continuous testing and The TESTOMAT Project will develop a stepwise improve- continuous delivery, which also includes testing back-end ment ladder for industries based on the Test Automation Im- aspects. One example is closing the loop by automatic fault provement Model (TAIM) [1]. This will become a handbook localization and automatic program repair. NEXTA 2018 that will serve as a reference guide, listing best practices, targets automation of all aspects in the test process, from and contain measurements to automatically assess whether automatic test requirements capturing, real-time analysis to the next level of TAIM has been reached. The TESTOMAT adapt the testing, new techniques that embrace automatic Project will measure the software testing teams active within tests, and challenges in testing both simple and complex the consortium, and utilize their experiences. The handbook systems. There is clearly a need for a new workshop on 25 Workshops

the subject, focusing on making efficient and effective test IV. W ORKSHOP ORGANIZATION automation. The roles of the NEXTA 2018 workshop organization To elaborate on the next level of test automation, we follow: accepted 11 papers to NEXTA 2018, organized into four Sigrid Eldh, general chair topics: • Serge Demeyer, program co-chair • Automated GUI Testing Markus Borg, program co-chair • • – Challenges in Automated Testing through Graphi- Adnan Caucevic, program co-chair • cal User Interface (Pekka Aho and Tanja Vos) Furthermore, the NEXTA 2018 was made possible thanks – Introducing automated GUI testing and observing to reviews by 20 Program Committee members from around its benefits: an industrial case study in the con- the world: text of law-practice management software (Vahid Alessandra Bagnato, Softeam, France Garousi and Erdem Yildirim) • Benoit Baudry, Royal Institute of Technology, Sweden Maintenance of Android Widget-based GUI Test- • – Hyunsook Do, University of North Texas, US ing: A Taxonomy of test case modification reasons • Leire Etxeberria Elorza, University de Mondragon, (Riccardo Coppola, Marco Torchiano and Maurizio • Spain Morisio) Michael Felderer, University of Innsbruck, Austria Automatically Generating Test Scripts for GUI • – Vahid Garousi, Wageningen University, Netherlands Testing (Toshiyuki Kurabayashi, Muneyoshi Iyama, • Andrew V. Jones, Vector Software, UK Hiroyuki Kirinuki and Haruto Tanno) • Peter M. Kruse, Assystem, Germany • Automated Testing in the Context of Embedded Sys- Yvan Labiche, Carleton University, Canada • • tems Bernard Legeard, Smartesting, France • – Design of High Confidence Embedded Software Mika Mantyl¨ a,¨ University of Oulu • Hardware-in-loop Simulation Test Platform Based Karl Meinke, The Royal Institute of Technology, Swe- • on Hierarchical Model (Gao Feng and Deng Fei) den Test maintenance Ali Parsai, University of Antwerp, Belgium • • Damiano Torre, Carleton University, Canada – Uncover Your Tracks: Using CodeScene to Su- • Tanja Vos, Universitat Politecnica de Valencia, Spain pervise the Quality of Automated Tests (Adam • Tornhill). and Open University, Netherlands Marc-Florian Wendland, Fraunhofer Fokus, Germany – Evaluating Test Data Generation for Untyped Data • Elaine Weyuker, Rutger University, US and Malardalen¨ Structures Using Genetic Algorithms (Ralf Gerlich • and Christian Praus) University, Sweden Kristian Wiklund, Ericsson AB, Sweden – Feature based testing by using model synthesis, test • Franz Wotawa, Graz University of Technology, Austria generation and parameterizable test prioritization • Tao Yue, Simula Research Laboratory, Norway (Martin Reider, Stephan Magnus and Jan Krause) • – Software metrics for policy-driven SDLC automa- V. ACKNOWLEDGEMENTS tion (Leonid Borodaev, Rix Groenboom, Rein Smedinga and Alex Telea) Thanks go to the participants of the workshop and all the authors of submitted papers for their important con- Organizational impact of test automation • tribution to the event. In addition, we want to thank our – The Next Level of Test Automation: What About sponsors for the workshop; IEEE Computer Society, as well the Users? (Kristian Wiklund and Monika Wiklund) as the Swedish chapter in Region 8, The Eureka ITEA 3 – A Preliminary Study on TESTOMAT Project’s Im- TESTOMAT Project this time the workshop is funded in pact: Software Test Automation Acceptance and Sweden by Vinnova, the ICST Workshop organisers Dr. ¨ Use at a Company in Turkey (Ozlem Albayrak) Gregory Gay and Dr. Mike Papadakis as well as the entire ICST 2018 organsation for their support.

REFERENCES [1] S. Eldh, K. Andersson, A. Ermedahl, and K. Wiklund. Towards a test automation improvement model (TAIM). In Proc. of the 7th IEEE International Conference on Software Testing, Verification, and Validation Workshops, ICSTW’14, pp. 337–342, 2014.

26 KeynoteWorkshops Talk Welcome Message from the Chairs Welcome MessageVVIoT from2018 the Chairs Welcome Message from the Chairs VVIoT 2018 VVIoT 2018 Verification and Validation of Internet of Things (IoTs) is an extraordinarily challenging and complex

topic.Verification Thus, anda forum Validation is needed of Internet for academics, of Things (IoindustrialTs) is an researchers, extraordinarily and challenging practitioners and to complex discuss topic.challenges,Verification Thus, shareanda forum Validation experiences, is needed of Internet and for advanceacademics, of Things verification (IoindustrialTs) is an andresearchers, extraordinarily validation and of challenging practitionersIoTs. VVIoT and to complex aimsdiscuss to challenges,topic.provide Thus, such share a forum.forum experiences, is needed and for advanceacademics, verification industrial andresearchers, validation and of practitionersIoTs. VVIoT to aimsdiscuss to providechallenges, such share a forum. experiences, and advance verification and validation of IoTs. VVIoT aims to This is the first edition of VVIoT 2018 and has four papers accepted after rigorous peer reviews. In provide such a forum. Thisaddition, is the the first workshop edition isof featuredVVIoT 2018with twoand excitinghas four keynotespapers accepted given by after Lionel rigorous Briand peer (University reviews. Inof addition,ThisLuxembourg) is the the first workshop and edition Christian isof featuredVVIoT Prehofer 2018with (Fortiss, twoand excitinghas Germany). four keynotespapers Both accepted givenof the by keynoteafter Lionel rigorous speakers Briand peer (Universityare reviews. the world Inof Luxembourg)addition,leaders in the IoTs workshop and and Christian Cyber-Physical is featured Prehofer withSystems. (Fortiss, two excitingWe Germany). will keynotesalso Bothhave given ofa specialthe by keynote Lionel session speakers Briand dedicated (Universityare tothe present world of leadersLuxembourg)relevant in National/ IoTs and and European/InternatiChristian Cyber-Physical Prehofer Systems. onal(Fortiss, projects We Germany). will relevant also Both haveto the ofa workspecialthe keynoteshop session topics. speakers dedicated We aimare totheto presenttrigger world relevantleadersinteresting in National/ IoTs discussions, and European/Internati Cyber-Physical generate new Systems.onal ideas, projects We and will relevant build also future haveto the a workcollaborationsspecialshop session topics. ondedicated We verification aim toto presenttrigger and interestingrelevantvalidation National/ ofdiscussions, IoTs. European/Internati generate newonal ideas, projects and relevant build future to the workcollaborationsshop topics. on We verification aim to trigger and validationinteresting ofdiscussions, IoTs. generate new ideas, and build future collaborations on verification and We look forward to seeing you in Västerås, Sweden. validation of IoTs. We look forward to seeing you in Västerås, Sweden.

We look forward to seeing you in Västerås, Sweden.

Shaukat Ali, Tao Yue, and Rui Abreu

ShaukatVVIoT 2018Ali, Tao Chairs Yue, and Rui Abreu ShaukatVVIoT 2018Ali, Tao Chairs Yue, and Rui Abreu

VVIoT 2018 Chairs VVIoT Organizing Committees

Program Chairs

Shaukat Ali, Simula Research Laboratory (Norway)

Tao Yue, Simula Research Laboratory (Norway)

Rui Abreu, Instituto Superior Técnico (IST) of the University of Lisbon (Portugal)

Program Committee

Benoit Baudry, Royal Institute of Technology (Sweden)

Bruno Legeard, University of Franche-Comté and Smartesting (France)

Tingting Yu, University of Kentucky (USA)

Vasco Amaral, Universidade Nova de Lisboa (Portugal) Simone Cirani, University of Parma (Italy)

Bruno Lima, University of Porto (Portugal)

Shuai Wang, Simula Research Laboratory (Norway) João Pascoal Faria, University of Porto (Portugal)

Darko Marinov, University of Illinois (USA) Dan Hao, Peking University (China)

Bao Nguyen, Google (USA)

Markus Schacher, KnowGravity (Switzerland) Tian Zhang, Nanjing University (China)

Alessandra Bagnato, SOFTEAM (France)

Zohaib Iqbal, QUEST Laboratory (Pakistan)

Sarfraz Khurshid, University of Texas (USA) Annibale Panichella, University of Luxembourg (Luxembourg) Neil Walkinshaw, University of Leicester (UK) Hironori Washizaki, WASEDA University (Japan)14

Ji Wu, Beihang University (China) 14 14 27

15

KeynoteWorkshops Talk

Welcome Message from the General Chairs A-MOST 2018

Welcome to the 14th edition of the workshop on Advances in Model-based testing Testing (A-MOST 2018) held on April 13th, 2018 in Västerås, Sweden.

Context of A-MOST: The complexity of software systems is increasing rapidly and thus warrants the development of novel automated testing techniques that can manage the increased complexity of software. Model-Based Testing (MBT) is one such approach towards automated testing that allows managing the complexity of the Software under Test with the use of models. A recent survey shows that MBT has been increasing, being used both in academic and in industrial settings and thus still remains an interesting area of research. In the past 13 years, A-MOST has served as a good venue for interaction among model-based testing researchers from both academia and industry.

In 2018, A-MOST received six high quality submissions, and all of them were accepted after a thorough review and discussion process. Each submission was reviewed by three members of the program committee. The accepted papers are divided into two main sections, i.e., test cases generation and management and application of model-based testing that will be presented during the workshop.

Moreover, we are also delighted to have two well-renowned MBT researchers as keynote speakers: Prof. Jeff Offutt and Dr. Brice Morin. We thank them very much for accepting our invitations and for their inspiring keynote speeches. We also appreciate the effort made by our program committee members for providing their timely reviews with high quality. We would like to thank the A-MOST steering committee for their support and advice. Last but not least, we thank all the authors who submitted their high quality work to A-MOST 2018.

A-MOST 2018 Organisers: Paolo Arcaini, National Institute of Informatics, Japan Xavier Devroey, Delft University of Technology, The Netherlands João Pascoal Faria, University of Porto, Portugal

19 28

KeynoteWorkshops Talk

A-MOSTA-MOST 2018 2018 OrganizingOrganizing and and Program Program Committee Committee Members Members

GeneralGeneral Program Program chairs chairs PaoloPaolo Arcaini, Arcaini, National National Institu Institute of Informatics,te of Informatics, Japan Japan XavierXavier Devroey, Devroey, Delft DelftUniversity University of Technology, of Technology, The Netherlands The Netherlands João PascoalJoão Pascoal Faria, Faria, University University of Porto, of Porto, Portugal Portugal

SteeringSteering Committee Committee Rob Hierons,Rob Hierons, Brunel Brunel University, University, UK UK ManuelManuel Núñez, Núñez, Universidad Universidad Complutense Complutense de Madrid, de Madrid, Spain Spain AlexanderAlexander Pretschner, Pretschner, Technische Technische Universität Universität München, München, Germany Germany

ProgramProgram Committee Committee BernhardBernhard Aichernig, Aichernig, Graz GrazUniversity University of Technology, of Technology, Austria Austria ShaukatShaukat Ali, Simula Ali, Simula Research Research Laboratory, Laboratory, Norway Norway MoussaMoussa Amrani, Amrani, University University of Namur, of Namur, Belgium Belgium PaoloPaolo Arcaini, Arcaini, National National Institu Institute of Informatics,te of Informatics, Japan Japan Aitor AitorArrieta Arrieta Marcos, Marcos, Mondragon Mondragon University, University, Spain Spain KirillKirill Bogdanov, Bogdanov, University University of Sheffield, of Sheffield, UK UK FabriceFabrice Bouquet, Bouquet, Franche-Comte Franche-Comte Comté Comté University, University, France France FrédéricFrédéric Dadeau, Dadeau, Franche-Comte Franche-Comte Comté Comté University, University, France France XavierXavier Devroey, Devroey, Delft DelftUniversity University of Technology, of Technology, The Netherlands The Netherlands João PascoalJoão Pascoal Faria, Faria, University University of Porto, of Porto, Portugal Portugal SudiptoSudipto Ghosh, Ghosh, Colorado Colorado State StateUniversity, University, USA USA Rob Hierons,Rob Hierons, Brunel Brunel University, University, UK UK BrunoBruno Legeard, Legeard, Smartesting, Smartesting, France France Levi Lúcio,Levi Lúcio, Fortiss, Fortiss, Germany Germany MercedesMercedes Merayo, Merayo, Complutense Complutense University University of Madrid, of Madrid, Spain Spain BrianBrian Nielsen, Nielsen, Aalborg Aalborg University, University, Denmark Denmark ManuelManuel Núñez, Núñez, Complutense Complutense University University of Madrid, of Madrid, Spain Spain Jeff Offutt,Jeff Offutt, George George Mason Mason University, University, USA USA Ana Paiva,Ana Paiva, University University of Porto, of Porto, Portugal Portugal Mike MikePapadakis, Papadakis, University University of Luxembourg, of Luxembourg, Luxembourg Luxembourg IoannisIoannis Parissis, Parissis, University University of Grenoble of Grenoble INP, FranceINP, France GillesGilles Perrouin, Perrouin, University University of Namur, of Namur, Belgium Belgium AlexandreAlexandre Petrenko, Petrenko, CRIM, CRIM, Canada Canada José MiguelJosé Miguel Rojas, Rojas, University University of Leicester, of Leicester, UK UK ShuaiShuai Wang, Wang, Simula Simula Research Research Laboratory, Laboratory, Norway Norway FranzFranz Wotawa, Wotawa, Graz GrazUniversity University of Technology, of Technology, Austria Austria

ExternalExternal Reviewers Reviewers Tao Ma,Tao Simula Ma, Simula Research Research Laboratory, Laboratory, Norway Norway

20 20 29 KeynoteWorkshops Talk

Welcome Message from the Chairs InSTA 2018

It is our pleasure to welcome you to the 5th International Workshop on Software Test Architecture (InSTA 2018) collocated with the 10th IEEE International Conference on Software Testing, Verification and Validation (ICST 2018) in Västerås, Sweden.

The workshop aims to discuss software architecture between industry and academia. Design better software test architectures is important for software testing activities. The software test architecture is a key part of the test strategy. As software becomes larger and more complicated, we must increase its reliability and users' delights. The need for better testing calls for emphasizing higher-level test designs with specialized software test architectures. This higher level approach allows the design concepts and big picture of the software to be grasped and communicated, and the higher level of abstraction can increase the productivity and reusability. This makes it possible for the software testing to cope with the high reliability requirements of large and complex programs.

Test architectures must be approached indirectly as a part of the test strategies. Some organizations are working to establish new ways to design novel test architectures, but there is no unified understanding of the key concepts of test architectures. This workshop is intended to allow researchers and practitioners to comprehensively discuss the central concepts of test architectures. InSTA 2018 provides sessions of research, industry experiences and emerging idea papers about software test architecture.

We would like to thank the program committee to contribute the workshop. Please enjoy InSTA 2018.

InSTA 2018 Organization

General Chairs: Satoshi Masuda, IBM Research - Tokyo, Japan Yasuharu Nishi, The University of Electro-Communications, Tokyo, Japan

Program Committee:

Jon Hagar, Grand Software Testing, USA Tetsuro Katayama, University of Miyazaki, Japan Zhenyu Liu, Shanghai Key Laboratory of Computer Software Testing and Evaluation, China Kazuhiko Tsuda, University of Tsukuba, Japan Hironori Washizaki, Waseda University, Japan

21 30

22

Workshops

General Message from the IWCT Workshop Chairs

Dimitris E. Simos Rachel Tzoref-Brill SBA Research - Austria IBM Research - Israel [email protected] [email protected]

We would like to welcome you to the 7th International Workshop on Combinatorial Testing (IWCT 2018), which is organized in conjunction with the IEEE International Confer- ence on Software Testing, Verification and Validation (ICST 2018). After the successful edition in Montreal (2012), Luxem- bourg (2013), Cleveland (2014), Graz (2015), Chicago (2016), and Tokyo (2017), the seventh edition is held in Vasteras, on April 13th 2018. Combinatorial Testing (CT) is a widely applicable generic methodology and technology for software verification and val- idation, considered a testing best practice. In a combinatorial test plan, all interactions between parameters up to a certain level are covered. For example, in pairwise testing, for every pair of parameters, every pair of values will appear at least once. Studies show that CT is more efficient and effective than random testing. Fig. 1. Word cloud of the IWCT 2018 paper titles CT has gained significant interest in recent years, both in research and in practice. However, many issues still remain unresolved, and much research is still needed in the field. which 11 papers were accepted for publication – 6 full papers, For example, while pairwise testing is a well recognized and 4 short ones, and 1 extended abstract. The program contains popular test planning method, investigations of actual failures research papers on various aspects of CT, as well as on in a number of software and systems convincingly show that applications and use cases of CT. pairwise testing is usually not sufficient so high strength CT In addition to satisfiability-based approaches for the gener- (i.e., t-way for t>2) may be needed. ation of constrained locating arrays [6], which constitutes a In addition, the combinatorial test suites need to exclude new class of covering arrays, scheduling techniques for CT invalid combinations of test values that cannot be executed, taken from the field of Design of Experiments are presented which limits the degrees of freedom the algorithms have, in [2]. An approach to enumerate all valid t-tuples that can thus complicating the problem. Moreover, modeling languages appear in covering arrays is given in [8]. and tools for easily capturing the input test space are also Work on modelling and tools includes two new prototype required for real-life applicability of CT. Other obstacles for web frameworks which can allow easy processing of CT wide acceptance of CT in industry are the gap between the models and test generation capabilities [5] and advanced generated test plans and executable tests, and the difficulty in combinatorial analysis and measurement of test sets [7]. There determining expected results for the generated tests. Finally, is also novel work on CT with constraints for negative test empirical studies on CT, as well as thorough comparison with cases [4]. other methods are also required. This year, applications of CT and use cases constitute a IWCT focuses on many aspects of combinatorial testing, little less than half of the program. In particular, with regard with emphasis on the less explored ones. Advancing CT and to applications, the effect of time between events for sequence increasing its deployment in industry requires research on the interaction testing is demonstrated over a real-time system one hand and validation of usefulness and applicability of re- [11], while a practical amplification of condition/decision test search results in the field on the other hand. To this aim, IWCT coverage applicable to CT is described in [1]. Use cases for brings together practitioners and researchers that propose novel combinatorial testing include one approach for adapting unit models, methods, and algorithms for combinatorial interaction tests by generating combinatorial test sets [3] and a method testing. Figure 1 presents a word cloud taken from the titles of to combine CT with metamorphic testing which allows testing the accepted papers, and gives a rapid overview of the subjects of logic-based non-monotonic reasoning systems [10]. presented in the workshop. Last but not least, this year we accepted one extended This year we received 15 high quality submissions, from abstract for publication. In detail, an experience report is de-

31 Workshops

scribed in [9] which presents lessons learned for combinatorial test design and intelligent ways to connect with the business for quality coverage. We are looking forward to an interesting IWCT and hope you enjoy both IWCT and ICST!

REFERENCES [1] Artur Andrzejak and Thomas Bach. Practical amplification of con- dition/decision test coverage by combinatorial testing. In Software Testing, Verification and Validation Workshops (ICSTW), 2018 IEEE Eleventh International Conference on - 7th International Workshop on Combinatorial Testing (IWCT), 2018. [2] Robert Binder. Optimal scheduling for combinatorial software testing and design of experiments. In Software Testing, Verification and Valida- tion Workshops (ICSTW), 2018 IEEE Eleventh International Conference on - 7th International Workshop on Combinatorial Testing (IWCT), 2018. [3] Hermann Felbinger, Franz Wotawa, and Mihai Nica. Adapting unit tests by generating combinatorial test data. In Software Testing, Verification and Validation Workshops (ICSTW), 2018 IEEE Eleventh International Conference on - 7th International Workshop on Combinatorial Testing (IWCT), 2018. [4] Konrad Fögen and Horst Lichter. Combinatorial testing with constraints for negative test cases. In Software Testing, Verification and Validation Workshops (ICSTW), 2018 IEEE Eleventh International Conference on - 7th International Workshop on Combinatorial Testing (IWCT), 2018. [5] Angelo Gargantini and Marco Radavelli. Bringing combinatorial in- teraction testing modeling and generation to the web. In Software Testing, Verification and Validation Workshops (ICSTW), 2018 IEEE Eleventh International Conference on - 7th International Workshop on Combinatorial Testing (IWCT), 2018. [6] Hao Jin, Takashi Kitamura, Eun-Hye Choi, and Tatsuhiro Tsuchiya. A satisfiability-based approach to generation of constrained locating arrays. In Software Testing, Verification and Validation Workshops (ICSTW), 2018 IEEE Eleventh International Conference on - 7th International Workshop on Combinatorial Testing (IWCT), 2018. [7] Manuel Leithner, Kristoffer Kleine, and Dimitris E. Simos. CAmetrics: A tool for advanced combinatorial analysis and measurement of test sets. In Software Testing, Verification and Validation Workshops (ICSTW), 2018 IEEE Eleventh International Conference on - 7th International Workshop on Combinatorial Testing (IWCT), 2018. [8] Hanefi Mercan, Kamer Kaya, and Cemal Yilmaz. Enumerator: An efficient approach for enumerating all valid t-tuples. In Software Testing, Verification and Validation Workshops (ICSTW), 2018 IEEE Eleventh International Conference on - 7th International Workshop on Combinatorial Testing (IWCT), 2018. [9] Saritha Route and Sudheer Pendela. Combinatorial test design - smarter way to connect with the business for quality coverage. In Software Testing, Verification and Validation Workshops (ICSTW), 2018 IEEE Eleventh International Conference on - 7th International Workshop on Combinatorial Testing (IWCT), 2018. [10] Franz Wotawa. Combining combinatorial testing and metamorphic testing for testing a logic-based non-monotonic reasoning system. In Software Testing, Verification and Validation Workshops (ICSTW), 2018 IEEE Eleventh International Conference on - 7th International Workshop on Combinatorial Testing (IWCT), 2018. [11] Chek Pin Yang, Gunwant Dhadyalla, James Marco, and Paul Jennings. The effect of time-between-events for sequence interaction testing of a real-time system. In Software Testing, Verification and Validation Workshops (ICSTW), 2018 IEEE Eleventh International Conference on - 7th International Workshop on Combinatorial Testing (IWCT), 2018.

32 KeynoteWorkshops Talk

IWCTIWCT 2018 Committees 2018 Committees

OrganizingOrganizing Committee Committee Dimitris SimosDimitris (Program Simos (Program Co-Chair), Co-Chair), SBA Research, SBA Research, Austria Austria Rachel Tzoref-BrillRachel Tzoref-Brill (Program (Program Co-Chair), Co IBM-Chair), Research, IBM Research, Israel Israel Jacek Czerwonka,Jacek Czerwonka, Microsoft Microsoft Research, Research, USA USA Angelo Gargantini,Angelo Gargantini, University University of Bergamo, of Bergamo, Italy Italy George Sherwood,George Sherwood, Testcover.com, Testcover.com, USA USA

Steering SteeringCommittee Committee Raghu Kacker,Raghu NIST, Kacker, USA NIST, USA Richard Kuhn,Richard NIST, Kuhn, USA NIST, USA Yu Lei, UniversityYu Lei, University of Texas ofArlington, Texas Arlington, USA USA Itai Segall,Itai Alcatel-Lucent Segall, Alcatel-Lucent Bell Labs, Bell Israel Labs, Israel

ProgramProgram Committee Committee Myra Cohen,Myra University Cohen, University of Nebraska-Lincoln, of Nebraska-Lincoln, USA USA Charles Colbourn,Charles Colbourn, Arizona StateArizona University, State University, USA USA Takashi Kitamura,Takashi Kitamura, Information Information Technolo Technology Researchgy ResearchInstitute, Institute,AIST, Japan AIST, Japan Peter Kruse,Peter Berner Kruse, & Berner Mattner, & GermanyMattner, Germany Jim Lawrence,Jim Lawrence, George Mason George University, Mason University, USA USA ChanghaiChanghai Nie, Nanjing Nie, University,Nanjing University, China China Murat Ozcan,Murat Siemens Ozcan, IndustrySiemens Inc.,Industry USA Inc., USA MohammadMohammad Raunak, LoyolaRaunak, University Loyola University Maryland, Maryland, USA USA Krishnan KrishnanRangarajan, Rangarajan, Dayananda Dayananda Sagar Co llegeSagar of Co Engineering,llege of Engineering, Bangalore, Bangalore, India India Keizo Tatsumi,Keizo Tatsumi,Fujitsu LTD Fujitsu & ASTER, LTD & JapanASTER, Japan TatsuhiroTatsuhiro Tsuchiya, Tsuchiya, Osaka University, Osaka University, Japan Japan Ziyuan Wang,Ziyuan Nanjing Wang, UniversityNanjing University of Posts and of Posts Telecommunications, and Telecommunications, China China Huayao Wu,Huayao Nanjing Wu, University,Nanjing University, China China Akihisa Yamada,Akihisa Yamada,University University of Innsbruck, of Innsbruck, Austria Austria Jian Zhang,Jian Institute Zhang, ofInstitute Software, of Software, Chinese AcademyChinese Academy of Sciences, of Sciences, China China Peter Zimmerer,Peter Zimmerer, Siemens AG,Siemens Germany AG, Germany

24 24 33

Workshops

Welcome Message from the Chairs TAIC PART 2018

It is our pleasure to welcome you to the 13th edition of the International Workshop on Testing: Academia-Industry Collaboration, Practice and Research Techniques (TAIC PART 2018), collocated with the 11th IEEE International Conference on Software Testing, Verification and Validation (ICST 2018) in Västerås, Sweden

The workshop aims to forge collaborations between industry and academia on the challenging and exciting problem of real-world software testing. It is promoted by representatives of both industry and academia, bringing together industrial software engineers and testers with researchers working on theory and practice of software testing. The goals of TAIC PART range from the articulation of open research questions in the field of software testing and analysis to the development and evaluation of solutions for real-world challenges faced by industry. The common theme is the discussion and advancement of approaches, methods and tools for engaging collaborations between academia and industry in software testing.

TAIC PART 2018 received 11 submissions, and each paper was reviewed by at least three members of the program committee. After the reviewing and discussion process, the program committee accepted 8 papers (7 full papers and 1 fast abstract).

In addition, TAIC PART 2018 featured two excellent industrial keynotes by Emily Bache, from PRAQMA, and Jonas Hermansson, from Inceptive Stockholm AB, who both talked about their experiences with testing in industrial practice and areas of work they think are important moving forward. Areas such as using microservices to transform monolithic system architectures, testing approaches and tools for such transitions and organisational aspects that influence future software development and testing.

Finally, we would like to thank the authors, keynote speakers and participants for their valuable contributions, the members of the program committee for assuring the quality and success of this workshop, and Software Competence Center Hagenberg (SCCH), Austria, for sponsoring TAIC PART 2018.

Enjoy the workshop!

Emil Alégroth

Takashi Kitamura

Guowei Yang

25 34

Workshops

TAIC PART 2018 Committees

Workshop Organizers Emil Alégroth, Blekinge Institute of Technology (BTH), Sweden Takashi Kitamura, Information Technology Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Japan Guowei Yang, Texas State University, USA

Program Committee Tao Yue, Simula, Norway Thomas Bach, Heidelberg University, Germany Hirohisa Aman, Ehime University, Japan Rudolf Ramler, SCCH, Austria Xiao Qu, ABB Corporate Research, USA Xusheng Xiao, Case Western Reserve University, USA Hongyu Zhang, The University of Newcastle, Australia Shaukat Ali, Simula, Norway Cyrille Artho, KTH, Sweden Shin Nakajima, National Institute of Informatics, Japan Tatsuhiro Tsuchiya, Osaka University, Japan Michael Unterkalmsteiner, Blekinge institute of Tech., Sweden Haruto Tanno, NTT, Japan Pekka, Aho, VTT, Finland Sigrid Eldh, Ericsson, Sweden Emilie Engström, Lund University, Sweden Vahid Garousi, Wageningen University, Netherlands Mark Harman, University College London, United Kingdom Mika Mäntylä, University of Oulu, Finland Stefan Mohacsi, Atos, Austria Reinhold Plösch, Hohannes Kepler University Linz, Austria Travassos Guilherme Horta, Universidade Federal do Rio, Brazil Peter Zimmerer, Siemens Corporate Tech. Germany Manuel Oriol, ABB Corporate Research, Switzerland Michael Felderer, University of Innsbruck, Austria Hasan Sözer, Özegin University, Turkey Darko Marinov, University of Illinois, USA Elizabeta Fourneret, University of Franche-Comte, France

Steering Committee Michael Felderer, University of Innsbruck, Austria Mark Harman, University College London, United Kingdom Ina Schieferdecker, Technische Universität Berlin, Germany Emelie Engström, Lund University, Sweden

26 35

Program Details

Workshops

Sunday 8th April Program Details 19:30 Informal activity. TBA

MondaySunday 8th9th AprilApril ICST Workshops 19:30 Informal activity. TBA 12:00 - 12:30 ITEQS 2nd International Workshop on Testing Learning-Based Self-Adaptive Assurance of Timing Properties in a Real-Time Extra-Functional Properties and QualityMonday 9th April Characteristics of Software Systems Embedded System ICST Workshops Mahshid Helali Moghadam, Mehrdad 12:00Saadatmand, - 12:30 Markus Borg, Markus Bohlin, ITEQS 9:00 - 9:10: Opening Björn Lisper 2nd International Workshop on Testing Learning-Based Self-Adaptive Assurance of Extra-Functional Properties and Quality Timing Properties in a Real-Time 12:30 - 14:00: Lunch Characteristics9:10 - 10:00 of Software Systems Embedded System Some Experiences from Testing and 14:00Mahshid - 15:00 Helali Moghadam, Mehrdad Verifying Complex Concurrent and TowardsSaadatmand, Compositional Markus Borg, Testing Markus of Bohlin, Real- 9:00Distributed - 9:10: OpeningSystems (Keynote 1) TimeBjörn SystemsLisper (Keynote 2) Prof. Konstantinos Sagonas, Uppsala Prof. Brian Nielsen, Aalborg University 9:10University - 10:00 12:30 - 14:00: Lunch Some Experiences from Testing and 15:00 - 15:30 14:00 - 15:00 Verifying Complex Concurrent and Scan Code Injection Flaws in HTML5- 10:00 - 10:30 Towards Compositional Testing of Real- Distributed Systems (Keynote 1) based Mobile Applications A Testability Analysis Framework for Non- Time Systems (Keynote 2) Prof. Konstantinos Sagonas, Uppsala Tuong Lau Functional Properties Prof. Brian Nielsen, Aalborg University University Michael Felderer, Bogdan Marculescu, 15:30 - 16:00: Coffee Break 15:00 - 15:30 Francisco Gomes, Robert Feldt, Richard Scan Code Injection Flaws in HTML5- Torkar 16:00 - 16:50 10:00 - 10:30 based Mobile Applications A Testability Analysis Framework for Non- Panel Discussion: Challenges of Testing 10:30 - 11:00: Coffee Break Tuong Lau Functional Properties Autonomous Systems Vahid Garousi, Cristina Seceleanu, 11:00Michael - 11:30 Felderer, Bogdan Marculescu, 15:30 - 16:00: Coffee Break Model-BasedFrancisco Gomes, Mutation Robert Testing Feldt, Richard of Real- Konstantinos Sagonas, Brian Nielsen, TimeTorkar Systems via Model Checking 16:00Bernhard - 16:50 Aichernig Panel Discussion: Challenges of Testing Florian Lorber, Kim Guldstrand Larsen, Brian 16:50 10:30 - 11:00: Coffee Break Autonomous Systems Nielsen Best Paper Award and Closing 11:00 - 11:30 Vahid Garousi, Cristina Seceleanu, 11:30 - 12:00 Model-Based Mutation Testing of Real- Konstantinos Sagonas, Brian Nielsen, Identifying useful mutants to test time Time Systems via Model Checking Bernhard Aichernig properties Florian Lorber, Kim Guldstrand Larsen, Brian Birgitta Lindström, Jeff Offutt, Loreto 16:50 Nielsen Gonzalez-Hernandez, Sten F. Andler Best Paper Award and Closing 11:30 - 12:00 Identifying useful mutants to test time properties 27 Birgitta Lindström, Jeff Offutt, Loreto Gonzalez-Hernandez, Sten F. Andler

27

36 Workshops

VVIoT 2018 The EUREKA ITEA3 TESTOMAT Project First International Workshop on – The Next Level of Test Automation Verification and Validation of Internet of Sigrid Eldh, Ericsson AB, Sweden Things Challenges in Cybersecure Internet-of- Things: a Basque Industry 4.0 Perspective 8:30 - 9:00: Registration Aitor Urbieta, IK4-IKERLAN, Spain 9:00 - 9:05: Welcome and Introduction 17:15 - 17:30: Closing Session VVIoT Chairs Announcement of the best paper

9:05 - 10:30 Closing Remarks Keynote Presentation 1: Automated Testing VVIoT Chairs of Autonomous Driving Assistance Systems Lionel BriandUniversity of Luxembourg, Luxembourg

10:30 - 11:00: Coffee Break

11:00 - 12:30: Technical Track USE 2018 A Framework for Threat-driven Cyber The Second International Workshop on Security Verification of IoT Systems Usages of Symbolic Execution (USE 2018) Tomas Kulik, Peter Würtz Vinther Tran-

Jørgensen, Jalil Boudjadar and Carl Peter Leslie Schultz Session 1 9:00-10:30: Invited Talks Architectures and Experiences in Testing Generating Heap Test Inputs with Symbolic IoT Communications Execution Teemu Kanstrén, Jukka Mäkelä and Pekka Mauro Pezzè Karhula Symbolic Execution for the Security Realm: A Brief Overview of Existing Tools for the Cyber Grand Challenge Story Testing the Internet-of-Things Emilio Coppa João Pedro Dias, Flávio Couto, Ana C.R. Session 2 11:00-12:30: Regular Talks Paiva and Hugo Sereno Ferreira Automatic Testing of Symbolic Execution Requirements for Testing and Validating Engines the Industrial Internet of Things Timotej Kapus and Cristian Cadar Liliana Antão, Rui Pinto, João Reis and Gil Symbolic Execution and Code-Coverage Gonçalves Oriented Testing 12:30 - 14:00: Lunch Michaël Marcozzi, Sébastien Bardin, Nikolai Kosmatov, Mickaël Delahaye, Virgile 14:00 - 15:30 Prevosto Keynote Presentation 2: Trusted Apps for DIVERSITY : An Extensible Symbolic the Internet of Things. Execution Tool Christian Prehoferfortiss GmbH, Germany Boutheina Bannour and Arnault Lapitre 15:30 - 16:00: Coffee Break

16:00 - 17:15 European/National Projects

28 37 Workshops

Session 3. 14:00-15:30: Invited Talk 11:00-12:30 Test Maintenance Using Test Ranges to Enhance Symbolic Session Chair: Markus Borg Execution Uncover Your Tracks: Using CodeScene to Sarfraz Khurshid Supervise the Quality of Automated Tests Test generation from FSM fault model by Adam Tornhill Constraint Solving Evaluating Test Data Generation for Omer Nguena-Timo Untyped Data Structures Using Genetic Algorithms Session 4. 16:00-17:30: Regular Talks Ralf Gerlich and Christian Prause Symbolic execution for binary-level security Sébastien Bardin and Richard Bonichon Feature based testing by using model synthesis, test generation and Synthesizing Adapters For Binary Code parameterizable test prioritization Using Symbolic Execution Martin Reider, Stephan Magnus and Jan Vaibhav Sharma and Stephen McCamant Krause

12:30 - 14:00: Lunch

14.00-15.30 Automated GUI Testing Session Chair: Serge Demeyer NEXTA 2018 1st IEEE Workshop on NEXt level of Test Introducing automated GUI testing and Automation 2018 observing its benefits: an industrial case study in the context of law-practice Registration open from 8.00 at AROS management software CONGRESS CENTER; Västerås Vahid Garousi and Erdem Yildirim Automatically Generating Test Scripts for 9.00-9.20: Introductions and Welcome from GUI Testing the Chairs Toshiyuki Kurabayashi, Muneyoshi Iyama, Session Chair: Sigrid Eldh Hiroyuki Kirinuki and Haruto Tanno Hello - introductions! Maintenance of Android Widget-based GUI Why this workshop Testing: A Taxonomy of test case modification reasons Best Twitter Award Riccardo Coppola, Marco Torchiano and Best Question Award Maurizio Morisio Best Paper Award 15:30 - 16:00: Coffee 9.20-10.10 16.00-17.30 Test Automation - Automation of What? Test Impact, Metrics and Embedded Keynote by Prof. Yvan Labiche, Carleton Systems University, Canada Session Chair: Adnan Causevic 10.10-11.30 The Next Level of Test Automation: What Challenges in Automated Testing through About the Users? Graphical User Interface Kristian Wiklund and Monika Wiklund Pekka Aho and Tanja Vos 10:30 - 11:00: Coffee

29 38 Workshops

A Preliminary Study on TESTOMAT 11:30 Project’s Impact: Software Test To Detect Abnormal Program Behaviours Automation Acceptance and Use at a via Mutation Deduction Company in Turkey Jie Zhang, Dan Hao, Lingming Zhang and Lu Özlem Albayrak Zhang

Design of High Confidence Embedded 11:50 Software Hardware-in-loop Simulation Test If You Can't Kill a Supermutant, You Have Platform Based on Hierarchical Model a Problem Gao Feng and Deng Fei Rahul Gopinath, Björn Mathis and Andreas Software metrics for policy-driven SDLC Zeller automation 12:10 Leonid Borodaev, Rix Groenboom, Rein Mull it over: mutation testing based on Smedinga and Alex Telea LLVM 17:30-17:35 Closing and Awards Alex Denisov and Stanislav Pankevich

19:00-22:00 Workshop dinner (self-paying 12:30: Lunch Sponsored by the City of Västerås) 14:00: Keynote 2 Mutation Testing: Evaluating Millions of

Mutants in 8 Programming Languages Goran Petrovic

15:30: Fika 16:00: Session 2 MUTATION 2018 16:00 The 13th International Workshop on Mutant Quality Indicators Mutation Analysis Mike Papadakis, Thierry Titcheu Chekam and Yves Le Traon Västerås, Sweden, April 9, 2018 - Co-located with ICST 16:20 MUSIC: MUtation analySIs tool with High 08:00: Registration Configurability and Extensibility 09:00: Opening Duy Loc Phan, Yunho Kim and Moonzoo Kim 09:15: Keynote 1 16:40 MUTATION: What Have We Done and An Industrial Application of Mutation What Can We Do Next? Testing: Lessons, Challenges, and Research Prof. John Clark Directions 10:30: Fika Goran Petrović, Marko Ivanković, Robert Kurtz, Paul Ammann and René Just 11:00: Session 1 17:00: Closing 11:00 A Systematic Review of Cost Reduction 17:00 Techniques for Mutation Testing: 18:30 Burgers & activities at Strike, Preliminary Results Västerås Fabiano Cutigi Ferrari, Alessandro Viola

Pizzoleto and Jeff Offutt

30 39 Tuesday 10th April

TuesdayICST Conference 10th April 09:00: Intro & Welcome ICST ConferenceJournal First 1 Main Conference 09:00:09:30: Intro Keynote & Welcome 1 Journal11:00: A First model 1 for estimating change Infer: Static Analysis at Scale propagation in software, SQJ 2017 09:30:Dulma Keynote Churchill 1 11:00:Kecia AA. modelM. Ferreira, for estimating Mariza A. change S. Bigonha, Tuesday 10th April Infer: Static Analysis at Scale propagationRoberto S. Bigonha, in software, Bernardo SQJ 2017N. de Lima, 10:30: Coffee Break Dulma Churchill ICST ConferenceKeciaBárbara A. M. M. Ferreira, Gomes and Mariza Luiz A.Felipe S. Bigonha, O. Mendes Roberto S. Bigonha, Bernardo N. de Lima, 10:30:Research Coffee 1 Testing Break & Debugging 1 09:00: Intro & Welcome JournalBárbara11:00-12:00:Tools/Demos FirstM. Gomes 1Tools/Demos and Lightning Luiz FelipeLightning Talks O. Mendes talks 11:00: Practical Test Dependency Detection Research 1 Testing & Debugging 1 12:30: Lunch 09:30:Alessio Keynote Gambi, 1Jonathan Bell and Andreas 11:00:11:00-12:00: A model Tools/Demos for estimating Lightning change talks 11:00:Infer:Zeller Static Practical Analysis Test Dependencyat Scale Detection propagation in software, SQJ 2017 12:30: Lunch AlessioDulma ChurchillGambi, Jonathan Bell and Andreas Kecia A. M. Ferreira, Mariza A. S. Bigonha, 11:30: DOMINO: Fast and Effective Test Research 2 Regression and Continuous Zeller Roberto S. Bigonha, Bernardo N. de Lima, 10:20:10:30:Data GenerationCoffee Break for Relational Database Integration ResearchBárbara M. 2 RegressionGomes and Luizand FelipeContinuous O. Mendes 11:30:Schemas DOMINO: Fast and Effective Test 14:00: Characterizing Defective Research 1 Testing & Debugging 1 Integration DataAbdullah Generation Alsharif, for Gregory Relational Kapfhammer Database and 11:00-12:00:Configuration Tools/Demos Scripts Used Lightning for Continuous talks Schemas 10:50:11:00:Phil McMinn Practical Test Dependency Detection 14:00:Deployment Characterizing Defective Abdullah Alsharif, Gregory Kapfhammer and 12:30: Lunch Alessio Gambi, Jonathan Bell and Andreas ConfigurationAkond Rahman Scripts and Laurie Used Williams for Continuous Phil12:00: McMinn Automated Generation of Zeller Deployment Constraints from Use Case Specifications to 14:30: REMAP: Using Rule Mining and ResearchAkond Rahman 2 Regression and Laurie and Williams Continuous 11:30:12:00:Support DOMINO:Automated System Testing FastGeneration and Effective of Test Multi-Objective Search for Dynamic Test Integration DataConstraintsChunhui Generation Wang, from Fabrizio forUse Relational Case Pastore Specifications Database and Lionel to 14:30:Case PrioritizationREMAP: Using Rule Mining and SchemasSupportBriand System Testing 14:00:Multi-ObjectiveDipesh Characterizing Pradhan, Search Shuai Defective Wang,for Dynamic Shaukat Test Ali, AbdullahChunhui Wang,Alsharif, Fabrizio Gregory Pastore Kapfhammer and Lionel and ConfigurationCaseTao YuePrioritization and Marius Scripts Liaaen Used for Continuous PhilBriand McMinn Dipesh Pradhan, Shuai Wang, Shaukat Ali, Industry 1 Industry Track #1 Deployment 15:00: Speeding up Mutation Testing via AkondTao Yue Rahman and Marius and Laurie Liaaen Williams 12:00: Automated Generation of Regression Test Selection: An Extensive Industry11:00: On 1 IndustryEvaluating Track Effectiveness #1 and Constraints from Use Case Specifications to 15:00:Study Speeding up Mutation Testing via Efficiency of Several Automated Software 14:30: REMAP: Using Rule Mining and Support System Testing Multi-ObjectiveRegressionLingchao Chen Test andSeleSearch Lingmingction: for An Dynamic Zhang Extensive Test 11:00:Verification On Evaluating Tools Effectiveness and Chunhui Wang, Fabrizio Pastore and Lionel CaseStudy Prioritization EfficiencyChristian Prause,of Several Rainer Automated Gerlich Softwareand Ralf Briand DipeshLingchao Pradhan, Chen and Shuai Lingming Wang, ZhangShaukat Ali, VerificationGerlich Tools Research 3 Testing & Debugging 2 Christian Prause, Rainer Gerlich and Ralf Tao Yue and Marius Liaaen 14:00: Automating Targeted Property- IndustryGerlich11:30: Automating 1 Industry TrackWeb Application #1 Research 3 Testing & Debugging 2 Testing from the Ground Up: Experiences 15:00:Based Speeding Testing up Mutation Testing via Regression Test Selection: An Extensive 11:00:11:30:and Lessons AutomatingOn Evaluating Learned Web Effectiveness in Application an Industrial and 14:00:Andreas Automating Löscher and Targeted Konstantinos Property- Sagonas Study EfficiencyTestingSetting from of Severalthe Ground Automated Up: Experiences Software Based Testing Lingchao14:30: Testing Chen and Cloud Lingming Applications Zhang under VerificationandVidroha Lessons Debroy, ToolsLearned Lance in Brimble an Industrial and Matt Yost Andreas Löscher and Konstantinos Sagonas Cloud-Uncertainty Performance Effect ChristianSetting Prause, Rainer Gerlich and Ralf 12:00: Improving Test Execution Efficiency 14:30:Wei Wang, Testing Ningjing Cloud Tian, Applications Sunzhou Huang,under GerlichVidroha Debroy, Lance Brimble and Matt Yost Research 3 Testing & Debugging 2 through Clustering and Reordering of Cloud-UncertaintySen He, Abhijeet Srivastava, Performance Mary Effect Lou Soffa 11:30:12:00:Independent AutomatingImproving Test Test StepsWeb Execution Application Efficiency 14:00:Weiand Wang, Lori Automating Pollock Ningjing TargetedTian, Sunzhou Property- Huang, TestingthroughBenedikt from Clustering Walter, the GroundMaximilian and Reordering Up: Schilling, Experiences of Marco BasedSen He, Testing Abhijeet Srivastava, Mary Lou Soffa

andIndependentPiechotta Lessons and LearnedTest Stephan Steps in Rudolph an Industrial Andreasand Lori LöscherPollock and Konstantinos Sagonas SettingBenedikt Walter, Maximilian Schilling, Marco 14:30: Testing Cloud Applications under VidrohaPiechotta Debroy, and Stephan Lance Rudolph Brimble and Matt Yost Cloud-Uncertainty Performance Effect

12:00: Improving Test Execution Efficiency Wei Wang, Ningjing Tian, Sunzhou Huang, through Clustering and Reordering of 31 Sen He, Abhijeet Srivastava, Mary Lou Soffa Independent Test Steps and Lori Pollock Benedikt Walter, Maximilian Schilling, Marco 31

Piechotta and Stephan Rudolph

31 40 KeynoteMain Conference Talk

15:00: Behavioral Fault Localization by Thomas Durieux, Youssef Hamadi, Zhongxing Sampling Suspicious Dynamic Control Flow Yu, Benoit Baudry and Martin Monperrus Subgraphs Tim Henderson and Andy Podgurski 16:30: Finding Substitutable Binary Code By Synthesizing Adapters 15:30: Coffee Break Vaibhav Sharma, Kesha Hietala and Stephen McCamant

Tool #1 17:00: Test Execution Driven Synthesis of API Sequences With Conditionals and 14:00: EvoMaster: Evolutionary Multi- Loops context Automated System Test Generation Zijiang Yang, Jinru Hua, Kaiyuan Wang and Andrea Arcuri Sarfraz Khurshid 14:30: AUnit: A Test Automation Tool for Alloy Journal First 2 Allison Sullivan, Kaiyuan Wang, Sarfraz Khurshid 16:00: Integrating software quality models into risk-based testing, SQJ 2017 15:00: Jaguar: a spectrum-based fault Harald Foidl and Michael Felderer localization tool for real-world software Henrique Lemos Ribeiro, Higor Amario de 16:30: Detecting potential deadlocks Souza, Roberto Paulo Andrioli De Araujo, through change impact analysis, SQJ 2017 Marcos Lordello Chaim, Fabio Kon C. Metcalf and T. Yavuz 17:00: Assessment of C++ object-oriented mutation operators: A selective mutation Research 4 Mobile and Web Application approach Testing P. Delgado-Pérez, S. Segura, I. Medina-Bulo 16:00: QBE: QLearning-Based Exploration 18:30 Reception / Welcome with drinks at of Android Applications the Plaza lobby bar Yavuz Koroglu, Alper Sen, Ozlem Muslu, Yunus Mete, Ceyda Ulker, Tolga Tanriverdi and Yunus Donmez

16:30: Automated Accessibility Testing of Mobile Apps Marcelo Medeiros Eler, José Miguel Rojas,

Yan Ge and Gordon Fraser 17:00: Context-Based Event Trace Reduction in Client-Side JavaScript

Applications Jie Wang, Wensheng Dou, Chushu Gao, Yu Gao and Jun Wei

Research 5 Program Repair and Synthesis

16:00: Exhaustive Exploration of the Failure-oblivious Computing Search Space

32 41 Main Conference

Wednesday 11th April

09:00 MIP #1 Doctoral Symposium #1

09:25 Keynote 2: But is it fun? Keynote: Hints for giving a presentation Magnus Nordin/David King/Stefan Posthuma Tanja Vos 10:20 Coffee Break 10:50 Taxonomy Based Testing using SW91, a Medical Device Software Defect Research 6 Visual Testing Taxonomy Hamsini Ketheswarasarma Rajaram 10:50 Continuous Integration and Visual GUI Testing: Benefits and Drawbacks in 11:20 Test automation maturity Industrial Practice assessment Emil Alégroth, Arvid Karlsson and Alexander Yuqing Wang Radway 11:50 Software Systems, Their Engineers 11:20 Automatic Detection of Visibility and Their Testers Faults by Layout Changes in HTML5 Web Gergő Balogh Pages 12:20 Lunch Yeonhee Ryou and Sukyoung Ryu 11:50 Web Canvas Testing through Visual Inference Research 8 Testing & Debugging 3 Mohammad Bajammal and Ali Mesbah 14:00 Invasive Software Testing: Mutating Target Programs to Diversify Test 10:50: Research 7 Search Based Exploration for High Test Coverage Approaches Yunho Kim, Shin Hong, Bongseok Ko, Duy Loc Phan and Moonzoo Kim 10:50 Uncovering Unknown System Behaviors in Uncertain Networks with 14:30 How Do Automatically Generated Model and Search-based Testing Unit Tests Influence Software Ruihua Ji, Zhong Li, Shouyu Chen, Minxue Maintenance? Pan, Tian Zhang, Tao Yue, Shaukat Ali and Sina Shamshiri, José Miguel Rojas, Juan Xuandong Li Pablo Galeotti, Neil Walkinshaw and Gordon Fraser 11:20 Automated Repair of Internationalization Failures Using Style 15:00 Localizing Faults in Cloud Systems Similarity Clustering and Search-Based Techniques Leonardo Mariani, Cristina Monni, Mauro Sonal Mahajan, Abdulmajeed Alameer, Phil Pezzè, Oliviero Riganelli and Rui Xin McMinn and William G.J. Halfond

11:50 Accelerating Search-based Program Repair Tool #2 Benjamin Mehne, Hiroaki Yoshida, Mukul Prasad, Koushik Sen, Divya Gopinath and 14:00 SimEvo: A Toolset for Simulink Test Sarfraz Khurshid Evolution & Maintenance Eric J.Rapos, James R.Cordy

33 42 Main Conference

14:30 Fuzzinator: An Open-Source Running Experiments and Performing Data Modular Random Testing Framework Analysis Using SchemaAnalyst and Renáta Hodován, Akos Kiss DOMINO Abdullah Alsharif, Gregory Kapfhammer, Phil McMinn Doctoral Symposium #2 Jaguar: a spectrum-based fault localization Keynote: How to get your paper rejected tool Jeff Offutt Henrique Lemos Ribeiro, Higor Amario de Souza, Roberto Paulo Andrioli De Araujo, 14:00 Testing Natural Language Grammars Marcos Lordello Chaim, Fabio Kon Inari Listenmaa Model-based security testing suite 14:30 Analysis of the logical consistency in supporting IoT Certification Benchmarking Cassandra and Labelling Pablo Suárez-Otero Elizabeta Fourneret, Abbas Ahmad, Bruno 15:00 Social activity for the participants Legeard, Fabrice Bouquet, Sara N. Matheu, Antonio Skarmeta 15:30 Coffee Break Generating Database Schema Test Suites with DOMINO 16:00-16:30 Demo Abdullah Alsharif, Gregory Kapfhammer, Phil McMinn TECOR: Automate the Testing of One

Product on Many Configurations Anatoly Vasilevskiy, Hui Song, Brice Morin Industry 2 Industry Track #2 OCLgen: Automated Generation of 16:00: Systematic Test Platform Selection Constraints from Use Case Specifications to Christian Schwarzl and Jens Herrmann Support System Testing 16:30 Bluetooth Low Energy Devices Chunhui Wang, Fabrizio Pastore, Lionel Security Testing Framework Briand Apala Ray, Vipin Raj, Manuel Oriol, Aurelien Regression Test Visualization Using the Monot and Sebastian Obermeier Unity Game Engine – An Example of Visual

Analytics in ASIC Design Verification Doctoral Symposium #3 Markus Borg, Daniel Hansson, Andreas Brytting 16:00: Knowledge Discovery Metamodel- Interactive Repair of Test Scripts Using based Unit Test Cases Generation Various Clues in Web Application João Paulo Pires Hiroyuki Kirinuki, Haruto Tanno, Katsuyuki 16:30 Anti-Patterns in Infrastructure as Natsukawa Code Fawlty: A test framework for automated Akond Rahman system-level testing Wrap up Per Erik Strandberg Test Script Generation Tool for Regression Testing using Static and Dynamic Analysis Muneyoshi Iyama, Hiroyuki Kirinuki, Toshiyuki Kurabayashi, Haruto Tanno

34 43 Main Conference 17:00 Panel Discussion:

When17:00 are Panel Software Discussion: Testing Research Contributions, Real Contributions? When are Software Testing Research 19:00Contributions, Banquet Real at Munktell Contributions? museum, Eskilstuna 19:00Prof. Lionel Banquet Briand at Munktell(Univ Luxembourg) museum, 22:30EskilstunaDr. Markus Departure Borg (RISE from SICS)Eskilstuna at 22.30

22:30Dr. Kristian Departure Wiklund from (Ericsson) Eskilstuna at 22.30

Session chairs: Sigrid Eldh, Elaine Weyuker

Thursday 12th April

09:00 MIP #2 Thursday 12th11:30 April An Empirical Study on Multi-Label Techniques for Software Failure 09:3009:00 MIPKeynote #2 3: Safety Verification for Classification11:30 An Empirical Study on Multi-Label Deep Neural Networks YangTechniques Feng, James for Software Jones, Zhenyu Failure Chen and 09:30 Keynote 3: Safety Verification for Marta Kwiatkowska ChunrongClassification Fang Deep Neural Networks Yang Feng, James Jones, Zhenyu Chen and Marta10:30 Kwiatkowska Coffee Break 12:00Chunrong Crashing Fang simulated planes is cheap: Can simulation detect robotics bugs early? 10:30Research Coffee 9 Mutation Break Analysis Christopher12:00 Crashing Steven simulated Timperley, planes Afsoon is Afzal,cheap: Can simulation detect robotics bugs early? 11:00Research An Investigation9 Mutation Analysis of Compression Deborah Katz, Jam Marcos Hernandez and Christopher Steven Timperley, Afsoon Afzal, Techniques to Speed up Mutation Testing Claire Le Goues Deborah Katz, Jam Marcos Hernandez and Qianqian11:00 An Zhu,Investigation Annibale ofPanichella Compression and Andy 12:30Claire Le Lunch Goues ZaidmanTechniques to Speed up Mutation Testing Qianqian Zhu, Annibale Panichella and Andy 14:0012:30 LunchOpen Steering Committee & Zaidman11:30 Approximate Transformations as Closing Mutation Operators 14:00 Open Steering Committee & 11:30 Approximate Transformations as Farah Hariri, August Shi, Owolabi Legunsen, 15:30Closing Coffee Break MilosMutation Gligoric, Operators Sarfraz Khurshid and Sasa MisailovicFarah Hariri, August Shi, Owolabi Legunsen, 19:0015:30 CoffeePC dinner Break at Pråmen, Västerås Milos Gligoric, Sarfraz Khurshid and Sasa Misailovic12:00 Virtualized-Fault Injection Testing: 19:00 PC dinner at Pråmen, Västerås a Machine Learning Approach

Hojat12:00 Khosrowjerdi, Virtualized-Fault Karl Meinke Injection and Testing: a Machine Learning Approach Andreas Rasmusson Hojat Khosrowjerdi, Karl Meinke and Andreas Rasmusson Research 10 Defect Analysis

11:00Research Investigating 10 Defect NLP-basedAnalysis Approaches for Predicting Manual Test Case Failure Fatemeh11:00 Investigating Sharifi and NLP-basedHadi Hemmati Approaches for Predicting Manual Test Case Failure Fatemeh Sharifi and Hadi Hemmati

35

35 44 Workshops

Friday 13th April ICST Workshops

InSTA 2018 15:55 A Test Architecture for Machine 5th International Workshop on Software Learning Product Test Architecture Yasuharu Nishi, et al.

16:20 A Survey of Software Quality for 9:00 Opening Machine Learning Applications Satoshi Masuda, et al. 9:15 Keynote From hacking to product - Our journey 16:45 Closing from ad hoc tool development to an

organised test system Kristian Wiklund

10:30 Break

11:00 Research paper IWCT 2018 Testing of big data analytics systems by 7th International Workshop on benchmark Combinatorial Testing Mingang Chen, et al. 09.15 - 10.30: Opening Session 11:30 Test case reduction based on the join condition in pairiwise coverage-based Opening Remarks: Welcome to IWCT database testing Dimitris E. Simos Yuper Lay Myint, et al. Keynote: Embedded functions in 12:00 Model-based security testing - combinatorial testing Deriving test models from artefacts of George Sherwood security engineering Armin Lunkeit, et al. 10.30 - 11.00: Coffee break

12:30 Lunch 11.00 - 12.20: Session 1: Generation, Scheduling and Fault Localization 14:00 Industry experience Software Test Architectures and Advanced A Satisfiability-Based Approach to Support Environments for IoT Generation of Constrained Locating Arrays Jon Hagar Hao Jin, Takashi Kitamura, Eun-Hye Choi and Tatsuhiro Tsuchiya 14:30 Extending the UML Testing Profile Best foundation paper award with a fine-grained test logging model Marc-Florian Wendland, et al. Optimal Scheduling for Combinatorial Software Testing and Design of 15:00 Break Experiments Robert V. Binder 15:30 Emerging idea Proposal for Adding a “Test Concern” Enumerator: An Efficient Approach for Concept to UTP2 Enumerating all Valid t-tuples Makoto Nakakuki, et al. Hanefi Mercan, Kamer Kaya and Cemal Yilmaz

36 45 Workshops

12.20 - 12.30: Session 2: Poster Session Adapting Unit Tests by Generating Combinatorial Test Combinatorial Test Design – smarter way Hermann Felbinger, Franz Wotawa and Mihai to connect with the business for quality Nica coverage Saritha Route and Sudheer Pendela 17.40 - 18.00: Break

12.30 - 14.00: Lunch 18.00 - 19.00: IWCT Organizing & Steering Committee Meeting 14.00 - 15.20: Session 3: Modelling and Tools

Bringing combinatorial interaction testing modeling and generation to the web Angelo Gargantini and Marco Radavelli Best application paper award TAIC PART 2018 CAmetrics: A Tool for Advanced The 13th Workshop on Testing: Academia- Combinatorial Analysis and Measurement Industry Collaboration, Practice and of Test Sets Research Techniques Manuel Leithner, Kristoffer Kleine and 09:00 - 09:15: Opening Dimitris E. Simos 09:15 - 10:00: Keynote 1 Combinatorial Testing with Constraints for Emily Bache, Praqma Negative Test Cases Konrad Fögen and Horst Lichter 10:00 - 10:30: Support for Finding Presentation Failures by Using Computer Coffee break: 15.30 - 16.00 Vision Techniques 16.00 - 17.40: Session 4: Applications and Haruto Tanno and Yuu Adachi Use Cases 10:30 - 11:00: Coffee break

The effect of time-between-events for 11:00 - 11:30: Applying Automated Test sequence interaction testing of a real-time Case Generation in Industry: A system Retrospective Chek Pin Yang, Gunwant Dhadyalla, James Rudolf Ramler, Claus Klammer and Georg Marco and Paul Jennings Buchgeher Practical Amplification of 11:30 - 12:00: On the automation of testing Condition/Decision Test Coverage by a logic-based diagnosis system Combinatorial Testing Franz Wotawa Artur Andrzejak and Thomas Bach 12:00 - 12:30: Risk-Driven Model-Based Combining Combinatorial Testing and Testing: A Unified Approach Metamorphic Testing for Testing a Logic- Abdulhadi Kirkici, Ceren Şahin Gebizli and based Non-monotonic Reasoning System Hasan Sozer Franz Wotawa 12:30 - 14:00: Lunch

37 46 Workshops

14:00 - 14:30: Tailoring ISO/IEC/IEEE 11:00-12:30 Session 1: Test cases generation 29119-3 Standard for Small and Medium- and management sized Enterprises A Model-based Test Case Management Paulo Eira, Pedro Guimarães, Mónica Melo, Approach for Integrated Sets of Domain- Miguel A. Brito, Ricardo J. Machado and Specific Models António Silva Reinhard Pröll and Bernhard Bauer 14:30 - 14:45: Verification of Cyber- Generation of C++ Unit Tests from physical Automotive Systems-of-Systems: Abstract State Machines Specifications Test Environment Assignment Silvia Bonfanti, Angelo Gargantini and Atif Hermann Kaindl, Franz Lukasch, Matthias Mashkoor Heigl, Sevan Kavaldjian, Christoph SPYH-method: an Improvement in Testing Luckeneder and Sebastian Rausch of Finite-State Machines 14:45 - 15:30: Keynote 2 Michal Soucha and Kirill Bogdanov Jonas Hermansson, Inceptive Stockholm 12:30-14:00 Lunch 15:30 - 16:00: Coffee break 14:00-15:30 Keynote 16:00 - 16:30: A Topic Model and Test Configuration testing of Docker-based History-Based Test Case Recommendation Microservices Method for Regression Testing Dr. Brice Morin Hirohisa Aman, Takashi Nakano, Hideto 15:30-16:00 Coffee break Ogasawara and Minoru Kawahara 16:00-17:30 Session 2: Application of 16:30 - 17:00: Effects of an Economic model-based testing Approach for Test Case Selection and Modelling Hand Gestures to Test Leap Reduction for a Large Industrial Project Motion Controlled Applications Thomas Bach, Ralf Pannemans and Sascha Thomas White, Gordon Fraser and Guy Schwedes Brown Around 17:00: Closing Temporal Logic Falsification of Cyber- Physical Systems: An Input-Signal-Space Optimization Approach Arend Aerts, Bryan Tong Minh, Mohammadreza Mousavi and Michel Reniers

Lightweight Model-Based Testing for A-MOST Enterprise IT 14th Workshop on Advances in Model Elodie Bernard, Fabrice Ambert, Bruno Based Testing Legeard and Arnaud Bouzy 17:30 Wrap Up and Closing 9:00-09:10 Opening

9:10-10:30 Keynote From Spec-based testing to Test Automation and Beyond Prof. Dr. Jeff Offutt

10:30-11h Coffee break

38 47 48