DOCSLIB.ORG

A Formal Approach to Conformance Testing Jan Tretmans

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A Formal Approach to Conformance Testing Jan Tretmans A Formal Approach to Conformance Testing Jan Tretmans A Formal Approach to Conformance Testing CIP–GEGEVENS KONINKLIJKE BIBLIOTHEEK, DEN HAAG Tretmans, Gerrit Jan A formal approach to conformance testing / Gerrit Jan Tretmans. – [S.l. : s.n.]. – Ill. Proefschrift Enschede. – Met lit. opg., reg. ISBN 90–9005643–2 Trefw.: communicatiesystemen ; tests. Copyright c 1992 by Jan Tretmans, Hengelo, The Netherlands. A Formal Approach to Conformance Testing PROEFSCHRIFT ter verkrijging van de graad van doctor aan de Universiteit Twente op gezag van de rector magnificus prof. dr. Th.J.A. Popma volgens besluit van het College van Dekanen in het openbaar te verdedigen op donderdag 10 december 1992 te 13.15 uur door Gerrit Jan Tretmans geboren op 27 augustus 1962 te Hengelo Ov. Dit proefschrift is goedgekeurd door de promotor prof. dr. H. Brinksma Abstract In order to assure successful communication between computer systems from different manufacturers, standardized communication protocols are being developed and spec- ified. As a next step implementations of these protocols are needed that conform to these specifications. Testing is a way to check correctness of protocol implementations with respect to their specifications. This activity is known as protocol conformance testing. This thesis deals with a formal approach to protocol conformance testing. Testing is performed based on a formal specification of the protocol. The final aim is to obtain methods for the (automatic) derivation of useful sets of tests from formal specifications. The derived tests should be provably correct, which means that they should not detect errors in correct implementations. Moreover, the derived tests should be meaningful: erroneous implementations should be detected with a high probability. An important aspect is a formal definition of what constitutes correctness, i.e. when does a protocol implementation conform to a protocol specification. Starting points for this thesis are the current, informal approach to conformance testing as it is described in the international standard ISO IS-9646 ”OSI Conformance Testing Methodology and Framework”, and the specification formalisms for distributed systems based on labelled transition systems and process algebra. The most important concepts of the standard ISO IS-9646, and of the specification formalisms used are introduced in chapter 1. Chapter 2 presents a framework for conformance testing. It is derived by giving a formal interpretation to the most important concepts from the standard ISO IS-9646, such as conformance requirement, the meaning of conformance, test purpose, test method, and different kinds of tests. This interpretation is shown to lead naturally to a definition of conformance as a (preorder) relation on the specification formalism. Such a relation is called an implementation relation. In chapter 3 the framework is elaborated with existing implementation relations for labelled transition systems. The relations are introduced using the principle of ob- servations: the behaviour of an implementation is correct if all observations made of the implementation by an environment, can be explained from the behaviour of the specification. In chapter 4 test derivation algorithms are developed for one particular implementation v vi Abstract relation, the relation conf. These algorithms can be used to derive tests from a labelled transition system specification, which are complete and correct: an implementation is correct according to conf if and only if all tests are successful. The test derivation algorithms take labelled transition systems, and consequently they can also be used to derive tests from a behaviour expression in a formal language for which the semantics is defined by a labelled transition system. A problem appears if this labelled transi- tion system is infinite in number of branches or number of states. Infinity prevents the algorithms from being implemented. Therefore the algorithms are transformed into algorithms that work on behaviour expressions. For a simple class of behaviour ex- pressions rules are given with which tests can be derived compositionally from these behaviour expressions. The implementation relations of chapter 3 and the test derivation algorithms of chap- ter 4 assume synchronous communication between the tester and the implementation, as can be modelled by the parallel synchronization operator on labelled transition sys- tems. Chapter 5 shows that this theory is not applicable when the tester and the implementation communicate via a FIFO buffer. Tests derived for synchronous com- munication cannot detect all erroneous implementations, while correct implementations are wrongly considered to be erroneous. To demonstrate this a queue-operator on la- belled transition systems is defined to formalize asynchronous communication between the tester and the implementation. This queue operator models two queues, one for input and one for output. A system that communicates asynchronously with its en- vironment is called a queue-context. It is shown that the behaviour of queue contexts is characterized by two sets of sequences of actions (traces): sequences that can be performed by a queue context, and sequences that result in a state in which no output is possible. Implementation relations for asynchronous communication are derived, and first steps towards test derivation algorithms for these relations are made. The de- rived tests are translated into TTCN, the standardized test notation from the standard ISO IS-9646. Since TTCN uses a mechanism for communication based on queues this translation makes sense, as opposed to a translation of synchronous tests. Using the test derivation algorithms of chapters 4 and 5, large numbers of tests can be derived. The reduction of the number of tests to an amount that can be handled eco- nomically and practically is called test selection. Chapter 6 studies a framework for test selection based on assigning values and costs to sets of tests. The value is related to the error detecting capability of a set of tests. The framework, which is independent from any specification formalism, is elaborated for labelled transition systems. Moreover, a test selection technique based on specification selection is presented for labelled tran- sition systems. Instead of selecting tests from a too large set of tests, the specification is transformed, such that the tests derived from the transformed, partial specification exactly constitute a selection of the tests derived from the original specification. The last chapter, chapter 7, presents conclusions and a few suggestions for further re- search: the relation between testing and verification, asynchronous communication with other kinds of contexts, extension of test selection methods to realistic specifications, the integration of values and costs, and design for testability of protocols. Finally it is Abstract vii noted that the applicability and shortcomings of all presented ideas need to be validated by applying them to testing realistic protocols. Acknowledgements I am indebted to my promotor Ed Brinksma for his ideas, inspiration, constructive criticism, assistance in the production of this thesis, and being co-author of some of the chapters. Without his help it would have been difficult to complete this thesis. I would like to thank Pim Kars for being my roommate, for reading and commenting almost all my drafts, some of them very immature, and for being there to answer my questions. Louis Verhaard and Jeroen van de Lagemaat were co-authors for some parts of this thesis, for which I thank them. Henk Eertink, Rom Langerak, Arend Rensink, Pippo Scollo, Rudie Alderden, Jan Kroon and Stan van de Burgt are thanked for commenting on earlier versions of some chapters. I would like to thank Chris Vissers for giving me the opportunity to work in the stim- ulating environment of the TIOS group, and all the members of the TIOS group for many enlightning discussions and mental support. Finally, I thank my parents for encouraging and supporting me during my whole study, and And`ele for her love, support, and understanding. Contents Abstract v Acknowledgements viii Contents ix 1 Introduction 1 1.1 Conformance Testing and the Use of Formal Methods . .... 1 1.1.1 ProtocolConformanceTesting . 2 1.1.2 FormalMethods........................... 4 1.1.3 Formal Methods in Protocol Conformance Testing . ... 7 1.2 OverviewoftheThesis ........................... 8 1.3 OverviewofISOIS-9646 .......................... 9 1.3.1 The Conformance Testing Process . 11 1.3.2 AConformingImplementation. 11 1.3.3 TestGeneration........................... 12 1.3.4 TestImplementation . 17 1.3.5 TestExecution ........................... 18 1.4 LabelledTransitionSystems . 19 1.4.1 Representation ........................... 19 1.4.2 Traces ................................ 22 1.4.3 Equality ............................... 23 2 A Formal Framework for Conformance Testing 27 2.1 Introduction................................. 27 2.2 FormalInterpretationofISOIS-9646 . ... 28 2.2.1 TheMeaningofConformance . 28 2.2.2 PICS................................. 31 2.2.3 TestPurposes............................ 32 2.2.4 TestCases.............................. 33 2.2.5 GenericTestCases ......................... 36 2.2.6 AbstractTestCases ........................ 38 2.2.7 TestGenerationinPractice . 42 2.2.8 LimitationsofTesting . 45 2.2.9 PIXIT ................................ 45 2.3 ConformanceasaRelation. .. .. 46 2.4 Examples .................................. 48 2.4.1 Natural Language Specification
Load more

Recommended publications
  • IEC 61850: Role of Conformance Testing in Successful Integration
    1 IEC 61850: Role of Conformance Testing in Successful Integration Eric A. Udren, KEMA T&D Consulting Dave Dolezilek, Schweitzer Engineering Laboratories, Inc. Abstract—The IEC 61850 Standard, Communications Net- a single standard solution for communications integration hav- works and Systems in Substations, provides an internationally ing high-level capabilities not available from protocols in prior recognized method of local and wide area data communications use. The most important technical objectives were: for substation and system-wide protective relaying, integration, control, monitoring, metering, and testing. It has built-in capabil- 1. Use self-description and object modeling technol- ity for high-speed control and data sharing over the communica- ogy to simplify the integration and configuration tions network, eliminating most dedicated control wiring as well process for the user. as dedicated communications channels among substations. IEC 2. Dramatically increase the functional capabilities, 61850 facilitates systems built from multiple vendors’ IEDs. Many vendors have supported the standard throughout its crea- sophistication and complexity of the integration to tion, and they are developing products to handle all the needed meet users’ ultimate relaying, control, and enterprise functions. data integration needs. This paper is the third in a series on the evolution of IEC 3. Incorporate robust, very high-speed control commu- 61850. It focuses on the purpose and value of conformance test- nications messaging that can operate among relays ing and certification. IEC 61850 is aimed at making it easy for and other IEDs to eliminate panel wiring and con- utilities to install and integrate single-vendor or multivendor trols. control and protection systems in substations and to integrate existing communications.
    [Show full text]
  • Construction Quality Assurance Plan Craighead County Solid
    CONSTRUCTION QUALITY ASSURANCE PLAN CRAIGHEAD COUNTY SOLID WASTE DISPOSAL AUTHORITY LEGACY CLASS 1 LANDFILL AFIN: 16-00199 ADEQ PERMIT NO. 0254-S1-R3 PREPARED FOR: Craighead County Solid Waste Disposal Authority 328 CR 476 PO Box 16777 Jonesboro, AR 72403-6712 (870) 972-6353 PREPARED BY: Terracon Consultants, Inc. 25809 Interstate 30 South Bryant, Arkansas 72022 (501) 847-9292 NOVEMBER 2008 CCSWDA Legacy Landfill Construction Quality Assurance Plan November 2008 TABLE OF CONTENTS SECTION 1 ..................................................................................................................................................................1 GENERAL ....................................................................................................................................................................1 1.0 INTRODUCTION .............................................................................................................................................1 2.0 DEFINITIONS RELATED TO CQA ................................................................................................................2 2.1 Construction Quality Assurance and Construction Quality Control ...........................................................2 2.2 Use of the Terms in This Plan ......................................................................................................................2 3.0 CQA AND CQC PARTIES .................................................................................................................................3
    [Show full text]
  • Fuzzing Radio Resource Control Messages in 5G and LTE Systems
    DEGREE PROJECT IN COMPUTER SCIENCE AND ENGINEERING, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2021 Fuzzing Radio Resource Control messages in 5G and LTE systems To test telecommunication systems with ASN.1 grammar rules based adaptive fuzzer SRINATH POTNURU KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE Fuzzing Radio Resource Control messages in 5G and LTE systems To test telecommunication systems with ASN.1 grammar rules based adaptive fuzzer SRINATH POTNURU Master’s in Computer Science and Engineering with specialization in ICT Innovation, 120 credits Date: February 15, 2021 Host Supervisor: Prajwol Kumar Nakarmi KTH Supervisor: Ezzeldin Zaki Examiner: György Dán School of Electrical Engineering and Computer Science Host company: Ericsson AB Swedish title: Fuzzing Radio Resource Control-meddelanden i 5G- och LTE-system Fuzzing Radio Resource Control messages in 5G and LTE systems / Fuzzing Radio Resource Control-meddelanden i 5G- och LTE-system © 2021 Srinath Potnuru iii Abstract 5G telecommunication systems must be ultra-reliable to meet the needs of the next evolution in communication. The systems deployed must be thoroughly tested and must conform to their standards. Software and network protocols are commonly tested with techniques like fuzzing, penetration testing, code review, conformance testing. With fuzzing, testers can send crafted inputs to monitor the System Under Test (SUT) for a response. 3GPP, the standardiza- tion body for the telecom system, produces new versions of specifications as part of continuously evolving features and enhancements. This leads to many versions of specifications for a network protocol like Radio Resource Control (RRC), and testers need to constantly update the testing tools and the testing environment.
    [Show full text]
  • Review of Model-Based Testing Approaches in Production Automation and Adjacent Domains—Current Challenges and Research Gaps
    Journal of Software Engineering and Applications, 2015, 8, 499-519 Published Online September 2015 in SciRes. http://www.scirp.org/journal/jsea http://dx.doi.org/10.4236/jsea.2015.89048 Review of Model-Based Testing Approaches in Production Automation and Adjacent Domains—Current Challenges and Research Gaps Susanne Rӧsch1, Sebastian Ulewicz1, Julien Provost2, Birgit Vogel-Heuser1 1Institute of Automation and Information Systems, Technische Universität München, München, Germany 2Assistant Professorship for Safe Embedded Systems, Technische Universität München, München, Germany Email: [email protected], [email protected], [email protected], [email protected] Received 18 August 2015; accepted 27 September 2015; published 30 September 2015 Copyright © 2015 by authors and Scientific Research Publishing Inc. This work is licensed under the Creative Commons Attribution International License (CC BY). http://creativecommons.org/licenses/by/4.0/ Abstract As production automation systems have been and are becoming more and more complex, the task of quality assurance is increasingly challenging. Model-based testing is a research field addressing this challenge and many approaches have been suggested for different applications. The goal of this paper is to review these approaches regarding their suitability for the domain of production automation in order to identify current trends and research gaps. The different approaches are classified and clustered according to their main focus which is either testing and test case genera- tion from some form of model automatons, test case generation from models used within the de- velopment process of production automation systems, test case generation from fault models or test case selection and regression testing.
    [Show full text]
  • Enter Filename
    NORTH CORRECTIVE ACTION MANAGEMENT UNIT QUALITY ASSURANCE / QUALITY CONTROL PLAN Exide Technologies Frisco Recycling Center 7471 Old Fifth Street, Frisco, Texas 75034 Submitted To: Exide Technologies 7471 Old Fifth Street Frisco, TX 75034 Submitted By: Golder Associates Inc. 14950 Heathrow Forest Parkway, Suite 280 Houston, TX 77032 NORTH CAMU QA/QC PLAN May 2019 Project No. 130208606 Golder, Golder Associates and the GA globe design are trademarks of Golder Associates Corporation May 2019 i Project No. 130208606 Table of Contents 1.0 INTRODUCTION AND PURPOSE .................................................................................................. 1 1.1 Introduction ................................................................................................................................... 1 1.2 Purpose ........................................................................................................................................ 1 2.0 GEOSYNTHETIC CLAY LINER EVALUATION ............................................................................... 2 2.1 Pre-Installation Material Evaluation ............................................................................................. 2 2.1.1 Manufacturer’s Quality Control Certificates ............................................................................. 2 2.2 Installation Procedures ................................................................................................................. 4 2.2.1 GCL Subgrade Preparation ....................................................................................................
    [Show full text]
  • Research Roadmap for Smart Fire Fighting Summary Report
    NIST Special Publication 1191 | NIST Special Publication 1191 Research Roadmap for Smart Fire Fighting Research Roadmap for Smart Fire Fighting Summary Report Summary Report SFF15 Cover.indd 1 6/2/15 2:18 PM NIST Special Publication 1191 i Research Roadmap for Smart Fire Fighting Summary Report Casey Grant Fire Protection Research Foundation Anthony Hamins Nelson Bryner Albert Jones Galen Koepke National Institute of Standards and Technology http://dx.doi.org/10.6028/NIST.SP.1191 MAY 2015 This publication is available free of charge from http://dx.doi.org/10.6028/NIST.SP.1191 U.S. Department of Commerce Penny Pritzker, Secretary National Institute of Standards and Technology Willie May, Under Secretary of Commerce for Standards and Technology and Director SFF15_CH00_FM_i_xxii.indd 1 6/1/15 8:59 AM Certain commercial entities, equipment, or materials may be identified in this document in order to describe an experimental procedure or concept adequately. Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended to imply that the entities, materials, or equipment are necessarily the best available for the purpose. The content of this report represents the contributions of the chapter authors, and does not necessarily represent the opinion of NIST or the Fire Protection Research Foundation. National Institute of Standards and Technology Special Publication 1191 Natl. Inst. Stand. Technol. Spec. Publ. 1191, 246 pages (MAY 2015) This publication is available
    [Show full text]
  • Network Protocol Testing Overview - Interworking Labs
    Network Protocol Testing Overview - InterWorking Labs While finding and fixing software defects constitutes the largest identifiable expense for the software industry(1), companies manufacturing network enabled devices have difficulty understanding how and where to go about software quality improvement. There are several activities that address improved software quality. These include formal design reviews, formal code reviews, unit test and system test. Formal design reviews require the design engineer to present his design to a group of peer engineers. The peer engineers make suggestions on areas of improvement. After implementing the design, formal code reviews require a group of peer engineers to read through the code and flag any errors or make suggestions for improvement. These are the two most effective activities for finding and removing software defects(2). What’s Inside... ‣ Load Testing (pg. 4) ‣ Stress Testing ‣ Endurance Testing ‣ Negative Testing (pg. 5) ‣ Inopportune Testing ‣ Conformance/Compliance Testing ‣ Syntax and Semantic Testing (pg. 6) ‣ Line Speed Testing ‣ Performance Testing (pg. 7) Robustness/Security Testing ‣ King Villages Center #66190 Scotts Valley, CA 95067 iwl.com ‣ Interoperability Testing (pg. 8) +1.831.460.7010 ‣ Deep Path Testing [email protected] 1 After design and code reviews, unit testing is the next most effective activity for finding and removing software defects. Each developer creates and executes unit tests on his/her code. The focus of unit testing is primarily on the quality and integrity of the code segment. Ideally, the unit test should deliver a pass or fail result for the particular code segment. System testing is the next most effective activity for finding and removing software defects.
    [Show full text]
  • An Overview of OSI Conformance Testing
    An Overview of OSI Conformance Testing Jan Tretmans Formal Methods & Tools group University of Twente January 25, 2001 1 Introduction The development of distributed systems, in which the computer functionality, such as processing functions, information storage, and human interaction, is distributed over different computer systems, raises the need for exchanging information between these systems. To have computer systems communicate successfully, the communication must occur according to well-defined rules. A protocol describes the rules with which computer systems have to comply in their communication with other computer systems. A protocol entity is that part of a computer system that takes care of the local responsibilities in communicating according to the protocol. To have successful communication among computer systems, also from different manufac- turers, many protocols are not developed in isolation, but within groups of manufacturers and users, with the aim of standardizing such protocols. This has led for instance to the development of the OSI Reference Model for Open Systems [ISO84], which serves as a framework for a set of standards that enable computer systems to communicate. How- ever, to assure successful communication it is not sufficient to specify and standardize communication protocols. Implementations of these protocol standards are required for which it must be ascertained that these implementations really behave according to these standards protocol specifications, i.e., conform to these standards. One way to do this is by testing these protocol implementations. This activity is known as protocol confor- mance testing. This note gives an introduction into some of the important concepts of protocol con- formance testing. It is largely based on the standard ISO 9646: “Conformance Testing Methodology and Framework” [ISO91].
    [Show full text]
  • Software Testing Methods and Techniques
    Software Testing Methods and Techniques Jovanović, Irena Abstract—In this paper main testing methods and such as to exercise a particular program path or techniques are shortly described. General to verify compliance with a specific requirement, classification is outlined: two testing methods – black see [11]) for which valued inputs always exist. box testing and white box testing, and their frequently In practice, the whole set of test cases is used techniques: . Black Box techniques: Equivalent considered as infinite, therefore theoretically Partitioning, Boundary Value Analysis, there are too many test cases even for the Cause-Effect Graphing Techniques, and simplest programs. In this case, testing could Comparison Testing; require months and months to execute. So, how . White Box techniques: Basis Path Testing, to select the most proper set of test cases? In Loop Testing, and Control Structure practice, various techniques are used for that, Testing. and some of them are correlated with risk Also, the classification of the IEEE Computer analysis, while others with test engineering Society is illustrated. expertise. Testing is an activity performed for evaluating 1. DEFINITION AND THE GOAL OF TESTING software quality and for improving it. Hence, the ROCESS of creating a program consists of goal of testing is systematical detection of P the following phases (see [8]): 1. defining a different classes of errors (error can be defined problem; 2. designing a program; 3. building a as a human action that produces an incorrect program; 4. analyzing performances of a result, see [12]) in a minimum amount of time program, and 5. final arranging of a product.
    [Show full text]
  • Cross-Fertilizing Formal Approaches for Protocol Conformance and Performance Testing Xiaoping Che
    Cross-fertilizing formal approaches for protocol conformance and performance testing Xiaoping Che To cite this version: Xiaoping Che. Cross-fertilizing formal approaches for protocol conformance and performance test- ing. Networking and Internet Architecture [cs.NI]. Institut National des Télécommunications, 2014. English. NNT : 2014TELE0012. tel-01127222 HAL Id: tel-01127222 https://tel.archives-ouvertes.fr/tel-01127222 Submitted on 7 Mar 2015 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. INSTITUT MINES-TÉLÉCOM/TÉLÉCOM SUDPARIS ÉCOLE DOCTORALE SCIENCES ET INGENIERIE EN CO-ACCREDITATION AVEC L’UNIVERSITÉ ÉVRY VAL D’ESSONNE THÈSE Pour obtenir le grade de DOCTEUR DE TÉLÉCOM SUDPARIS Spécialité : Informatique Présentée et soutenue par Xiaoping CHE Cross-Fertilizing Formal Approaches for Protocol Conformance and Performance Testing Soutenue le 26 Juin 2014 Devant le jury composé de : Directeur de thèse Prof. Stéphane MAAG --- Télécom SudParis Rapporteurs Prof. Mercedes MERAYO --- Universidad Complutense de Madrid Prof. Joanna TOMASIK --- Supélec Examinateurs
    [Show full text]
  • 100% Design Construction Quality Assurance Plan
    ERA Region 5 Records Ctr. ' 100% DESIGN 1 CONSTRUCTION QUALITY ASSURANCE PLAN »,J 3 HIMCO DUMP SUPERFUND SITE FINAL LANDFILL CLOSURE ELKHART, INDIANA 1 :r Prepared For: United States Enviromental Protection syEPA Agency Region 5 Chicago, Illinois Prepared By: US Army Corps of Engineers Omaha District APRIL 1998 CONSTRUCTION QUALITY ASSURANCE (COA) PLAN HIMCO DUMP SUPERFUND SITE ELKHART, INDIANA APRIL 1998 TABLE OF CONTENTS Page No. SECTION I - GENERAL 1. INTRODUCTION 1-1 2. DEFINITIONS 1-1 3. RESPONSIBILITY AND AUTHORITY I-4 4. REFERENCES I-5 SECTION II - GENERAL EARTHWORK CONSTRUCTION QUALITY ASSURANCE 1. GENERAL 11-1 ) 1.1 CQA Personnel 11-1 2. EARTHWORK/GRADING 11-1 ; 2.1. Equipment 11-1 i 2.2. Submittals 11-1 2.3. Quality Assurance Conformance Testing Requirements II-2 3. PRODUCTS II-2 ) 4. EXECUTION II-2 I 4.1. Excavation II-2 4.2. Backfill II-3 4.3. Compaction II-3 4.4. Finished Excavation, Fills, and Embankments II-4 4.5. Protection II-4 4.6. Adjustment of Existing Structures II-4 SECTION III - WASTE REGRADING AND RANDOM AND FOUNDATION FILL CONSTRUCTION QUALITY ASSURANCE 1. GENERAL 111-1 r 1.1 CQA Personnel HI-1 1.2. Submittals 111-1 2. WASTE REGRADING 111-1 2.1. Equipment 111-1 2.2. Quality Assurance Conformance Testing Requirements HI-1 2.3. Execution HI-2 3. RANDOM FILL LAYERS HI-2 3.1. Equipment HI-2 3.2. Quality Assurance Conformance Testing Requirements HI-2 3.3. Execution HI-2 4. FOUNDATION LAYERS HI-3 4.1. Equipment HI-3 4.2.
    [Show full text]
  • Automated Conformance Testing for Javascript Engines Via Deep Compiler Fuzzing
    Automated Conformance Testing for JavaScript Engines via Deep Compiler Fuzzing Guixin Ye Zhanyong Tang Shin Hwei Tan [email protected] [email protected] [email protected] Northwest University Northwest University Southern University of Science and Xi’an, China Xi’an, China Technology Shenzhen, China Songfang Huang Dingyi Fang Xiaoyang Sun [email protected] [email protected] [email protected] Alibaba DAMO Academy Northwest University University of Leeds Beijing, China Xi’an, China Leeds, United Kingdom Lizhong Bian Haibo Wang Zheng Wang Alipay (Hangzhou) Information & [email protected] [email protected] Technology Co., Ltd. University of Leeds University of Leeds Hangzhou, China Leeds, United Kingdom Leeds, United Kingdom Abstract all tested JS engines. We had identified 158 unique JS en- JavaScript (JS) is a popular, platform-independent program- gine bugs, of which 129 have been verified, and 115 have ming language. To ensure the interoperability of JS pro- already been fixed by the developers. Furthermore, 21 of the grams across different platforms, the implementation ofa JS Comfort-generated test cases have been added to Test262, engine should conform to the ECMAScript standard. How- the official ECMAScript conformance test suite. ever, doing so is challenging as there are many subtle defini- CCS Concepts: • Software and its engineering ! Com- tions of API behaviors, and the definitions keep evolving. pilers; Language features; • Computing methodologies We present Comfort, a new compiler fuzzing framework ! Artificial intelligence. for detecting JS engine bugs and behaviors that deviate from the ECMAScript standard. Comfort leverages the recent Keywords: JavaScript, Conformance bugs, Compiler fuzzing, advance in deep learning-based language models to auto- Differential testing, Deep learning matically generate JS test code.
    [Show full text]