INFORMATIK 2011 - Informatik schafft Communities www.informatik2011.de 41. Jahrestagung der Gesellschaft für Informatik , 4.-7.10.2011, Berlin Testing Microcontroller Software Simulators Thomas Reinbacher Dominique Guckel,¨ Stefan Kowalewski Embedded Computing Systems Group Embedded Software Laboratory Vienna University of Technology RWTH Aachen University Treitlstr. 3, 1040 Vienna, AT Ahornstraße 55, 52074 Aachen, DE [email protected] [email protected] Martin Horauer Dept. of Embedded Systems Univ. of Applied Sciences Technikum Wien Hochst¨ adtpl.¨ 5, 1200 Vienna, AT [email protected] Abstract: Software simulators that emulate equivalent behavior of physical micro- controllers play an important role in the process of software development for embed- ded systems from an early development stage (e.g. when no target hardware is avail- able) to the final verification process (e.g. used in combination with formal methods). Thus, much reliance is put on the correctness of these simulators. This paper presents a practicable approach to test auto-generated and custom microcontroller simulators (both closed and open-source) against a physical device. We show how to set up a test oracle that allows to run the simulators in parallel, validate individual runs based on a comparison of their accumulated state-space, and – in case an error is found – finger-point to the root cause of the error, thus giving valuable support for fixing the discrepancies. A case study shows that the presented testing framework was able to reveal non-trivial bugs in several implementations. 1 Introduction A microcontroller (MCU) software simulator emulates the functional and (occasionally) temporal execution environment offered by a physical microcontroller. In practice, MCU software simulators are used both by computer scientists and electronics engineers through- out different product development stages for various different activities (e.g., debugging, profiling, testing and verification). For example, firmware development can start prior to the availability of a functional physical target board, using the simulator for debugging purposes. An orthogonal example of an MCU simulator application is the [MC]SQUARE binary code verification framework, cf. [Sch08]. The model-checker of [MC]SQUARE re- lies on dedicated microcontroller simulators to construct the state-space out of a given binary program image. The state space is then traversed to prove the validity of given properties. Even though there is no theoretical limitation to precisely imitate a target microcontroller by a software simulator (by the Church-Turing thesis), a faithful simulator is hard to obtain erschienen im Tagungsband der INFORMATIK 2011 weitere Artikel online: Lecture Notes in Informatics, Band P192 http://informatik2011.de/519.html ISBN 978-3-88579-286-4 INFORMATIK 2011 - Informatik schafft Communities www.informatik2011.de 41. Jahrestagung der Gesellschaft für Informatik , 4.-7.10.2011, Berlin in practice. Thus, a vexing question arises in the context of simulation based verification: How “correct” is the simulation of the respective physical microcontroller? 1.1 Behavioral equivalence among heterogeneous MCU implementations The execution environment of a MCU simulator is typically made up of a linear address space that contains: general purpose CPU registers, the stack, special function registers (for the peripherals, the I/O subsystem and the interrupt controller), and the program and data memory (RAM, EEPROM, Flash, etc.). The simulator executes every instruction of a program in this emulated execution environment, mimicking the behavior of the instruc- tions of the physical MCU. Note that, the simulator does not necessarily need to emulate all the internals of a physical implementation (e.g. how a specific timer is implemented internally); it is usually sufficient to have an equivalent behavior on the register level ac- cessible via the programs’ instructions. In contrast, however, an MCU simulator often implements additional means, e.g., to model the environment and thus provide input to the MCU simulator from the outside. We evaluate the equivalence of a physical MCU and the respective simulator by their configurations CP and CS as follows: Let Addr = f0 x < jMemj : x 2 N0g denote the set of memory locations of the microcontroller, where Mem represents the (linear) address space of the microcontroller memory. We assume a memory mapped I/O architecture (e.g. Intel MCS-51), thus, registers and special function registers reside within Mem. In the following, let Nk = f0; : : : ; k 1g. A configuration C of a microcontroller is a tuple hpc; mi 2 Locs ¢ (Addr ! N2w ), where Locs is a finite set of program counter values, and m : Addr ! N2w is a map from memory locations (with bit-width w) to memory configurations. The state space of the program is thus a subset of Locs¢(Addr ! 0 0 0 N2w ). The initial microcontroller configuration C is h0x00; m i where m represents the configuration of all memory locations after power-up and 0x00 is the assumed reset vector. Two arbitrary configurations C hpc; mi and C0 hpc0; m0i are considered equivalent if the program counters as well as the memory configurations of C and C0 are identical, thus pc pc0 ^ m m0 holds for all combinations of valid and invalid instruction executions. 1.2 Related Work Commercial and freely available MCU simulators are either manually coded – see e.g. [LDG+08] – or auto-generated from some high-level description, see e.g. [GBK10]. Ap- plications of software simulators in practice are manifold, e.g.: • debugging when no target hardware is available, e.g., in an early development stage to shorten the time-to-market • firmware profiling to optimize performance, lower memory or power consumption, or for execution time analysis to verify real-time properties, see e.g. [MOZB10] erschienen im Tagungsband der INFORMATIK 2011 weitere Artikel online: Lecture Notes in Informatics, Band P192 http://informatik2011.de/519.html ISBN 978-3-88579-286-4 INFORMATIK 2011 - Informatik schafft Communities www.informatik2011.de 41. Jahrestagung der Gesellschaft für Informatik , 4.-7.10.2011, Berlin • evaluation of sensor-networks with regard to their timing behavior, power consump- tion, interoperability, etc., see [TLP05, EOF¨ +09] • evaluation of multiprocessors with regard to performance, etc., see [GKK+08] • formal verification as used by the [MC]SQUARE model-checker, to verify properties of binary firmware applications, see [RHS+10] From the above list it becomes obvious that an incorrect emulation by the MCU simulator of the physical implementation may eventually lead to hazardous outcomes. Related work in [MPRB09] reports on testing IA-32 CPU emulators, i.e., QEMU, VAL- GRIND, PIN and BOCHS. Their approach relies on fuzz-testing, that is they set both CP and CS to a synthetic state, execute one random instruction, and compare the resulting state. Using this approach, they were able to uncover several behavioral deviations in all considered CPU emulators. A method for verifying instruction-level simulators via step-by-step register state compar- ison to a hardware implementation is reported in [GL01]. They were able to verify two SPEC benchmarks on a MIPS 12000 CPU simulator. Other related literature on MCU simulator verification shows that, even carefully engi- neered CPU simulators do deviate from their silicon counterpart [Fer06, RKK07]. Formal verification, such as model checking [CGP99, BK08], equivalence checking by e.g. mul- tiway decision graphs [BT98], or interactive theorem proving [Bon10], seems to be well suited to cope with this kind of verification problem, however, often fail when applied to reasonably-sized real-life instances. Besides, formal approaches often need a clear under- standing and insight into verification goals and the desired correctness criteria. [Mon00] proposes a black-box testing method that relies on hardware analysis for test- case generation. The validation method aims at both determining the accuracy of the simulator and to pinpoint simulator errors for improving the accuracy. Based on their method they were able to improve the NEC V850 CPU core simulator from an average error rate of 11.2% down to 1.3%. 1.3 Contribution In contrast to the related work, we propose a lightweight verification framework for mi- crocontrollers that allows to run various simulators against each other and a hardware im- plementation as oracle that allows to vote over the individual simulation runs. Whenever a deviation is detected, the framework is able to highlight differences in the corresponding data memory dumps, guiding the test engineer to the root cause of the error. We use a mature microcontroller IP-core as a reference device along with a dedicated test-interface to perform the information exchange with a host computer. The presentation in this paper targets the Intel MCS-51 microcontroller architecture, however, is easily transferable to other platforms. erschienen im Tagungsband der INFORMATIK 2011 weitere Artikel online: Lecture Notes in Informatics, Band P192 http://informatik2011.de/519.html ISBN 978-3-88579-286-4 INFORMATIK 2011 - Informatik schafft Communities www.informatik2011.de 41. Jahrestagung der Gesellschaft für Informatik , 4.-7.10.2011, Berlin To summarize, the paper makes the following contributions: • a fully automated testing methodology targeting MCU simulators • an effective algorithm to generate test-programs stressing the simulators under test • a prototype implementation of our testing