arXiv:1208.0594v1 [cs.SE] 2 Aug 2012 oua eoyerrdtco n akn ninvariant paper. in this errors an in published findings Daikon, two our and report with and detector experimented BusyBox have error Valgrind, We of for i memory analyzer. top Linux methodology on popular for a works standard propose a methodology de-facto we Our a paper systems. BusyBox, embedded this product in In the errors in time. debugging uncovered run being at errors T time. of code development chance during high checks sanity to of leads lot a typ with softwares low away embedded of do consumption, requirements resource to less software Due and high. softwa embedded size quite the of is in context bugs time of the of probability In amount cycle. dif significant life a a development is takes errors often o program and produced of task being Debugging value violation. output assertion suc unexpected an programs, an crash, in program behaviors a observable unexpected of causes rmtemnfsaino t odbgrpr hudb able be should report bug source itself, good bug bug A the it. back the trace of from to manifestation is the different debugging from very in mani task be that main may the compilers note thus bug by to a caused worthwhile program of are is the few festation It a in code. and either incorrect design, mistakes producing programmer, its to or the due code by pro arise source made (e.g. bugs errors intended Most as result). and undocumented behaving incorrect mistake, from an flaw, it ducing error, prevents an that is feature in bug[4] time software A production high. at quite uncovered is result software being a embedded (e.g bugs As aids of etc.). assertions checking probability handlers, sanity signal sufficient In low exceptions, without software. a size, designed softwares normal system to often code embedded compared objectives, low these as with etc.) (e.g. keeping designed print objectives are foot systems additional memory software thereby some of Embedded pieces sophistication, time. with complex less of large in design level code to high programmers fairly the technologies allowing a compiler reached and have languages programming communi Today’s industry and signifi- academic aroused the in has debuggin interest the systems research result cant and software a embedded process As for deadlines. development problem market paced to time fast stringent param today’s of in issue our an importance in inextricably and system computation dominate invariably, embedded therefore and of to is debugging computer and come with Validation life. interact have everyday we systems way and the software Embedded Abstract eugn eoyIse nEbde Linux: Embedded In Issues Memory Debugging Dbgigdntstepoeso eetn root detecting of process the denotes —Debugging .I I. NTRODUCTION adaIsiueo ehooyIda ttsia Institu Statistical Indian Technology of Institute Haldia ataPai a nua Banerjee Ansuman Ray Pratim Partha aeStudy Case A ically ficult the , ount code the , as h ion his ty. re re n g r s - - . npriua,w okwt w fte2 us namely, bugs, 21 the some of of two analysis. causes with invariant and root work and we the checking particular, locate memory In using to bugs is Our the these paper [8]. of of BusyBox this in many in bugs 21 objective provides Researchers reported size. and It code BusyBox smaller analyzed devices. a has but embedded utilities standard standard in de-facto the Linux bug. BusyBox[1], the for on of method our causes this employ possible We of of manifestatio set result bug a the The to to points found. respect and with thus a analyzed invariants and is specificatio comparison program same the the buggy compare for bug) the symbolically and the from we having invariants (not bug, program the of stable demonstrates set that the program input extract buggy test a a Given a detectors. and oldest followed invariant the is used of This one widely Daikon[2], error. most using memory a analysis observed invariant we deduce an the by errors, of and causes Valgrind[3], observable of the using set Given checking test. method memory debugging under perform Our analysi program for invariant software. the and methodology on embedded checking a memory in performing propose involves errors cause. root we usage the paper, locate memory and this bug a In of manifestation a in take to nti approach, this softw propos deployed has In in al errors patches et. automatically Perkins[6] that H. system Jeff publication, recent a In ecie u xeietto oewt Daikon. with Section done while experimentation Valgrind, Section our using BusyBox. describes done V of experiments overview presents rela an presents IV presents II III Section Section follows: the as work. organized to is us paper versions. This lead A program two produced Daikon. the thus using between invariants anomaly extraction demonstrate behavioral the invariant not of does perform which comparison and one bug another and the namely, one BusyBox, i buggy of access versions the memory two where choose source we of Secondly faulty. lines memory the reports the to directly pointing on Valgrind errors, Valgrind BusyBox. steps. using of two checking version in buggy memory works perform analysis we cause Firstly, root for methodology Our 1.4.2. version BusyBox in execution top oho hc eutin result which of both , Clearview I R II. LTDWORK ELATED 6 sue ocretunknown correct to used is [6] te emnainviolation segmentation have ares. da ed arp ted nd on n n s s errors in the commercial off-the-shelf (COTS) softwares. As III. BUSYBOX per its architecture, learning, monitoring, correlated invariant identification, candidate repair generation and candidate re- BusyBox is a fairly comprehensive set of programs needed pair evolution phases perform invariant analysis and mem- to run a Linux system. Moreover it is the de-facto standard ory checking rigorously. Daikon[3] is used in its learning for Embedded Linux systems, providing many standard Linux component to analyze the various invariants present in the utilities, but having small code size (size of executables), code. HeapGuard and Determina Memory Firewall[5] help in than the GNU Core Utilities. BusyBox provides compact monitoring phase incorporating the monitor to detect a failure replacements for many traditional full-blown utilities found and the failure location. on most desktop and embedded Linux distributions. Examples include the file utilities such as ls, cat, cp, dir, head, and tail. The work [7] illustrates invariant analysis by Daikon while BusyBox also provides support for more complex operations, performing mutation test. Mutation test basically measures such as ifconfig, netstat, route, and other network utilities. the adequacy of a test suite by introducing artificial defects (mutations) into a test program. The assessment of the muta- BusyBox is remarkably easy to configure, compile, and use, tions are done by dynamic invariants. Different versions (with and it has the potential to significantly reduce the overall invariant checkers) originating from a given source program, system resources required to support a wide collection of are checked by JAVALANCE (relies on invariant detection common Linux utilities. BusyBox in general case can be built engine of DAIKON) to find the best survived mutation. on any architecture supported by gcc. One of the first efforts for debugging program changes is [9]. BusyBox is modular and highly configurable, and can be This paper identifies the changes across program versions and tailored to suit particular requirements. The package includes a searches among subsets of this change set to identify which configuration utility similar to that used to configure the Linux changes could be responsible for the given observable error. kernel. The commands in BusyBox are generally simpler In evolving program debugging, a buggy program version is implementations than their full-blown counterparts. In some simultaneously analyzed with older stable program version. cases, only a subset of the usual command line options is supported. In practice, however, the BusyBox subset of Recently a paper proposed the DARWIN approach [10] for command functionality is more than sufficient for most general debugging program versions. DARWIN performs dynamic embedded requirements. symbolic execution along the execution of a given test input in two programs. DARWIN method is basically suited for debug- The BusyBox bundle functions as a single executable where ging branch errors (or code missing errors where the missing the different utilities are actually passed on at the command code contains branches). Dynamic slicing [11] has so far been line for separate invocation. It is not possible to build the studied as an aid for program debugging and understanding. A individual utilities separately and run them stand alone. For ex- recent work[12] uses dynamic program dependencies to seek ample, for running the arp utility, we need to invoke BusyBox the involved parts of an input that are responsible for a given as busybox arp -Ainet and record the execution trace. failed output. Research such as [13],[14] combine symbolic Since we work on the binary level, the buggy implementation execution and dependency analysis for test suite augmentation. for us is the BusyBox binary, which has a large code base (about 121000 lines of code). [15] focuses on debugging a given failing test case based upon golden implementation driven software debugging. There are various published methodologies that search for failing tests A. Bugs in BusyBox (that demonstrate an observable error), such as – the DSD- crasher which combines static and dynamic analysis[16], and KLEE [8] has reported some bugs in BusyBox 1.10.2. by a bug finding methodology approaches relying upon software test generation method. This paper checked all 279 BusyBox model checking (e.g.,[17]). Symbolic execution has also been tools in series to demonstrate its applicability to bug finding. used for generating problematic or failing tests. The work on It has been seen that 21 bugs are present in BusyBox. We Directed Automated Random Testing (DART) [18] combines tried them on BusyBox version 1.4.2 and found 6 of them symbolic and concrete execution to explore different paths of a still persist namely, arp -Ainet, tr [, top d, printf %Lu, ls -co, program. A recent work [8] uses symbolic execution on GNU install -m. Coreutilities as well as BusyBox to compute test-suites with high path coverage. Our objective in this paper is to locate the root causes of some of these bugs using memory checking and invariant analysis. Our work proposes an experimental framework for analyzing In particular, we work with two of the 6 bugs, namely, arp some published bugs in BusyBox. We employ Valgrind and and top, both of which result in segmentation violation on Daikon in an attempt to locate the root causes of two memory execution in BusyBox 1.4.2. To do so, we created a debug bugs in BusyBox utilities. build of BusyBox using appropriate compiler options so that debugging symbols are present in the binary. We explain our findings in the next two sections. ==1571== Use of uninitialized value of size 8 IV. EXPERIMENTS WITH VALGRIND ==1571== at 0x4A06794: strcmp (mc_replace_strmem.c:341) ==1571== by 0x44D7BD: get_hwtype (interface.c:936) ==1571== by 0x444B77: arp_main (arp.c:463) The Valgrind tool suite provides a number of debugging and ==1571== by 0x407C08: run_applet_by_name (applets.c:489) profiling tools that helps programmers identify memory errors ==1571== by 0x407DDC: busybox_main (busybox.c:143) ==1571== by 0x407AB7: run_applet_by_name (applets.c:480) in programs. Valgrind is an instrumentation framework for ==1571== by 0x407E56: main (busybox.c:72) building dynamic analysis tools. It can detect many memory- ==1571== ==1571== Invalid read of size 1 related errors that are common in C and C++ programs and ==1571== at 0x4A06794: strcmp (mc_replace_strmem.c:341) that can lead to crashes and unpredictable behavior. Valgrind ==1571== by 0x44D7BD: get_hwtype (interface.c:936) ==1571== by 0x444B77: arp_main (arp.c:463) is an instrumentation framework for building dynamic analysis ==1571== by 0x407C08: run_applet_by_name (applets.c:489) tools. Valgrind architecture is modular, so new tools can be ==1571== by 0x407DDC: busybox_main (busybox.c:143) created easily and without disturbing the existing structure. Valgrind comes with a set of tools each of which performs Fig. 1. BusyBox Arp run with valgrind some kind of debugging, profiling, or similar task that helps 930 const struct hwtype *get_hwtype (const char *name) to improve programs. Some of them are as below. 931 { 932 const struct hwtype * const *hwp; 933 hwp = hwtypes; • The default tool that comes along with Valgrind, is 934 while (*hwp != NULL) Memcheck, a memory error detector. 935 { 936 if (!strcmp((*hwp)->name, name)) • Cachegrind – a cache and branch-prediction profiler. 937 return(*hwp); • 938 hwp++; Massif – a heap profiler. 939 } • Helgrind – a thread error detector. 940 return NULL; • Memory leak detector. 941 } • Conditional jump or uninitialized value dependent move 444 int arp_main(int argc, char **argv) detector. 445 { 446 char *hw_type; • Invalid Read / Write Detector. 447 char *protocol; 448 449 /* Initialize variables... */ Valgrind is designed to be as non-intrusive as possible. It 450 ap = get_aftype(DFLT_AF); works directly with existing executables. We invoke Valgrind 451 if (!ap) 452 bb_error_msg_and_die("%s: %s not as valgrind –options executable program in command line. supported", DFLT_AF, "address family"); The most important option is –tool which dictates which 453 454 getopt32(argc, argv, "A:p:H:t:i:adnDsv", Valgrind tool to run. Regardless of which tool is in use, &protocol, &protocol, Valgrind takes control of the program before it starts. Debug- 455 &hw_type,&hw_type,&device); 456 argv += optind; ging information is read from the executable and associated 457 if (option_mask32 & ARP_OPT_A || option_ libraries, so that error messages and other outputs can be mask32 & ARP_OPT_p) { 458 ap = get_aftype(protocol); phrased in terms of source code locations, when appropriate. 459 if(ap==NULL) Program is then run on a synthetic CPU provided by the 460 bb_error_msg_and_die("%s: unknown %s",protocol, "address family"); Valgrind core. As new code is executed, the core hands the 461 } code to the selected tool. The tool adds its own instrumentation 462 if (option_mask32 & ARP_OPT_A || option_ mask32 & ARP_OPT_p) { code to this and hands the result back to the core, which 463 hw = get_hwtype(hw_type); coordinates the continued execution of this instrumented code. 464 if(hw==NULL) 465 bb_error_msg_and_die("%s: unknown %s", hw_type, "hardware type"); A. Locating the arp bug in BusyBox 466 hw_set=1; 467 } 468 //if (option_mask32 & ARP_OPT_i)... -i The arp utility manages the kernel’s network neighbor cache. 469 It can add or delete entries to the cache, or display the 470 if (ap->af != AF_INET) { 471 bb_error_msg_and_die("%s: kernel only cache’s current content. There is a bug in the BusyBox arp supports ’inet’", ap->name ); implementation: running with the command-line option 472 } arp 473 -Ainet results in a segmentation fault. 474 /* If no hw type specified get default */ 475 if (!hw) { To isolate the root cause of this error, we ran the arp utility 476 hw = get_hwtype(DFLT_HW); 477 if(!hw) of BusyBox through Valgrind using the argument -Ainet. This 478 bb_error_msg_and_die("%s: %s not results in segmentation fault when run in BusyBox version supported",DFLT_HW, "hardware type"); 479 } 1.4.2, on valgrind. Figure 1 shows the test results. .... Figure 2 shows a fragment of the source code of arp in } BusyBox. With the command-line argument -Ainet, line 454 sets the ARP_OPT_A mask in the variable option_mask32. Fig. 2. BusyBox arp application ==1575== Invalid read of size 1 ==1575== at 0x43310E: xstrtou_range_sfx ( xatonum_template.c:27) 414 int top_main(int argc, char **argv) ==1575== by 0x463605: top_main (top.c:431) 415 { ==1575== by 0x407C08: run_applet_by_name 416 int count, lines, col; (applets.c:489) 417 unsigned interval = 5; ==1575== by 0x407DDC: busybox_main /* default update rate is 5 seconds */ (busybox.c:143) 418 unsigned iterations = UINT_MAX; ==1575== by 0x407AB7: run_applet_by_name /* 2ˆ32 iterations by default :) */ (applets.c:480) 419 char *sinterval, *siterations; ==1575== by 0x407E56: main (busybox.c:72) 420 #if ENABLE_FEATURE_USE_TERMIOS ==1575== Address 0x0 is not stack’d, malloc’d 421 struct termios new_settings; or (recently) free’d 422 struct timeval tv; ==1575== Process terminating with default action 423 fd_set readfds; of signal 11 (SIGSEGV) 424 unsigned char c; ==1575== Access not within mapped region at 425 #endif /* FEATURE_USE_TERMIOS */ address 0x0 426 ==1575== at 0x43310E: xstrtou_range_sfx 427 /* do normal option parsing */ (xatonum_template.c:27) 428 interval = 5; ==1575== by 0x463605: top_main (top.c:431) 429 opt_complementary = "-"; ==1575== by 0x407C08: run_applet_by_name 430 getopt32(argc, argv, "d:n:b", (applets.c:489) &sinterval, &siterations); ==1575== by 0x407DDC: busybox_main 431 if (option_mask32 & 0x1) (busybox.c:143) interval = xatou(sinterval);// -d ==1575== by 0x407AB7: run_applet_by_name 432 if (option_mask32 & 0x2) (applets.c:480) iterations = xato(siterations); // -n ==1575== by 0x407E56: main (busybox.c:72) 433 //if (option_mask32 & 0x4) // -b 434 435 /* change to /proc */ Fig. 3. BusyBox top run with valgrind 436 xchdir("/proc"); 437 #if ENABLE_FEATURE_USE_TERMIOS 438 tcgetattr(0, (void *) &initial_settings); 439 memcpy(&new_settings, &initial_settings, Because no or option was given in the command line, sizeof(struct termios)); H t 440 /* unbuffered input, turn off echo */ hw_type was set to NULL. The bug is at line 462: instead 441 new_settings.c_lflag &= ˜(ISIG | ICANON | ECHO | ECHONL); of checking the mask of hardware type, the program checks 442 for the mask of address family, ARP_OPT_A, which led to line 443 signal(SIGTERM, sig_catcher); 444 signal(SIGINT, sig_catcher); 463, which passed the NULL hw_type into get_hwtype 445 tcsetattr(0, TCSANOW, function, and caused a segmentation fault at line 936 due to a (void *) &new_settings); NULL argument in the string comparison function . 446 atexit(reset_term); strcmp 447 #endif /* FEATURE_USE_TERMIOS */ 448 449 #if ENABLE_FEATURE_TOP_CPU_USAGE_PERCENTAGE B. Locating the top bug in BusyBox 450 sort_function[0] = pcpu_sort; 451 sort_function[1] = mem_sort; 452 sort_function[2] = time_sort; The top program provides a dynamic real-time view of a 453 #else running system. It can display system summary information 454 sort_function = mem_sort; 455 #endif /* FEATURE_TOP_CPU_USAGE_PERCENTAGE */ as well as a list of tasks currently being managed by the Linux 456 kernel. The types of system summary information shown and 457 while (1) { 458 procps_status_t *p = NULL; the types, order and size of information displayed for tasks are all user configurable and that configuration can be made Fig. 4. BusyBox top application persistent across restarts. Top which is run with option d is buggy in busybox v1.4.2. If we run BusyBox top with option d, it will crash and .... a segmentation fault is reported. Option d in top must be if (spec_flgs & ALL_ARGV_IS_OPTS) { /* process argv is option, for example "ps" applet */ followed by argument which specifies the length of the refresh if (pargv == NULL) delay in the process statistic (in seconds). The top utility pargv = argv + optind; while (*pargv) { should behave similarly when invoked with -d or d. The - // printf ("argv non option: %s\n", *pargv); is complementary in this command. In the normal execution c = **pargv; if (c == ’\0’) { of top on GNU, both top d and top -d wait for the parameter pargv++; for refresh delay. However, BusyBox top behaves differently. } else { Figure 3 shows the top results. (*pargv)++; goto loop_arg_is_opt; Figure 4 shows a fragment of the source code of top in } } BusyBox. With the command-line argument d, there is a crash } in line 431 as reported by Valgrind. BusyBox’s top uses a .... native getopt32 and checks for ’-’ which should have been ignored by top program. Going inside the getopt32 function Fig. 5. BusyBox getopt32 in Line 430, we find the code snippet shown in Figure 5. The pargv here is not NULL, thus there are some non-options Utility Trace size (MB) Time for Trace collection (s) arp 298, 569 99, 194 to be processed. In our case here, the non-options to be top 298, 570 92, 184 processed is string ’d’ and the following while-loop treat it as an option. However as soon as ’d’ is treated as an option, it TABLE I is no longer checked whether ’d’ requires an argument. Thus Experimental Results on BusyBox bugs using Daikon the argument stored in sinterval in line 431 of top main is NULL leading to the crash.

from BusyBox 1.16.0. Trace Size is the size of the trace in V. EXPERIMENTS WITH DAIKON Megabytes. The time for trace collection is the time required by kvasir to record the execution run and generate the trace. Daikon is a dynamic invariant detector that reports likely However, in all the cases, Daikon ends up with an error on program invariants. An invariant can be defined as a property the trace files thus generated and aborts, without reporting any that holds at a certain point or points in a program. Invariants invariants. Hence, we are unable to generate the invariants. have a lot of applications. Some example invariants are as However, we tested our methodology on smaller examples and > arr follows: x y, y = 2*x-1, array is sorted. found diagnostic results. Currently, we are investigating the The underlying approach of dynamic invariant detection is cause of the scalability issue of Daikon. to run a program and observe the values computed by the program. From the observed values, a dynamic invariant VI. CONCLUSION detector tries to infer properties that were true over the observed executions. Daikon interacts with the program, firstly In this paper we propose a methodology for debugging errors by obtaining the data trace by running the program under the in BusyBox, a de-facto standard for Linux in embedded control of a front end (also known as an instrumenter or tracer) systems. Our methodology works on top of Valgrind, a popular that records information about variable values, and finally, by memory error detector and Daikon, an invariant analyzer. We running the invariant detector over the data trace generated. have experimented with two published errors in BusyBox This detects invariants in the recorded information generated. with Valgrind and found promising results. Currently, we are Daikon creates a .inv file that contains the invariants in binary investigating the possibility of correlating the published bugs form. with the invariant differences after generating the invariants produced by Daikon. A. Experimental Findings on arp and top using Daikon REFERENCES We tried to run Daikon on two versions of BusyBox namely, 1.4.2 and 1.16.0. Our objective is to collect the invariants from [1] BusyBox. http://busybox.net/. the two versions when run on the same utilities. However, we [2] Daikon. http://pag.csail.mit.edu/daikon/. [3] Valgrind. http://valgrind.org/. were unsuccessful in our attempt due to limitations of Daikon [4] Software bug. http://en.wikipedia.org/wiki/Software bug. in processing large trace files. [5] Kiriansky,V.,Bruening,D.,and Amarasinghe,S.Secure execution via pro- gram shepherding. In USENIX Security(Aug.2002). Our methodology has two main steps: [6] Jeff.H. Perkins, Sunghun Kim, Sam Larsen, Saman Amarasinghe, Jonathan Bachrach, Michael Carbin, Carlos Pacheco, Frank Sherwood, 1) Generate the program trace using using the kvasir util- Stelios Sidiroglou, Greg Sullivan, Weng-Fai Wong, Yoav Zibin, Michael D. Ernst, and Martin Rinard, Automatically patching errors in deployed ity [2] software. In SOSP’ , 2009. 2) Analyze the trace file produced above using daikon to [7] David Schuler, Valentin Dallmeier,and Andreas Zeller, Efficient mutation generate the invariants. testing by checking invariant violations. In ISSTA, 2009. [8] C. Cadar, D. Dunbar, and D. Engler. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In The collected traces are significantly large for both the ver- OSDI, 2009. sions. Table I shows a summary of our findings. During [9] A. Zeller. Yesterday my program worked, today it does not. Why? In the trace collection phase, kvasir invokes valgrind internally ESEC-FSE, 1999. [10] D. Qi, A. Roychoudhury, Z. Liang, and K. Vaswani. DARWIN: An and shows similar segmentation fault as obtained by us by approach for debugging evolving programs. In ESEC-FSE, 2009. running Valgrind standalone. Having generated the trace files, [11] B. Korel and J. W. Laski. Dynamic program slicing. Information we proceed to Step 2 to generate the invariants. Unfortunately, Processing Letters, 29(3):155163, 1988. [12] J. Clause and A. Orso. Penumbra: Automatically identifying failure- Daikon is unable to process the large trace files thus generated relevant inputs using dynamic tainting. In ISSTA, 2009. and aborts, without reporting any invariants. [13] D. Qi, A. Roychoudhury, and Z. Liang. Test generation to expose changes in evolving programs. In ASE, 2010. The first column of Table I is marked ”Utility” — this [14] R. Santelices, P. Chittimalli, T. Apiwattanapong, A. Orso, and M. represents the utility whose observable error is being diag- Harrold. Test-suite augmentation for evolving software. In ASE, 2008. [15] A. Banerjee, Abhik R. Choudhury, Johannes A. Harlie, Zhenkai Liang, nosed. Each entry in the second and third columns is a tuple, Golden implementation driven software debugging, to be processed in where the first entity is from BusyBox 1.4.2 and the second FSE-18, 2010. [16] C. Csallner and Y. Smaragdakis. DSD-Crasher: a hybrid analysis tool for bug nding. In ISSTA, 2006. [17] B. Gulavani, T. Henzinger, Y. Kannan, A. Nori, and S. Rajamani. SYNERGY: A new algorithm for property checking. In FSE, 2006. [18] P. Godefroid, N. Klarlund, and K. Sen. DART: Directed automated random testing. In ESEC-FSE, 2005.