Testing Error Handling Code in Device Drivers Using Characteristic Fault Injection Jia-Ju Bai, Yu-Ping Wang, Jie Yin, and Shi-Min Hu, Tsinghua University https://www.usenix.org/conference/atc16/technical-sessions/presentation/bai

This paper is included in the Proceedings of the 2016 USENIX Annual Technical Conference (USENIX ATC ’16). June 22–24, 2016 • Denver, CO, USA 978-1-931971-30-0

Open access to the Proceedings of the 2016 USENIX Annual Technical Conference (USENIX ATC ’16) is sponsored by USENIX. Testing Error Handling Code in Device Drivers Using Characteristic Fault Injection

Jia-Ju Bai, Yu-Ping Wang, Jie Yin, and Shi-Min Hu Department of Computer Science and Technology, Tsinghua University

Abstract tion [34]. For example, “bad address” (EFAULT) is a Device drivers may encounter errors when communi- common error should be handled, but it happens only cating with OS kernel and hardware. However, error when the memory or I/O address is invalid. Another handling code often gets insufficient attention in driver example is hardware error, which happens only when development and testing, because these errors rarely the hardware malfunctions. Triggering these errors in occur in real execution. For this reason, many bugs are real environment is very hard and uncontrollable. hidden in error handling code. Previous approaches for To simulate software and hardware errors at runtime, testing error handling code often neglect the characteris- software fault injection (SFI) is often used in driver tics of device drivers, so their efficiency and accuracy testing. This technique mutates the code to inject specif- are limited. In this paper, we first study the source code ic errors into the program, and enforces error handling of Linux drivers to find useful characteristics of error code to be executed at runtime. Linux Fault Injection handling code. Then we use these characteristics in fault Capabilities Infrastructure (LFICI) [43] is a well-known injection testing, and propose a novel approach named project integrated in Linux kernel. It can simulate com- EH-Test, which can efficiently test error handling code mon errors, such as memory-allocation failures and bad in drivers. To improve the representativeness of injected data requests. Inspired by LFICI, other fault injection faults, we design a pattern-based extraction strategy to approaches [7, 13, 23, 32] have been proposed in recent automatically and accurately extract target functions years, and they have shown promising results in driver which can actually fail and trigger error handling code. testing and bug detection. However, these approaches During execution, we use a monitor to record runtime still have some limitations in practical use. information and pair checkers to check resource usages.  The representativeness of injected faults is often We have evaluated EH-Test on 15 real Linux device neglected, and most injected faults are random or drivers and found 50 new bugs in Linux 3.17.2. The manually selected. Random faults can not reflect code coverage is also effectively increased. Comparison real errors well. Manually selected faults often omit experiments to previous related approaches also show representative injected faults. the effectiveness of EH-Test.  Numerous redundant test cases are generated. In fact, many generated test cases may cover the same 1. Introduction error handling code, but they all need to be actually tested at runtime. For this reason, they often spend As important components of the operating system, de- much time in runtime testing. vice drivers control hardware and provide fundamental  Only several kinds of faults can be injected, such as supports for high-level programs. During driver execu- memory-allocation failures. But these faults can not tion, different kinds of occasional errors may occur, cover most error handling code in drivers. such as kernel exceptions and hardware malfunctions  Much manual effort is needed. The kinds and plac- [31]. Therefore, device drivers need error handling code es of injected faults are often manually decided. to assure reliability. But in some drivers, error handling In fact, previous fault injection approaches aim to code is incorrect or even missed. In these drivers, seri- support general software, but they neglect the character- ous problems like system crashes and hangs may occur istics of target programs. To relieve their limitations, we when occasional errors are triggered. According to our should consider the key driver characteristics in SFI. study on Linux driver patches, more than 40% of ac- For example, because drivers are often written in C, so cepted patches add or update corresponding error han- built-in error handling mechanisms (such as “try-catch”) dling code. It shows that error handling code in device are not supported. For this reason, the developers often drivers is not reliable enough, so testing error handling use an if check to decide whether the error handling code and detecting bugs inside are very necessary. code should be triggered in device drivers. This charac- A challenge of testing error handling code is that oc- teristic can help to decide which functions can actually casional errors are infrequent to happen in real execu- fail and should be fault-injected.

1

USENIX Association 2016 USENIX Annual Technical Conference 635 n his aer e irs sud inu drier code and 4) High automation and scalability. os oring ind hree useul characerisics in error handling code rocedure o es is auomaed including arge function return value trigger few branches and check uncion eracion aul inecion and escase eecu decision hen ased on hese characerisics and ion nd i can suor man inds o eising driers e roose a racical aroach named es o n his aer e mae he olloing conriuions eicienl es error handling code and deec ugs in  e sud he source code o inu deice driers side irsl es uses a pattern-based extraction and ind hree useul characerisics in error han strategy o erac arge uncions hich can ail rom dling code he caured runime races o normal eecuion his  ased on he aerns o error handling code in sraeg can auomaicall and accurael erac real driers e design a aernased eracion srae arge uncions o imroe he reresenaieness o g o auomaicall and accurael real arge unc inec auls hen e generae es cases corruing ions as reresenaie ineced auls he reurn alues o arge uncions e e run each  ased on he characerisics e design a racical es case on he real hardare and use a monior o rec aroach named es hich can eicienl es ord runime inormaion and air checers o chec re error handling code in deice driers and deec sourceusage iolaions hese air checers conain he ugs inside asic inormaion o resourceacuiring and resource  e ealuae es on deice driers in inu releasing uncions hich can e oained rom secii and and reseciel ind and caion mining echniues and user con ugs ll he deeced ugs in hae een iguraion uring drier eecuion ssem crashes and conirmed deeloers he code coerage is also hangs can e easil ideniied hrough ernel crash logs eeciel increased in runime esing e also or user oseraion er drier eecuion es erorm comarison eerimens o reious a can reor resourcerelease omissions e hae imle roaches and ind ha es can deec he ugs mened es using and ealuaed i on hich are missed hem he eerimenal resuls inu driers o hree classes he resuls sho ha sho ha es can eicienl erorm drier esing and accurael ind real ugs es can accurael ind real ugs in error handling he res o his aer is organied as ollos ecion code and imroe code coerage in runime esing inroduces he moiaion ecion resens he hree omarison eerimens o reious aroaches also characerisics o deice driers ound our sud on sho is eecieness inu drier code ecion resens our aernased omared o reious aroaches or esing eracion sraeg ecion inroduces es in driers our aroach hae our adanages deail ecion shos our ealuaion on inu de 1) Representative injected faults. e design a a ice driers and comarison eerimens o reious ernased eracion sraeg o auomaicall and ac aroaches ecion inroduces he relaed or and curael erac real arge uncions as reresenaie ecion concludes his aer ineced auls uses code aerns o decide heher a uncion can acuall ail in drier eecuion his sra 2. Motivation eg can largel imroe he eecieness o 2) Efficient test cases. ccording o our sud man o ensure he reliaili o deice driers error han driers hae e ranches in error handling code so dling code should e correcl imlemened o handle inecing a single aul in each es case is enough o dieren inds o occasional errors u in ac error coer mos error handling code oreoer our aern handling code is incorrec or een missed in some dri ased eracion sraeg can iler man unreresena ers so hardoind ugs ma occur during eecuion n ie ineced auls hereore he es cases generaed his secion e irs reeal his rolem using a con es are eicien and he ime usage o runime cree eamle and our sud on inu drier aches esing can e largel shorened and hen e sech he soare aul inecion ech 3) Accurate bug detection. inecing reresena niue used in his aer ie auls es can realisicall simulae dieren 2.1 Motivating Example inds o occasional errors o coer error handling code e irs moiae our or using a real inu drier oreoer es runs on he real hardare and uses bnx2 his drier manages roadcom ereme eac eecuion inormaion o erorm analsis hese herne onroller igure shos a ar o is source oins assure he accurac o ug deecion code in inu he uncion bnx2_init_board calls

1 es rogram can e donloaded rom he lin pci_request_regions (line ) o reues and http://oslab.cs.tsinghua.edu.cn/EHTest/index.html memor resources hen iniialiing he hardare an

636 2016 USENIX Annual Technical Conference USENIX Association occasional error occurs hen maing us memor ino Driver Class Accepted Patches Error Handling he uncion ioremap_nocache (line ) ill () ail and reurn n his siuaion he drier calls () pci_release_regions (line ) o release allocaed oe () resources in error handling code eieing he code () he uncion kzalloc (line ) is called o allocae eor () ernelsace memor u his memor is no reed in Total () error handling code so a memor lea occurs n ac his ug sill remains in inu ale ud resul o inu drier aches n his eamle e hae hree indings irsl error 2.2 Study on Linux Patches handling code in driers is oen used o release alloca ed resources and undo recen oeraions is e o clearl illusrae he reliaili o curren error han cause ha man driers are ased on he fail-stop model dling code in deice driers e mae a sud on inu drier aches e manuall read aches in he ach namel a simle error can orce he drier o ei ue o his eaure man ugs in error handling are or roec and selec acceed aches rom hem in relaed o resourceusage iolaions such as resource ul hese aches are rom drier classes leas and deadlocs econdl error handling code is namel us driers us driers oer oen rien in a searae segmen in driers (line driers realime cloc () driers and neor in igure is an eamle) and dieren driers mong hem e ideni hose hich add or “goto” targe laels handle dieren errors his goto- udae corresonding error handling code he resul is based sraeg is recommended he inu ernel lised in ale he irs column resens he drier documenaion ecause his sraeg can simli class name he second column shos he numer o error handling logic and reduce reeaed code hirdl acceed aches he hird column shos he numer ugs in error handling code are hardoind is e and ercenage o acceed aches add or udae corre cause ha error handling code is rarel eecued and sonding error handling code mainainers a insuicien aenion o i n he eam rom ale e ind ha acceed aches add le rom inu (released in oemer ) o or udae corresonding error handling code n hese (released in coer ) he memor lea in acceed aches man are used o i common ugs igure had no een ied hus i is er necessar such as memor leas and null oiner dereerences o reeal and deec ugs in error handling code ne reason or his henomenon is ha comle conrol los and dieren inds o occasional errors mae i Path: linux-3.1.1/drivers/net/bnx2.c diicul o imlemen correc error handling code n 7869. static int __devinit bnx2_init_board(….) oher reason is ha error handling code is oen rig 7870. { gered seciic and inreuen condiions (such as …… insuicien memor and hardare errors) so deelo 7885. bp->temp_stats_blk = kzalloc(…); ers hardl es i ell a runime …… n rie curren error handling code in deice driers 7906. rc = pci_request_regions(pdev, …); is no reliale enough as e eeced and man ugs …… are hidden in i nce hese ugs are riggered serious 7937. bp->regview = ioremap_nocache(…); 7938. if (!bp->regview) { ssem rolems ma occur such as crashes and re 7939. dev_err(“Cannot map register space, aborting\n”); source leas hereore i is imoran and necessar o 7940. rx = -ENOMEM; es error handling code in deice driers and deec 7940. goto err_out_release; ugs inside 7941. } …… 2.3 Software Fault Injection 8247. err_out_release: oare aul inecion () is a idel used echniue 8248. pci_release_regions(pdev); o esing error handling code inenionall inroduc 8249. err_out_disable: es auls or occasional errors ino he rogram and hen 8250. pci_disable_device(pdev); ess heher he rogram can correcl handle he in 8251. pci_set_drvdata(pdev, NULL); eced auls or errors a runime n his aer e use 8252. err_out: 8253. return rc; o es driers and deec ugs o hel eer un 8254. } dersand his aer e elain seeral erms aou

igure ar o he n drier code in inu 2 achor roec http://patchwork.ozlabs.org/

USENIX Association 2016 USENIX Annual Technical Conference 637 Fault Injection. t t t 3. Characteristics t t t t t t tt t t t t t t t t t tt t t t t t t t t t t t ioremap_nocache t t tt t t t t t t Fault Representativeness. t t t tt t t t t t t t t t t 3.1 Function Return Value Trigger t t tt t t t t t t t t tt t tt t t t t t tt t t t t tt t t t t ioremap_nocache t t Target Function. tt t t t tt error handling code is trig- t t gered by a bad function return value t tt tt t t t t t t t If a tar- t tt t get function is real, its failure can be a representative injected fault, because its failure can cause a real error t and actually trigger error handling code t t t t t t t tt t t t In the study, we search for “goto” statements in the tt t t t t t t ioremap_nocache t t t t goto-based tt t t t t t t tt t t t t False Positive. t t t t t t t t t the number of “goto” statements; the fourth column t t t t tt t t the number and proportion of “goto” statements t t t t “if” t t t tt t tt tt t of “goto” ttt t “if” t Driver Class Number “Goto” Statement Return Value t t t tt t t t t tt t t t t t tt t t t t tt t t t t t t t t Total t t t ttt t t t of “goto” statements t “goto” ttt Driver Class Number Error handling Without Branch t t t t t t tt tt 3.2 Few Branches t t tt t if tt t t fail-recovery t t t t Total t t t t t t t tt

638 2016 USENIX Annual Technical Conference USENIX Association ifferent from usermode applications, many deice ee atternbased etraction stratey driers are based on the fail-stop model amely, func_set ; cand_set ; fault_set ; when an error occurs, the drier only handles it and 2 func_set caed nctions in norma exection traces; prepares to eit, but other errors are neer handled at 3 foreach func in func_set do that time hus, there are few if branches in error han- if etetpe(func) == integer or pointer then dling code of device drivers o alidate this character ddet(cand_set, func); istic, we also write a proram to automatically analye 6 end if the source code of these inu driers In the study, 7 end foreach we first filter out all annotations and blan lines, and 8 foreach func in cand_set do then count source code lines with and without if branch 9 if func’s RetVal is ceced b “if” in te driver then es in error handlin code he result is shown in able ddet(fault_set, func); ’s is ceced in oter drivers he first column shows the drier class name; the se else if func RetVal then 2 ddet(fault_set, func); cond column shows the number of driers in each class; 3 else if func’s RetVal is speciied to be ceced then the third column presents the number of source code ddet(fault_set, func); lines in error handlin code; the fourth column presents end if the number and proportion of source code lines without 6 end foreach if branches in error handlin code rom able , we can see that nearly of error iure rocedure of etractin taret functions handlin code is not in if branches in these driers It help us inect more representatie and efficient faults indicates that inectin a sinle fault in each test case is for I pecifically, we can inect faults in the data enouh to coer most error handlin code his charac checed by these if checs, to simulate more realistic teristic can help to simplify the compleity of inected errors in deice driers faults and improe the efficiency of I he remainin error handlin code is in if branches because dif attenae tatn ferent resourceusae states or deice states need to be he representatieness of inected faults is a ey factor separately handled in the same error handlin code of I his property larely determines the accu his characteristic commonly eists in fail-stop dri racy of bu detection and the efficiency of runtime test ers oweer, some driers lie are based on the in Inectin representatie faults can simulate realistic fail-recovery model, namely they will restart when an errors to trier real error handlin, so detected bus error occurs hus, many branches are needed to handle are ery probably real eanwhile, useless test cases the recoery procedure or these driers, inectin a are less enerated when the inected faults are repre sinle fault is not enouh to coer most error handlin sentatie, so the time usae can be larely reduced code In this paper, we mainly focus on fail-stop driers, common stratey is to inect random faults, which because they occupy a lare part of eistin driers has been used in many preious I approaches , , , ut some studies , , hae proed this e en stratey can not well represent real errors, and they also inu driers are often implemented in , so builtin introduces many false posities in bu detection e error handling mechanisms (such as “try-catch”) are not cause most error handlin code in driers is triered by supported To check whether an occasional error occur, bad function return alues in ection , it is feasible an if check is often used in the source code he if to inect faults in some manually selected taret func statement checs whether the ey data is erroneous and tions which can fail at runtime his stratey has been decides whether error handlin code should be eecuted used in some preious approaches , , to test his ey data can be a common ariable or a function driers, but it has three problems irstly, new taret return alue hus, the characteristic in ection can functions should be manually selected when testin a be rearded as an aspect of it or eample in able , new drier econdly, it is hard to assure the selected all “goto” statements triggered by bad function return taret functions can actually trier realistic errors at alues are in if checs articularly, most if checs for runtime hirdly, many real taret functions may be function return alues only chec whether the alue is a omitted in manual selection null pointer or nonero inteer line in iure is ased on the characteristics mentioned in ection an eample amely, different bad function return al and , we propose a pattern-based extraction strategy ues are often handled by the same error handlin code to automatically and accurately etract real taret func his if chec decision characteristic is also recom tions from the source code iure shows the main mended by the inu ernel documentation It can procedure of this stratey, which consists of two phases

USENIX Association 2016 USENIX Annual Technical Conference 639 OS source code a

Target Other Interface o eiciently test error handling code in deice driers + + Driver Drivers Functions e roose est based on drier characteristics code instrumentation and dynamic analysis igure EH-Test shos the oerall architecture o est hich con Fault Fault Probe Runtime Pair Checkers Extractor Injector Inserter Monitor sists o ie modules  Fault extractor. his module uses the attern Fault-injected Test Cases Bug Reports based etraction strategy to automatically etract Functions target unctions t needs the source code o the tar get drier other driers and ernel interace unc igure erall architecture o est tions as inut hich can be obtained rom the irstly e run the drier normally on the hardare and source code record the runtime traces during normal eecution ll  Fault injector. his module uses code instrumenta unctions hose return alues are ointers and integers tion to inect aults by corruting the return alues are selected rom the recorded runtime traces hese o target unctions single ault is inected in each unctions are regarded as candidate functions or ault test case inection econdly or each candidate unction e  Probe inserter. his module instruments robes in udge hether it can trigger a realistic error ccording the drier code to collect runtime inormation and to the code atterns o inu deice driers a candidate count code coerage during eecution t oututs unction can be regarded as a real target unction under test cases o the tested drier ach test case is a three atterns loadable drier hich can be directly installed in Pattern 1 he return alue o the candidate unction is the oerating system checed by an if statement in the drier ecause an if  Runtime monitor. his module runs test cases and chec is oten used to decide hether error handling records the runtime inormation o the tested drier code should be triggered (in ection ) n most cases during eecution t also detects bugs at runtime this if chec is oten closely behind the unction call  Pair Checkers. hey are used to chec resource Pattern 2 he return alue o the candidate unction is usages in driers ach air checer contains the checed in other driers n some cases the deeloer basic inormation o a air o resourceacuiring may orget to chec the return alue o a certain unc and resourcereleasing unctions e hae ritten tion in the driver. But this function’s bad return value some air checers in est based on the result can be deemed to trigger a realistic error hen it is o seciication mining techniues checed by an if statement in other driers ased on the architecture to hases are erormed Pattern 3 he return alues o some ernel interace hen est ors namely test case generation and unctions are clearly seciied to be checed in their runtime testing he manual or only includes riting declarations or annotations because they can trigger air checers chec ing etracted target unctions and errors or eamle in the inu ernel code a seciic rebooting the system hen crash bugs are detected macro “__must_check” is defined. If this macro is noted et ae eneatn in the declaration o a unction its return alue must be checed he unction pci_request_regions in igure n this hase e hae to tass namely etracting tar uses this macro esides some ey hrases in the unc get unctions rom the code and generating test cases o tion annotation also indicate the unction return alue the drier by inecting aults on target unctions he should be checed hereore the declaration and anno detailed stes are as ollos tation o candidate unctions should be checed as ell irstly e inut the drier code and source code his strategy has three adantages irstly hen the to the ault etractor t uses the atternbased etrac drier source code and hardare are aailable this tion strategy to etract target unctions ter etraction strategy can automatically etract target unctions ith the user also is alloed to chec and modiy target out manual eort econdly by using eact runtime unctions as needed inormation and common code atterns many unreal econdly e inect aults into target unctions ey target unctions are iltered out hirdly no real target uestion is that ho many aults should be inected in unctions in the catured runtime traces are omitted y each test case any reious aroaches using this strategy e can automatically and accurately inect multile aults in each test case because they aim etract real target unctions as reresentatie inected to coer as much error handling code as ossible ut aults to imroe the eectieness o ault scenario elosion may occur in this situation

640 2016 USENIX Annual Technical Conference USENIX Association hen driver crashes occur the oututs the du

Source Code Kernel Object File inforation into the ernel crash lo. herefore e can (driver.c) Driver Compilation (driver.ko) chec the ernel crash lo to detect and locate crash Clang with Clang and GCC Clang + GCC bus lie null ointer dereferences. or driver hans e can detect the b observin hether the sste LLVM Bytecode Instrumentation LLVM Bytecode* (driver.ll) Clang (driver.ll*) freees. hese to inds of bus are eas to observe in real eecution.

Runtime Fault Injector Probe Inserter Monitor Function Names Description Data EH-Test request_irq Enable / disable the interrupt Para1 free_irq line and IRQ handling Para1 iure oilation rocedure of the tested driver. pci_enable_device Initialize / disable the device on Para1 hich can larel reduce testin efficienc. o relieve pci_disable_device the PCI bus Para1 this roble and seed u testin these aroaches dma_pool_alloc Allocate / free a block of con- RetVal have to use soe eedients such as liitin the nu dma_pool_free sistent memory for DMA Para1 ber of inected faults or searchin aths and resortin to user uidance . or an inu drivers able elected aired functions in device drivers. a e characteristic is that there are fe if branches in s for resourcerelease oissions the are hardto error handlin code in ection .. ael the error find in real eecution because the rarel lead to obvi handlin code in an device drivers onl handles a ous ecetions. oever the often cause resource sinle error at a tie. hus to cover ost error han usae robles such as resource leas and eor dlin code ith less testin tie e onl corrut the leas. oreover resourcerelease oissions often occur return value of one taret function in each test case. he in device drivers eseciall in error handlin code . taret function call is relaced b an error function in or these reasons est should detect resource the code. hat this error function does is onl returnin release oissions in device drivers. resourcerelease a bad value. If the return value of the taret function is a oission occurs hen a resourceacuirin function is ointer the error function ill return a null ointer if successfull called but its resourcereleasin function is the return value of the taret function is an inteer the not called. or eale in iure kzalloc is a re error function ill return a rando neative nuber. sourceacuirin function and it is used to allocate er hirdl e instruent robes to collect runtie in nel eor but the resourcereleasin function kfree is foration and count code coverae durin eecution. not called hich leads to a resourcerelease oission. he runtie inforation is used to detect bus in the resourceacuirin function and its resourcereleasin net hase. he code coverae is used to uantif the function should be called in airs so the can be called effectiveness of runtie testin. inall driver test cas paired functions . Besides the should oerate the es are enerated. ach test case is a ernel obect file sae aed data araeter or return value as the nael a loadable driver. handled resource. In est e ileent soe air In the second and third stes code instruentation is checers to detect resourcerelease oissions. ach air used. e ileent it at coile tie usin the lan checer contains the basic inforation of a air of coiler. iure shos the coilation roce aired functions includin function naes and aed dure of the tested driver. irstl e use the lan co data. oe revious aroaches for secification in iler to coile the source code of the driver into the in can be used to etract aired func btecode. econdl e utilie the fault inector tions fro the code. In this aer e use the inin and robe inserter to instruent our handled code in the result of PF-Miner hich is a static aroach for btecode. hirdl e use the lan coiler to co inin aired functions in inu drivers to build the ile the btecode into the assebl code and then build air checers. able shos soe selected aired func the obect file usin . inall e lin the obect tions in the checers. he first colun shos function file and the runtime monitor’s program together, and naes the second colun shos the descrition the enerate a ernel obect file as a test case. third colun shos the aed data. urin driver eecution the runtie onitor uses the ntme etn inserted robes to record the runtie inforation of In this hase e run each test case on the real hardare function calls and aintains a resourceusae list. or and detect bus durin eecution. hree inds of bus each function call the onitor checs hether it is in are detected in current ileentation nael crashes the air checers. hen a resourceacuirin function hans and resourcerelease oissions. is called the onitor checs its return value to ude

USENIX Association 2016 USENIX Annual Technical Conference 641 hether the resoure is suessfull alloated f it is Driver Candidate Target Real true, the monitor ill reate a node ontaining the fun rtl tion name and mapped data, and add it into the re soureusage list hen a resourereleasing funtion is il alled, the monitor sans the list to math the node ith rt usstorage the funtion information f it is mathed, the node ill uhihd e deleted to indiate the resoure is released hen the ehihd drier is remoed, the monitor hes the nodes in the e list f the list is not empt, it indiates resourerelease ee omissions our, so the monitor ill report them ig r aatn too ementa Set s o alidate the effetieness of est, e ealuate it ipg on real deie driers he tested driers should satisf ta three riteria irstl, the should e ommonl used in ale esult of the patternased etration pratie eondl, the should e ithin the drier aet ntn tatn lasses in etion , eause the an satisf the hara teristis found the stud hirdl, the should run as he representatieness of ineted faults is a e fator ernel modules, eause the test ases of them an e of n this paper, ineted faults are ad return al diretl installed and remoed ithout reooting the ues of target funtions, and all target funtions are au operating sstem ording to these riteria, inu tomatiall etrated our patternased etration deie driers are seleted, inluding ireless, and strateg hus, the fault representatieness of large thernet driers ale shos the asi information of l depends on the effetieness of our patternased tested driers in inu etration strateg here are three important researh uestions aout its effetieness Class Driver Hardware Lines RQ1: How many unrepresentative candidate functions rtl ealte ireless ontroller are automatically filtered out? roadom ireless ontroller Wireless RQ2: How much is the false positive rate of the strategy? il ntel ireless ontroller RQ3: How many real target functions are omitted? rt alin ireless ontroller usstorage ingston dis o anser these uestions, e first ealuate est USB uhihd ntel ontroller on the driers to etrat andidate funtions and tar ehihd ntel ontroller get funtions hen e manuall he the etrated e ntel thernet ontroller target funtions to udge their realness ale shos ee ntel thernet ontroller the result in inu he first olumn presents the ig ntel thernet ontroller drier name the seond olumn shos the numer of r ealte thernet ontroller andidate funtions the third olumn shos the numer Ethernet too ealte thernet ontroller of etrated target funtions the fourth olumn shos om thernet ontroller the numer of real target funtions s arell thernet ontroller rom ale , e an find that target funtions ipg lus thernet ontroller are etrated from andidate funtions t indiates that andidate funtions are automatiall filtered ale ested driers in inu out eause the are unrepresentatie, hih an an he eperiment runs on a enoo ith to ntel ser manuall heing the douments and i proessors and phsial memor implementations of the etrated target funtions, e and lang are used for ompilation e find that target funtions are real, hih means the rite pair heers ased on the result of PF-Miner an atuall fail and trigger error handling ode t indi or eah test ase of the driers, e install it in the ss ates that the false positie rate of our patternased tem, run it on the orload, and finall remoe it he etration strateg is onl , hih an anser orload onsists of three inds or ireless driers, an false target funtions return integers hih are e turn on ii, ping another omputer and turn off also heed if statements, ut the reflet different ii for thernet driers, e ping another omputer drier onfigurations or states ut neer trigger oa or driers, e op a file to the dis sional errors nsering is diffiult, eause target

642 2016 USENIX Annual Technical Conference USENIX Association Linux 3.1.1 Linux 3.17.2 Driver Test case Time usage Crash / Hang Resource Bugs Test case Time usage Crash / Hang Resource Bugs rtl 16 03:14 0 / 0 1 (0) 1 18 04:21 0 / 0 3 (2) 3 62 23:57 0 / 0 1 (1) 1 55 26:34 0 / 0 1 (1) 1 il 100 36:42 5 / 0 7 (7) 12 79 25:18 5 / 0 8 (8) 13 rt 62 19:21 1 / 0 1 (0) 2 65 21:37 0 / 0 1 (0) 1 usstorage 25 03:35 0 / 0 0 (0) 0 20 03:07 0 / 0 0 (0) 0 uhihd 22 03:47 0 / 0 0 (0) 0 24 03:20 0 / 0 0 (0) 0 ehihd 24 03:50 0 / 0 1 (0) 1 23 03:58 0 / 0 10 (9) 10 e 33 03:02 1 / 0 0 (0) 1 33 02:28 1 / 0 1 (1) 2 ee 66 11:01 0 / 0 0 (0) 0 62 10:30 3 / 0 3 (0) 6 ig 62 10:09 0 / 0 0 (0) 0 59 12:56 0 / 1 6 (6) 7 r 15 01:24 0 / 0 0 (0) 0 15 01:43 0 / 0 0 (0) 0 too 9 00:45 0 / 0 1 (0) 1 9 00:46 0 / 0 1 (0) 1 18 01:26 0 / 0 2 (2) 2 15 01:24 0 / 0 2 (2) 2 s 26 01:43 3 / 0 8 (0) 11 30 02:14 4 / 0 0 (0) 4 ipg 17 01:16 0 / 0 0 (0) 0 16 01:28 0 / 0 0 (0) 0 Total 557 125:12 10 / 0 22 (10) 32 523 121:42 13 / 1 36 (29) 50 ale ugdetetion result of est funtions are etrated from normal eeution traes, eleenth olumns “Bugs” sho the numer of deteted ut different eeution paths ma hae different runtime ugs peifiall, the numer of memor leas is traes hus, the real target funtions ithin the unee shon in the parenthesis of the fifth and tenth olumns uted paths ill e omitted oeer, e find that all “Resource”, eause the memor lea is an important target funtions in the aptured runtime traes are e ind of resourerelease omission trated our strateg rom ale , e mae the folloing oserations eieing the result, e also find an interesting irstl, est finds ugs in the driers in phenomenon ost target funtions are in the initialia inu , inluding rashes and resoure tion proedure he data in the parenthesis of the fourth release omissions mong these resourerelease omis olumn sho the numers of these funtions he o sions, are memor leas eieing the drier ode, up of all target funtions amel, most inds of resourerelease omissions sky2 drier and rash oasional errors in driers our in the initialiation rt2800 drier hae een fied in inu t indi proedure n fat, it has een noted in the inu drier ates that est an find the non ugs manual , and our results an suessfull erif it eondl, est finds ugs in the driers in he eplanation for this phenomenon is that different inu , inluding rashes, hang and re inds of onfigurations need to e made in the initiali sourerelease omissions mong the resourerelease ation, and eah onfiguration an ause a ind of oa omissions, are memor leas oreoer, ugs are sional error fter the drier is initialied, onl seeral resered from the lega ode in , and ugs are inds of errors an our in the running proedure introdued due to ne implementations e send all the etetn ugs to the drier deelopers, and all of them hae een onfirmed e also send pathes to fi them, and ith the etrated target funtions, e perform runtime hae een applied the maintainers t indiates testing to detet ugs in error handling ode ah test that est an auratel find ne ugs in driers ase is generated maing one target funtion fail o tuall, a threat to alidit is that false etrated alidate hether est an find the non ugs ha target funtions ma introdue false ugs n our ealua ing een fied, e first use est to test the dri tion, no false ugs are deteted for this reason ers in an older inu ersion released in oem hirdl, the time usage of est is short out er hen e test these driers in a neer inu hours are spent in totall testing the driers, and onl ersion released in toer to alidate seeral minutes are spent for most driers his time hether est an find ne ugs ale shos the usage is shorter than man preious approahes , result he first olumn shos the drier name the se , for testing driers ne reason is that est ond and seenth olumns “Test case” sho the num uses the patternased etration strateg to filter out er of generated test ases the third and eighth olumns man unrepresentatie andidate funtions, so no re “Time usage” present the time usage of the runtime dundant test ases are generated hus, the test ases are testing the fourth and ninth olumns “Crash / Hang” effiient, and the testing time is largel shortened sho the numer of deteted rashes and hangs the

fifth and tenth olumns “Resource” present the num he pathes an e found in the lin er of deteted resourerelease omissions the sith and http://oslab.cs.tsinghua.edu.cn/EHTest/patch.html

USENIX Association 2016 USENIX Annual Technical Conference 643

e eae e100 pci_pool_create nic->cbs_pool pci_pool_create pci_pool_alloc pci_pool_create if man ement pci_pool_create Path: linux-3.17.2/drivers/net/ethernet/intel/e100.c Stae at netn s c_cbs(…) …… nic->cbs = pci_pool_alloc(nic->cbs_pool, …); s …… 2843. static int e100_probe(….) …… 2967. nic->cbs_pool = pci_pool_create(…); tif_info(…); …… Figure 5: A detected bug in the e100 driver4.

http://marc.info/?l=linux-netdev&m=143993218231729&w=2

644 2016 USENIX Annual Technical Conference USENIX Association e100 r8169 ehci_hcd Sm etn Sm etn e100 pci_pool_create sky2 iwl4965 eate Stae at netn Stat na

USENIX Association 2016 USENIX Annual Technical Conference 645 [7] Proceedings of the 2015 International Symposium on Software Testing and Analysis nn [8] Proceedings of the 13th International Con- ference on Quality Software [9] Linux Device Drivers, 3rd ed. O’Re [10] IEEE Transactions on Software Engineering [11] Proceed- nement ings of the 2004 International Symposium on Software Testing and Analysis [12] Proceedings of the 19th Pacific Rim Symposium on Dependable Computing [13] Pro- eeene ceedings of the 37th International Conference on De- [1] S. Amani, L. Ryzhyk, A. F. Donaldson, G. Heiser, A. pendable Systems and Networks Legg and Y. Zhu. Static analysis of device drivers: we [14] can do better. In Proceedings of the 2nd Asia-Pacific Pro- Workshop on Systems, 2011. ceedings of the 2011 International Conference on Object [2] Oriented Programming Systems Languages and Applica- tions [15] Proceedings of the 1st European Conference on Com- puter Systems Proceedings of the 2014 International Symposium on [3] Software Reliability Engineering Workshops Proceedings of the 7th European [16] conference on Computer Systems Pro- [4] ceedings of the 2010 USENIX Annual Technical Confer- Proceedings of the 2005 USENIX Annual Technical ence Conference [17] [5] Proceedings of the 2014 International Proceedings of the 16th International Confer- Symposium on Software Testing and Analysis ence on Architectural Support for Programming Lan- guages and Operating Systems [18] [6] Proceedings of the 10th Interna- Pro- tional Conference on Tools and Algorithms for the Con- ceedings of the 39th International Conference on De- struction and Analysis of Systems pendable Systems and Networks

646 2016 USENIX Annual Technical Conference USENIX Association [19] . d . . Rer er ence on Languages, Compilers and Tools for Embedded rr re d dee Systems, e , . re re de. Proceedings of the 13th Interna- [31] . , . . , . , . . d . tional Symposium on Foundations of Software Engineer- er. er dee rerereee ing, . 33, . errrd de r e re. Proceedings [20] . , . , . d . . er e of the 43rd International Conference on Dependable red ed r drd ere er Systems and Networks, e , 3. rr . Proceedings of the 38th International Com- [32] . . er, . . er d . . rr. puter Software and Applications Conference, e 33 ee drer R. Interna- , . tional Journal of Advanced Research in Computer Engi- [21] . . re d . de. r d neering and Technology, e , . eer rree er. Proceedings of the [33] . . , . , . . erd d . . 39th International Conference on Dependable Systems e. Reer dee drer. ACM Transactions and Networks, e 33, . on Computer Systems, e , e 3333, . [22] . . re d . de. e e re [34] . . , . . erd d . . e. r er de e. ACM Transactions e re d er e. Pro- on Computer Systems, e , e , . ceedings of the 19th ACM Symposium on Operating Sys- [23] . ed d . ee. Re e e tems Principles, e , 3. d . Proceedings of the 37th International [35] . er, . reer, . er d . r. Conference on Dependable Systems and Networks, e r e e re e. , . Proceedings of the 43rd International Conference on [24] R. e, . re, . . re d . . der. Dependable Systems and Networks, e , 3. O rereeee re e. [36] . , . , . re d . ee IEEE Transactions on Software Engineering, e 3, er. de e rre dee drer. e , e , 3. Proceedings of the 22nd International Conference on [25] R. e, . re, . re d . der. Re Automated Software Engineering, e , . reeee eed re [37] . , . , . , . e d . e. ere e re. Proceedings of the 40th Interna- rereree e. Proceed- tional Conference on Dependable Systems and Networks, ings of the 26th International Conference on Automated e 3, . Software Engineering, e 33, . [26] . Odre, . d . r. [38] . , . , . rd, . d . . d rer r rr. Mathematical and err er re r ere Engineering Methods in Computer Science, e , re. Proceedings of the 28th International Confer- e , . ence on Software Engineering, e , . [27] . d . . ered r [39] . d . . e de e dee drer er. Proceedings of the e d de. Proceedings of the 34th Inter- 6th International Conference on Integrated Formal national Conference on Software Engineering, e Methods, e 3, . , . [28] . . Ree, . d d . . . re [40] er. e drer dee. Proceedings of the http://clang.llvm.org/ 10th International Conference on Operating Systems De- sign and Implementation, e , . [41] er. http://clang-analyzer.llvm.org/ [29] . . R d . . . Re er ere de ed ere. [42] e r de . Proceedings of the 4th International Conference on http://cppcheck.sourceforge.net/ Software Testing, Verification and Validation, e [43] e e rrre. , . http://www.kernel.org/doc/Documentation/fault- [30] . , . d . er. r injection/ fault-injection.txt r e rre errrd de e [44] ere d e. ere. Proceedings of the 2011 International Confer- http://www.kernel.org/doc/Documentation/CodingStyle

3

USENIX Association 2016 USENIX Annual Technical Conference 647