Workshop on the Evaluation of Software Defect Detection Tools
Total Page:16
File Type:pdf, Size:1020Kb
Workshop on the Evaluation of Software Defect Detection Tools Sunday, June 12th, 2005 Co-located with PLDI 2005 Workshop co-chairs: William Pugh (University of Maryland) and Jim Larus (Microsoft Research) Program chair: Dawson Engler (Stanford University) Program Committee: Andy Chou, Manuvir Das, Michael Ernst, Cormac Flanagan, Dan Grossman, Jonathan Pincus, Andreas Zeller Time What Time What 8:30 am Discussion on Soundness 2:30 pm break 2:45 pm Research presentations The Soundness of Bugs is What Matters, Patrice Godefroid, Bell Laboratories, Lucent Technologies Experience from Developing the Dialyzer: A Static Soundness and its Role in Bug Detection Systems, Analysis Tool Detecting Defects in Erlang Yichen Xie, Mayur Naik, Brian Hackett, Alex Applications, Kostis Sagonas, Uppsala University Aiken, Stanford University Soundness by Static Analysis and False-alarm Removal by Statistical Analysis: Our Airac 9:15 am break Experience, Yungbum Jung, Jaehwang Kim, Jaeho 9:30 am Research presentations Sin, Kwangkeun Yi, Seoul National University Locating Matching Method Calls by Mining 3:45 pm break Revision History Data, Benjamin Livshits, 4:00 pm Discussion of Benchmarking Thomas Zimmermann, Stanford University Evaluating a Lightweight Defect Localization Dynamic Buffer Overflow Detection, Michael Tool, Valentin Dallmeier, Christian Lindig, Zhivich, Tim Leek, Richard Lippmann, MIT Andreas Zeller, Saarland University Lincoln Laboratory Using a Diagnostic Corpus of C Programs to 10:30 am break Evaluate Buffer Overflow Detection by Static 10:45 am Invited talk on Deployment and Adoption, Manuvir Das, Analysis Tools, Kendra Kratkiewicz, Richard Microsoft Lippmann, MIT Lincoln Laboratory 11:15 am Discussion of Deployment and Adoption BugBench: A Benchmark for Evaluating Bug Detection Tools, Shan Lu, Zhenmin Li, Feng Qin, The Open Source Proving Grounds, Ben Liblit, Lin Tan, Pin Zhou, Yuanyuan Zhou, UIUC University of Wisconsin-Madison Benchmarking Bug Detection Tools. Roger Issues in deploying SW defect detection tools, Thornton, Fortify Software David Cok, Eastman Kodak R&D A Call for a Public Bug and Tool Registry, Jeffrey False Positives Over Time: A Problem in Foster, Univ. of Maryland Deploying Static Analysis Tools, Andy Chou, Bug Specimens are Important, Jaime Spacco, Coverity David Hovemeyer, William Pugh, University Maryland 12 noon lunch NIST Software Assurance Metrics and Tool Evaluation (SAMATE) Project, Michael Kass, 1:00 pm Research presentations NIST Model Checking x86 Executables with CodeSurfer/x86 and WPDS++, Gogul 5:00 pm Discussion of New Ideas Balakrishnan, Thomas Reps, Nick Kidd, Akash Deploying Architectural Support for Software Lal, Junghee Lim, David Melski, Radu Gruian, Defect Detection in Future Processors Suan Yong, Chi-Hua Chen, Tim Teitelbaum, , Yuanyuan Univ. of Wisconsin Zhou, Josep Torrellas, UIUC Empowering Software Debugging Through Using Historical Information to Improve Bug Finding Techniques Architectural Support for Program Rollback, Radu , Chadd Williams, Jeffrey Teodorescu, Josep Torrellas, UIUC Computer Hollingsworth, Univ. of Maryland Locating defects is uncertain Science , Andreas Zeller, EXPLODE: A Lightweight, General Approach to Saarland University Is a Bug Avoidable and How Could It Be Found? Finding Serious Errors in Storage Systems, , Junfeng Yang, Paul Twohey, Ben Pfaff, Can Sar, Dan Grossman, Univ. of Washington Dawson Engler, Stanford University 5:45 pm wrap up and discussion of future workshops 6:00 pm done ¢¡¤£¦¥¨§ © ¤ ¤£§©¤ ¡¤ !£#"$ %'& §()$§ *¥+(£#,-£. 0/ 1243656798;:=<>!?(:A@B56>C79? D :;E9EF#24GH>C56243I>J567K:'L;MFN8;:;OP3RQ.:'8TS(O(>CE9>CUC79:;L VXW$YAZIY[K\^]`_aYcbdW![d_!ef\AbgZI]`_!e!YAZbdWhJ_ihJjdjXbBW$YkhJZmln[oYT\[d_pbBW$Ycq ]`Zmjsr`t hJ_0rubdWhJb[K\khJ_v[wr!Y6hxqW$]y\yYbw[dlxY¤Wh`\nzT]`lxY;{}|~[ozAb]`Z }Ce!] d o0gAg m oIT'$moA`H o`T H ;B Hg'A)mA`m m¦`w g)A g¡A ¡¡ØAoIóÙ¡+T ooAgA`4¬¡D¡¡¤ð9 ñ g$m` Ag4 T g¡A ogm) )¢g£+$ A¤¥`w6m¦JAy $ o ¡4m)ÃoA¦¡£ ð¥K¬ '¬`m¤© g¤g6î'`oa`g£'gg¡w oA` ñ § o+m g¨A;mCm¤$gm¤© BªXmo.y£¤¥ « y o « o'¬ 𥦠g AnÑ£Å;ÉAÝ©ÅJÑ)ѣЫݩÅ;ɨÝÊ)Ê)È`Ñëðsg£Ø¦4Tª ¬ ñ ñ ¤© g£0g6m¡ . g4T IH%JK¢`Ñ6LPÏTÈyÅCÆTÅJÑÊ)Ê è¨µ¥¸6°;ùI»´y¸6¶£E4°Ù·'ÀGF +AÂ)A g ®¯±°²}³J´µ¨¶9·f¸³º¹¶s»¼½ ¾¿4·'ÀºÁ ÏBÔMÈAÉTÊ+ÝÊONP¢ÚmÒ¡QcÚTÒKÒdÑÐ)ÊSR±m gg4T g¡A6Ø££oA'¬PTg £m``w£¤¥¡o0A¤wA¤© BªaAgy£¤¥ Hy o g¡AÃ$ o.A¦¡ mo oga £A` £g£ gaóm¡my)Iª¨¤¥oA Tª¢ªH'gßCï g ÄÅCÆfÇÈAÉmÊ«Xm + oko£JAg + g4T fËw`k¦`'`I'¤¦4£ à X¢gm¤© BªXmoyy£ )` )ßA¤¥'am`A`AA ÀV¨°;ùI¶ ³JûK¸ U ¤¥A``̬iÍ#y¤¥Ag g4T g g`cT w o£}¡}gA oR 'y ¡ßAXW¢m`YW6Z[Zª )f`g¢oyK g¡ Rg )6y¡£¤¥A ¤¥`w6k¤¥AuÎoÏmЫÐgÑ)ÎÒKÅJÑÊ)ÊÓJÐgÏ6ÏBÔ)¬ dn)A g ' g`£oá w gm g £A yg¡Ù`ë g`.£óÙ6 y o'`Hm¤± wA¤© BªaAg# w` ) g¡R'.kÎ)ÏTÕ#ÓJÖÑ£Ò×ÑØ £Aog6 « oooÙm¤BPw¡` ££'£ ¡££; om g¡Au goA`Ax§ )ÃB )T o¤g`' nð¥nA¦`B oo' « o' Ê£ÏTÈyÅCÆÚTÅCÚTÖÜÛIÊ«ÝÊ¡££; o=Þ4Iª¡ow ^R¦mK¤¥o£ ¡=`g ¬]\Ù¡` u g^w6m) ) ¡c £m¡ ¡` £A¡ g' gÙAJ¡X¦JA``+ o )6 )ßA ¢gJ 4 w£ aA¤!oAJ£g g¡¬ ñ ^$£o_\ym¤© #oA o4 6£y¦ g'`gA`oXm¤ 'go g6g à y;B y¤¥6 « y£ g ³±á!âã¯Cäæå´'äCçoèR»±´µsä·¶9·Iékê o;A¤s¨ðËow'`y6goÌ ¤¥AgA``y6gXA¤±¦;£¬ g¡Ak gÙAëAgÃgogg' ¦`AgkAn`g'A)mãAg4 T ñ mom¦`w g)A g¡A`'mou om À å¾·£¸x´ba(·£¸ùI´`úm¸¶9³4»· ` g¡Afy'A`n+mß'Ã`g¨m¤¢Î)ÏmÅÊ£ÑЫìTÚTÒKÝ©ìTÑÚAÇÊ«Ò9ÐgÚ'ÎÒKÝsÏTÅ`Ê«¬ ªaAo`ë`4y`kgoA)$ª¡ )xy`A }+ m¦`w g)A g¡A` í yg'Xw` )u gÙ'¡+y£ g6 « æ; g4T Õ+ÚTÛ}`mJ£!¬ A6moæ gTªXm)oT;¡f £Aog6 « g`o£¬cD§ o '¦¡ ¢`g ¢ g}` ¤¥A g`¡¨î;6B o'`m¦¡¢g'=y6 w¡A Á + `w RA¦`B oo' « o'`£ Ç)ÏTÒ£¢ Î)ÏTЫÐoÑoΣÒKÅJÑÊoÊÓCÐgÏ6ÏBÔoÊkÚmÅJÆ ¡ØAoA`ï.g` )= gÙA¨mo¤Æ'ÏÏmÕÑ)Æ¢ o}o£JAg +ð¥+m Ad ñ ÇÈAÉmÊ$ÔÏTÈyÅJÆëmo.'`m)m; gà g+¦Jëw'`P¬ ¤smw}mmo(9¬ A¬¡ÈyÅ`ÊÏmÈyÅJÆ^Ç£ÈAÉTÊ«¬nò#6w`§ oRoA'g6gA ðsò.o g gA+m ggjA`yA w g¡ Àgf+èihØ® F Á e ¡§ oäsmwëA¡Ag+ëðsA¬ `¬¡`¦ `g¡ÃAo¨o gë ñ æ`£ª Ao'A )m` gÙA+ðsg£lknm±ò#o¥p8qSqSr ¤¥AfÚmÈyÒ×Ïmü ¦4' óy6 y o'ÃAammoº Aog§4 T o' g )`¡î;6 ;A ñ ñ ÕÚmÒ9ÝsÎ)ÚmÖ©ÖÜÛ ow g¡ wA¤© BªaAgRð¥K¬ '¬`¨ ow ¨g¡A.y6 ÕÚmÛoAA)mvm4m g!¦JA4ë o#A£)T o$¤smgaA¡Ag+ ñ g`m A¢¦¡¨ð2s ÚTÈyÒdÏTÕ+ÚTÒdÑ)Æ; gw¤s' ØóÙ oo' « g¡A¨¤¥g' m`R£4 Ø goî;¡g+ð9w¡A4 m; Ù+m}ôCAg ¬ ñ ñ w') v yyAuð p ÐoÚTÅJÆ'ÏTÕ ow g¡T  o`T ¡' o£g¤sA £A ñ ¡+ £óÙ`¡A ®¯±°öõ÷nøH³`ùI¸´`»±ú'°æ³Jû¢®a°Ù·£¸6¶s»¿!ÀkÁ m`pðt `m ¨ ow ÃA£)T o'n g ÆmÝ©ÐgÑ)ÎÒ£óÙ6 y o'` ñ ª ÒdÑ«Ê«ÒKÝ©Å;É4AH¦4£¨¨¢ gK¦¡¡¡A¨A¡¡A4y`w g ¤¥' m¡A.m g£o`m g¡AoA'oA `m g`¬ § oA`Aò j `g F F Á A AoIª¡Xmy oA+T o £yy×4wJ g¡A£o£ o £¡guA¦`w g)A « o'p g )`¡î;6£m¡ ¦'¤¥A4¦ ££oA¡APX¢§ o§×¡¡'ëyA¡¡AP¦`g`o£¬ y g£; Á ò j moØA`AoA' o£6+ g¦Jëw'`Py¦ yg'¬ F Á £)m0'o£)ëm¤+AA g`¨AoÃôCAg }ðs4A¡A0 ' à d A¡¡A+Ëow'`Ù h¨³Jµs°v³4ûvu ³4¾»¼±»°Ù··-w(À g¡ ¡(g4' (A g6B o. g4m'¢ £ÙH¡`g46 « g¡A¤ð¥+Ay ñ ooÌ+ð© g``cÚmÖ©ÖmJm g; gmJ¦'0m¤¨`Aw o¡ £m$ J ñ `m;m`ëAy g'+T g6 ¬ 0o'w'Ø¡w¡¡AïÒdÑ«Ê«ÒKÝ©Å;É$Ä0ÅJÆTÊ Á ñ JAg oA' m .¤¥Aóy`A`B o'£6g#g6AgA`¦y ØA ¤¥A ÊÏmÈyÅJÆÃÇ£ÈAÉmÊHª¡+Õ+ÚTÛTüwÚmÅJÚmÖÜÛIÊ«ÝÊ¢ÆAÏ6Ñ«ÊëÅCÏTÒK¬ %xà ¡£4 o'w'`£ï oÃm¦¡¡§ o oTA¢ gÃA¦`w` A¤ ÔØÆAÑ9ÔÑ)ÎÒaÆAÑ£Ò×Ñ)ΣÒ9ÝsÏmÅ ÝÊ.Ò£¢ÑÉ;ÏÚmÖ¥¤§¦¨¢ÙÛÃÚTÐoÑ èþý#´yùT´¼³Ùÿ$À¡ ¦;H mæJ.4w6 g B oA¨¢y`mCgm) )¤¥AXgA`æ'¬ ÊÏÕ+ÚTÅ`ÛÆAÑ9ÔÑ)ÎÒÆAÑ£ÒdÑoΣÒKÝsÏTŤÒ×Ï6ÏTÖÜÊÇ)ÚIÊ£Ñ)Æ¢ÏmÅ}Õ+ÚTÛTüwÚmÅJÚmÖÜÛIÊ«ÝÊ © y zQcÚTÛRm`k oIQ ÈÙÊ«ÒHAg o ³J»úAµs¾·6¶s³J»0À^Á ê à × ¡¢JAg oA' g=¡w ³J¾!ùTúA°Ù·k³4ûõ÷fø±ùT°ÙúA¶9·¶9³4»HÀ m` o AÃm¤`g'A)miA`m w£¬ 6Ù`Aw 0ogm) ) ê ê g¡Ag Ò ¦0ÏÆTÝÊ«ÒKÝ©ÅJΣÒJgA`o £$A¤C¡o g'++oA'oA oAA)mpm`A g¡±m`ëA£o4 £m g¡A¨`' o g6¨ g`0¦`A¡A` m`A g¡ïðs w¡` £¨ªHëªXm; oÃm`A ÃÏ)ÓѣŨoA'oA+£ ñ gTªXm)P o?QÚmÛHA`¨` ao£A; gªy£(Ay'y g¡A ªHÃ`£^oAw g cÑÅ4ìÝ©ÐoÏTÅ`Õ+Ñ£Å`ÒëÚIÊ)Ê«ÈyÕÓCÒ9ÝsÏmÅÊ#ðs¦ 4w¡ ñ m¤oA'oAm4m g gÙ'¡¬w`AA6B # gÃo£`A# g. ¦`mm4 ¨¦ ow g'g¡ oR`g'4 oA¡¨A¤{Q ÈÙÊ«Ò9$K¬ '¬0 g ¥ ! "$#%"'&$( #% )+*-,¥/.0"12% 234¥-562%¨!7!8") 2):9 y ëÒ£¢ÑëÊ£ÏTÈyÅJÆmÅJÑÊ)ʨÏwÔÇÈAÉmÊÄÐ)Ê«ÒK¬ -,¥$4) /34;§ <!¥):!"4 >=1 <? <!§@34¥?"4.A)34, 1 :B!)#%2)2C Soundness and its Role in Bug Detection Systems Yichen Xie Mayur Naik Brian Hackett Alex Aiken Computer Science Department Stanford University Stanford, CA 94305 The term soundness originated in mathematical logic: a The most widely used languages of the 1980's (C and C++) deductive system is sound with respect to a semantics if it deliberately had unsound type systems because of perceived only proves valid arguments. This concept naturally extends usability and performance problems with completely sound to the context of optimizing compilers, where static analysis type systems.