Leveraging Light-Weight Analyses to Aid Software Maintenance
Total Page：16
File Type：pdf, Size：1020Kb
Leveraging Light-Weight Analyses to Aid Software Maintenance A Dissertation Presented to the Faculty of the School of Engineering and Applied Science University of Virginia In Partial Fulfillment of the requirements for the Degree Doctor of Philosophy (Computer Science) by Zachary P. Fry May 2014 c 2014 Zachary P. Fry Abstract While software systems have become a fundamental part of modern life, they require maintenance to continually function properly and to adapt to potential environment changes [1]. Software maintenance, a dominant cost in the software lifecycle [2], includes both adding new functionality and fixing existing problems, or “bugs,” in a system. Software bugs cost the world’s economy billions of dollars annually in terms of system down-time and the effort required to fix them [3]. This dissertation focuses specifically on corrective software maintenance — that is, the process of finding and fixing bugs. Traditionally, managing bugs has been a largely manual process [4]. This historically involved developers treating each defect as a unique maintenance concern, which results in a slow process and thus a high aggregate cost for finding and fixing bugs. Previous work has shown that bugs are often reported more rapidly than companies can address them, in practice [5]. Recently, automated techniques have helped to ease the human burden associated with maintenance activities. However, such techniques often suffer from a few key drawbacks. This thesis argues that automated maintenance tools often target narrowly scoped problems rather than more general ones. Such tools favor maximizing local, narrow success over wider applicability and potentially greater cost benefit. Additionally, this dissertation provides evidence that maintenance tools are traditionally evaluated in terms of functional correctness, while more practical concerns like ease-of-use and perceived relevance of results are often overlooked. When calculating cost savings, some techniques fail to account for the introduction of new workflow tasks while claiming to reduce the overall human burden. The work in this dissertation aims to avoid these weaknesses by providing fully automated, widely-applicable techniques that both reduce the cost of software maintenance and meet relevant human-centric quality and usability standards. This dissertation presents software maintenance techniques that reduce the cost of both finding and fixing bugs, with an emphasis on comprehensive, human-centric evaluation. The work in this thesis uses lightweight analyses to leverage latent information inherent in existing software artifacts. As a result, the associated techniques are both scalable and widely applicable to existing systems. The first of these techniques clusters closely-related, automatically generated defect reports to aid in the process of bug triage and repair. This clustering approach is complimented by an automatic program repair technique that generates and validates candidate defect patches by making sweeping optimizations to a i Chapter 0 Abstract ii state-of-the-art automatic bug fixing framework. To fully evaluate these techniques, experiments are performed that show net cost savings for both the clustering and program repair approaches while also suggesting that actual human developers both agree with the resulting defect report clusters and also are able to understand and use automatically generated patches. The techniques described in this dissertation are designed to address the three historically-lacking properties noted above: generality, usability, and human-centric efficacy. Notably, both presented approaches apply to many types of defects and systems, suggesting they are generally applicable as part of the maintenance process. With the goal of comprehensive evaluation in mind, this thesis provides evidence that humans both agree with the results of the techniques and could feasibly use them in practice. These and other results show that the techniques are usable, in terms of both minimizing additional human effort via full automation and also providing understandable maintenance solutions that promote continued system quality. By evaluating the associated techniques on programs spanning different languages and domains that contain thousands of bug reports and millions of lines of code, the results presented in this dissertation show potential concrete cost savings with respect to finding and fixing bugs. This work suggests the feasibility of further automation in software maintenance and thus increased reduction of the associated human burdens. Approval Sheet This dissertation is submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy (Computer Science) Zachary P. Fry This dissertation has been read and approved by the Examining Committee: Westley R. Weimer, Advisor Worthy N. Martin, Committee Chair Jack W. Davidson Stephanie Forrest Greg Gerling Accepted for the School of Engineering and Applied Science: James H. Aylor, Dean, School of Engineering and Applied Science May 2014 iii We are stuck with technology when what we really want is just stuff that works. — Douglas Adams iv Acknowledgments “Could it be semantics generating the mess we’re in?” – Michael Burkett I must first thank my advisor, Westley Weimer, for the heaps of wisdom he has imparted to me throughout my tenure as his student. Wes has far exceeded my expectations as an advisor, consistently going above and beyond the usual duties to make me a well-rounded, effective researcher. Though his advising style is often not for the weak-hearted, I am endlessly thankful for the wisdom and patience Wes has provided me throughout the past six years. I have been fortunate enough to have many sources of inspiration and research advice over the years. I’d like to thank Stephanie Forrest for her considerable help throughout my graduate research career. She has consistently provided an excellent perspective and ample help with many collaborative research projects and for that I am very thankful. Additionally, I’d like to express an immense amount of gratitude to both of my undergraduate advisors, Lori Pollock and K. Vijay-Shanker — without your help, I would not be where I am today. Lori and Vijay opened me up to research early on and helped get me started off on the right foot; for that I am very thankful. Additionally, I would like to thank David Shepherd and Emily Hill for acting as excellent research mentors throughout my undergraduate research career. One could not have asked for a smarter and more enjoyable research group than the “WRG.” Kinga, Pieter, Ray, Claire, Adam, and Jon have provided both insightful contributions to my work and also ample distraction when necessary. I thank them all for the entertaining and informative adventure. Thank you finally to my family and friends — it has been a wonderful, but sometimes trying, ride through graduate school. You have all been a constant source of support and much appreciated diversion, when appropriate. Special thanks to Jeremy Lacomis for ample help proofreading this document. v Contents Abstract i Acknowledgments v Contents vi List of Tables.................................................. vii List of Figures.................................................. viii List of Terms ix 1 Introduction 1 1.1 Why state of the art software maintenance tools are inadequate.....................2 1.2 A approach to improving aspects of software maintenance.......................3 1.3 Scientific intuition — using latent information in software artifacts...................6 1.4 Metrics and criteria for success......................................7 1.4.1 Generality.............................................7 1.4.2 Comprehensive Evaluation....................................8 1.4.3 Usability............................................. 10 1.5 Broader Impact.............................................. 10 1.6 Contributions and outline......................................... 11 2 Background and Related Work 13 2.1 Software bugs are prevalent, impactful, and expensive.......................... 14 2.2 Common strategies for avoiding bugs.................................. 16 2.2.1 Avoiding bugs during design and implementation........................ 16 2.2.2 Avoiding bugs before deployment................................ 17 2.3 Bug reporting as a means to describe software defects.......................... 20 2.3.1 Manual bug reporting....................................... 21 2.3.2 Automatic bug reporting..................................... 21 2.4 Fixing bugs, both manually and automatically.............................. 22 2.4.1 Manual bug fixing........................................ 22 2.4.2 Automatic bug fixing....................................... 23 2.5 Ensuring continued system quality throughout the maintenance process................ 25 2.5.1 Software maintainability and understanding........................... 25 2.5.2 Documentation.......................................... 26 2.6 Summary................................................. 27 3 Clustering Static Analysis Defect Reports to Reduce Triage and Bug Fixing Costs 28 3.1 Introduction................................................ 28 3.2 Motivation................................................. 30 3.3 Methodology............................................... 32 3.3.1 Modeling Static Analysis Defect Reports............................ 32 3.3.2 Defect Report Similarity Metrics................................. 33 vi Contents vii 3.3.3 Modeling Report Similarity..................................