Program Comprehension in Feature-Oriented Software Development
Total Page:16
File Type:pdf, Size:1020Kb
Framework for Measuring Program Comprehension Dissertation zur Erlangung des akademischen Grades Doktoringenieurin (Dr.-Ing.) angenommen durch die Fakultat¨ fur¨ Informatik der Otto-von-Guericke-Universitat¨ Magdeburg von Dipl.-Inform. Dipl.-Psych. Janet Siegmund geb. am 15. Februar 1982 in Blankenburg Gutachter: Prof. Dr. Gunter Saake Prof. Arie van Deursen, Ph.D. Prof. Don Batory, Ph.D. Promotionskolloquium: Magdeburg, 27. November 2012 Siegmund, Janet Framework for Measuring Program Comprehension Dissertation, Otto-von-Guericke-Universitat¨ Magdeburg, 2012. i Abstract Program comprehension is a major human factor in software development. It often makes the difference between success and failure of a software product, because main- tenance programmers spend most of their time with understanding code, and because maintenance is the main cost factor in software development. Thus, if program com- prehension is not supported properly, time and cost for software development increase significantly. Although program comprehension has such importance, researchers and practition- ers do not consider it properly, but use plausibility arguments to claim positive or neg- ative effects on program comprehension. Program comprehension is an internal cogni- tive process, so we cannot rely on plausibility arguments only, but need to conduct con- trolled experiments to empirically evaluate what improves and what impairs program comprehension. We believe that one reason for this mismatch between the importance of program comprehension and its insufficient evaluation is the effort of conducting con- trolled experiments. Thus, a plethora of new programming paradigms and techniques exist that suppos- edly improve program comprehension, but have not been evaluated empirically to con- firm a positive effect on program comprehension. Although there is effort to improve program comprehension, it is not clear whether this effort has success. To improve the current situation, we define two goals in this thesis: First, reducing the effort for con- ducting controlled experiments, and, second, creating first empirical evidence how one of the plethora of new approaches, feature-oriented software development, affects pro- gram comprehension. To address our first goal, we first evaluate whether software measures represent a suitable alternative to controlled experiments. Software measures rely only on source- code properties and do not require observing human participants. However, the results indicate that software measure are not a reliable substitute for controlled experiments with human participants. Hence, we aim at reducing the effort for controlled experi- ments with human participants. To this end, we conduct a literature survey of controlled experiments of the last ten years published in seven major journals and conferences of the software-engineering domain. This way, we get an overview of the state of the art of measuring program comprehension, on which we base a framework to support plan- ning, conducting, and replicating experiments. Specifically, we provide a list of con- founding parameters for program comprehension, including control techniques, so that researchers know how the validity of experiments is threatened and how they can ad- dress the threats. Doing so, we reduce the major obstacle for conducting controlled exper- iments with human participants. Furthermore, we develop and evaluate a questionnaire to reliably and conveniently measure programming experience, the most important con- founding parameter for program comprehension. Additionally, we provide a tool that helps researchers to design and replicate experiments. With our second goal, we analyze how a specific programming approach, feature- oriented software development, affects program comprehension. To this end, we conduct a series of controlled experiments, in which we show how different facets of feature- oriented software development affect program comprehension. Specifically, we show ii how background colors speed up program comprehension in terms of locating code frag- ments. Furthermore, we evaluate whether a claimed benefit for program comprehension, which is separation of code along features, really improves program comprehension. Acknowledgements There are numerous people I like to thank for their support during the last three years. First, I thank Norbert, my loving husband. He always cheered me up when nobody wanted to read my papers, and shared the happy moments when I finally started to get papers accepted. Thank you so much. I also like to thank Christian Kastner¨ and Sven Apel. Christian constantly doubted my work, and Sven always believed in it. This way, I learned how to convince others of my work: If I can convince Christian, then I can convince everyone. They were a great team of advisors and I am very glad I know them both. I consider Christian and Sven not only as advisors, but also as colleagues and friends. Without you, Norbert would have much harder work in cheering me up. My supervisor Gunter Saake gave me the freedom to follow my research ideas. He always had time to listen to my concerns and taking care of my funding. Furthermore, he supported me with my visit of Don Batory at the University of Texas at Austin. My six-months trip to the University of Texas was an awesome experience. I thank Don Batory for his great support during this time. He added a lot of interesting input to my research and helped to improve my English and style of writing. Furthermore, I like to thank all my friends for taking my mind off research and helped to make the last three years really awesome. Also many thanks to all my participants, who made this work possible in the first place. I also like to thank Arie van Deursen, who agreed to review my thesis on short notice, although he knew it would be more than 200 pages to read. CONTENTS iii Contents Contents iii List of Figures ix List of Tables xii List of Abbreviations xiv 1 Introduction1 1.1 Contributions...................................3 1.2 Outline.......................................4 1.2.1 Part I....................................4 1.2.2 Part II....................................5 1.3 Limitation.....................................5 1.4 How to Read this Thesis.............................6 2 Background7 2.1 Program Comprehension.............................7 2.1.1 Top-down Models............................7 2.1.2 Bottom-up Models............................8 2.1.3 Integrated Models............................8 2.1.4 Discussion of Comprehension Models.................9 2.1.5 Measuring Program Comprehension.................. 10 2.2 Experiments.................................... 11 2.2.1 Objective Definition........................... 12 2.2.2 Design................................... 13 2.2.3 Experiment Execution.......................... 15 CONTENTS iv 2.2.4 Data Analysis............................... 15 2.2.5 Interpretation............................... 18 2.2.6 Ethical Issues............................... 19 2.3 Feature-Oriented Software Development................... 19 2.3.1 Physical Separation of Concerns.................... 20 2.3.2 Virtual Separation of Concerns..................... 22 2.4 MobileMedia.................................... 23 2.5 Summary...................................... 24 I Framework for Conducting Program-Comprehension Experiments 26 3 Software Measures and Program Comprehension 27 3.1 Background: Software Measures........................ 28 3.1.1 Size Measures............................... 29 3.1.2 Complexity Measures.......................... 30 3.1.3 Concern Measures............................ 30 3.2 Experimental Design............................... 31 3.2.1 Objective.................................. 32 3.2.2 Material.................................. 32 3.2.3 Participants................................ 33 3.2.4 Tasks.................................... 35 3.2.5 Experiment Execution.......................... 36 3.3 Experiment Results................................ 36 3.4 Software Measures and Program Comprehension.............. 38 3.4.1 Software Measures of Complete System................ 39 3.4.2 Software Measures in Terms of Concerns............... 39 3.4.3 Software Measures Related to Files................... 39 3.4.4 Software Measures Weighted with Response Time.......... 40 3.5 Discussion..................................... 42 3.6 Threats to Validity................................. 43 3.6.1 Internal Validity.............................. 43 3.6.2 External Validity............................. 43 3.7 Related Work................................... 44 3.8 Summary...................................... 45 CONTENTS v 4 Confounding Parameters 46 4.1 Importance of Confounding Parameters.................... 47 4.2 Methodology of Literature Survey....................... 48 4.3 State of the Art................................... 51 4.3.1 Describing Confounding Parameters.................. 51 4.3.2 Measuring and Controlling for Confounding Parameters...... 52 4.3.3 Number of Confounding Parameters................. 53 4.4 Background: Controlling for Confounding Parameters........... 53 4.4.1 Randomization.............................. 54 4.4.2 Matching.................................. 54 4.4.3 Keep Confounding Parameter Constant................ 55 4.4.4 Use Confounding Parameter as Independent Variable........ 56 4.4.5 Analyze the Influence of Confounding Parameter on Result.... 56 4.4.6 Summary of Control Techniques.................... 56 4.5 Identified Confounding Parameters.....................