• Abstract and Figures
• Public Full-text

Rochester Institute of Technology RIT Scholar Works Theses 6-2021 The State of Practice for Security Unit Testing: Towards Data Driven Strategies to Shift Security into Developer's Automated Testing Workflows Danielle Nicole Gonzalez [email protected] Follow this and additional works at: https://scholarworks.rit.edu/theses Recommended Citation Gonzalez, Danielle Nicole, "The State of Practice for Security Unit Testing: Towards Data Driven Strategies to Shift Security into Developer's Automated Testing Workflows" (2021). Thesis. Rochester Institute of Technology. Accessed from This Dissertation is brought to you for free and open access by RIT Scholar Works. It has been accepted for inclusion in Theses by an authorized administrator of RIT Scholar Works. For more information, please contact [email protected]. The State of Practice for Security Unit Testing: Towards Data Driven Strategies to Shift Security into Developer's Automated Testing Workflows by Danielle Nicole Gonzalez A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computing and Information Sciences B. Thomas Golisano College of Computing and Information Sciences Rochester Institute of Technology Rochester, New York June 2021 The State of Practice for Security Unit Testing: Towards Data Driven Strategies to Shift Security into Developer's Automated Testing Workflows by Danielle Nicole Gonzalez Committee Approval: We, the undersigned committee members, certify that we have advised and/or supervised the candidate on the work described in this dissertation. We further certify that we have reviewed the dissertation manuscript and approve it in partial fulfillment of the requirements of the degree of Doctor of Philosophy in Computing and Information Sciences. Dr. Mehdi Mirakhorli Date Dissertation Advisor Dr. Laurie Williams Date Dissertation Committee Member Dr. Mohamed Wiem Mkaouer Date Dissertation Committee Member Dr. Christian Newman Date Dissertation Committee Member Dr. Darren Narayan Date Dissertation Defense Chairperson Certified by: Dr. Pengcheng Shi Date Ph.D. Program Director, Computing and Information Sciences ii iii ©2021 Danielle Nicole Gonzalez All rights reserved. The State of Practice for Security Unit Testing: Towards Data Driven Strategies to Shift Security into Developer's Automated Testing Workflows by Danielle Nicole Gonzalez Submitted to the B. Thomas Golisano College of Computing and Information Sciences Ph.D. Program in Computing and Information Sciences in partial fulfillment of the requirements for the Doctor of Philosophy Degree at the Rochester Institute of Technology Abstract The pressing need to \shift security left" in the software development lifecycle has motivated efforts to adapt the iterative and continuous process models used in practice today. Security unit testing is praised by practitioners and recommended by expert groups, usually in the context of DevSecOps and achieving \continuous security". In addition to vulnerability testing and standards adherence, this technique can help developers verify that security controls are implemented correctly, i.e. functional security testing. Further, the means by which security unit testing can be integrated into developer workflows is unique from other standalone tools as it is an adaptation of practices and infrastructure developers are already familiar with. Yet, software engineering researchers have so far failed to include this technique in their empirical studies on secure development and little is known about the state of practice for security unit testing. This dissertation is motivated by the disconnect between promotion of security unit testing and the lack of empirical evidence on how it is and can be applied. The goal of this work was to address the disconnect towards identifying actionable strategies to promote wider adoption and mitigate observed challenges. Three mixed- method empirical studies were conducted wherein practitioner-authored unit test code, Q&A posts, and grey literature were analyzed through three lenses: Practices (what they do), Perspectives and Guidelines (what and how they think it should be done), and Pain Points (what challenges they face) to incorporate both technical and human factors of this phenomena. Accordingly, this work contributes novel and important insights into how developers write functional unit tests for at least nine security controls, including a taxonomy of 53 authentication unit test cases derived from real code and a detailed analysis of seven unique pain points that developers seek help with from peers on Q&A sites. Recommendations given herein for conducting and adopting security unit testing, including mitigating challenges and addressing gaps between available and needed support, are grounded in the guidelines and perspectives on the benefits, limitations, use cases, and integration strategies shared in grey literature authored by practitioners. iv Acknowledgments \I am the combined effort of everyone I've ever known." Chuck Palahniuk, Invisible Monsters It takes a village to raise a PhD student, and I would like to thank all those who have supported, motivated, comforted, inspired, and challenged me. I thank Dr. Darren Narayan for first seeing my research potential and mentoring me as a REU student. This experience was my introduction to the research world and instilled a sense of con- fidence in my ability to succeed in unfamiliar territory. I credit my interest in pursuing research as a career to the time spent in my PhD advisor Dr. Mehdi Mirakhorli's lab as an undergraduate student. I thank him for encouraging and sponsoring my entry into the field of software engi- neering research and for helping me navigate the intimidating worlds of academia, paper writing, rejections, conferences, funding, presentations, international travel, and much more throughout my time in his lab. Special thanks are extended to Dr. Thomas Zimmermann for his mentorship and for the opportunities to work with him and the SAINTes group at Microsoft Research in person and virtually during the pandemic. These experiences broadened my perspectives, challenged me in new ways, and left me with a lasting desire to make impactful and actionable contributions with my research in support of those whose work I study, i.e. developers. I am also incredibly grateful for Microsoft Research's support in awarding me a PhD Fellowship, which empowered me to concentrate on pursuing my ideas to the depths described herein. To any students who may read this in the future, having a group of PhD friends to lean on and learn from is vital to maintaining your health and sanity. Thanks to Nuthan Munaiah, Joanna Cecilia da Silva Santos, Blagovesta Kostova, Jamie Davis, and all of the PhD students who have climbed this hill alongside me and I am so fortunate to now call my friends. There are countless others I am lucky to have in my personal and professional life who have also contributed to my success | if you think this is about you, it probably is. v To my parents Christine and Manuel, who empowered and enabled me to pursue my dreams through their sacrifices and unwavering belief in my potential. To Joseph, the light by which I persevere, and so much more. vi Contents 1 Introduction 1 1.1 Motivation and Research Goal . .2 1.2 Research Questions . .2 1.3 Contributions . .3 1.4 Outline . .5 2 Background and Related Work 6 2.1 The Software Development Lifecycle . .6 2.2 Shifting Security Left . .6 2.2.1 Microsoft Secure Development Lifecycle (SDL) . .7 2.2.2 OWASP Secure Application Maturity Model (SAMM) . .7 2.2.3 DevSecOps: Adding Continuous Security to the DevOps Model . .7 2.3 Security Practices . .8 2.3.1 Building Security In: Architectural Tactics and Control Mechanisms . .8 2.3.2 Testing Security Throughout the SDLC . .9 2.3.3 Security Unit Testing . 11 vii CONTENTS viii 2.3.4 Human Factors Influencing Developer's Security Practices . 12 2.4 Automated Unit and Integration Testing . 13 2.4.1 Use in Practice . 15 2.4.2 Human Factors Influencing Developer's Testing Practices . 17 2.5 Empirical Studies of Developer's Q&A Posts . 19 2.6 Grey Literature Reviews in Software Engineering . 21 3 Research Design 24 3.1 Study 1: Grounded Theory Analysis of Unit Test Code . 25 3.2 Study 2: Mixed-Method Analysis of Developer's Q&A Posts . 26 3.3 Study 3: Qualitative Analysis of Grey Literature . 27 4 Security Unit Testing in Practice 28 4.1 What Security Controls Are Unit Tested? . 28 4.2 How Do Developers Unit Test Authentication? . 29 4.3 Conclusion . 34 4.4 Study Limitations . 34 5 Security Unit Testing Pain Points 36 5.1 The Unique Pain Points of Writing Security Unit Tests . 36 5.2 Conclusion . 41 5.3 Study Limitations . 42 6 Perspectives and Guidelines 43 CONTENTS ix 6.1 Characterization of Practitioner-Authored Grey Literature . 43 6.1.1 Sources, Types, and Tones . 45 6.1.2 Scopes, Purposes, and Content Types . 48 6.2 Perspectives on Security Unit Testing . 52 6.2.1 When, How, & Why Practitioners Want You to Write Security Unit Tests . 53 6.3 Instruction and Guidelines for Security Unit Testing . 56 6.3.1 In Formal Security Testing Guides . 56 6.3.2 In Informal Guidelines from Practitioners . 57 6.4 Conclusion . 58 6.5 Study Limitations . 58 7 Discussion 61 7.1 The State of Practice for Security Unit Testing . 61 7.2 Recommendations . 63 7.2.1 Who Should Follow This Advice? . 63 7.2.2 Strategies for Formalization and Workflow Integration . 64 7.2.3 Summary & Important Considerations . 66 7.3 A Roadmap to Supporting Developer-Driven Security Testing . 67 7.3.1 Towards Actionable Guidelines for Test Design & Construction . 67 7.3.2 Towards Automated Tools that Recognize Developer's Needs & Priorities . 71 8 Conclusions 73 Appendices 94 CONTENTS x A Grounded Theory Analysis of Security Unit Test Code 95 B Mixed-Method Analysis of Developer's Q&A Posts 101 List of Figures 3.1 Grounded Theory Approach for Inductive Analysis of Security Unit Tests .

Load more

  118

Copy Link