Assessing Bibliometrics for the Automation of Technology Readiness Level Assessments by Safa Faidi A thesis submitted in conformity with the requirements for the degree of Master of Applied Science Department of Mechanical & Industrial Engineering University of Toronto © Copyright by Safa Faidi 2021 Assessing Bibliometrics for the Automation of Technology Readiness Level Assessments Safa Faidi Master of Applied Science Department of Mechanical & Industrial Engineering University of Toronto 2021 Abstract Technology Readiness Levels (TRL) are used to assess technology maturity, an important factor in technology investment and onboarding decisions. Expertly assessed TRLs are time- consuming, financially demanding, and subject to bias. While quantitative TRL assessment tools have been suggested, their accuracy is yet to be confirmed. To address this, we explore an existing method that used prevailing bibliometric models of maturity to assess TRLs. We automated and replicated the method and investigated its reliability by estimating the TRLs of 23 technologies. We show that the results of the automated method do not agree with expertly assessed TRLs and we offer hypotheses as to why bibliometrics fail to provide accurate technology maturity estimates. We argue that the single indicator model commonly used to assess technology life cycle stages must be revised. Finally, we suggest machine learning and Natural Language Processing tools to replace outdated models of maturity and automate TRL assessments. ii Acknowledgments The experiences and knowledge I gained during these last two years at UofT are unforgettable. I would like to start by thanking my supervisor and mentor, Alison Olechowski. Alison, I’ve learned so much from you. You taught me how to be a better leader (and how to manage an excellent meme slack channel). I am very grateful for the opportunity to be a part of Ready Lab and the friendships I’ve made there. Thank you for your constant support and guidance, especially through the past few challenging months. I would also like to thank Florian Wolf for his input on this project. Florian is the founder and CEO of Mergeflow, an exciting analytics company based in Germany, and I’ve had the privilege of having him as an industry partner on this project. His professional expertise and feedback on this work have influenced its direction greatly. Thank you, Florian. I would also like to thank my committee member, Greg Jamieson, for his time and feedback on this research project. I’ve made lifelong friends at the University of Toronto. I was lucky to be a part of the amazing ILead Grad team, where we connected graduate students together. The community we built there will be something I cherish forever. Thank you to my wonderful parents for all they’ve done to get me here, and a big thank you to my sisters, and a special one for my little sister Ameera, for being my number one supporters. Finally, I would like to honor the memory of my good friend Faizan Butt who unfortunately left us too soon. I had the pleasure of meeting Faizan when we were both grad students at UofT, and our friendship will be something I always treasure. Thank you all. iii Table of Contents Acknowledgments ..................................................................................................................... iii Table of Contents ...................................................................................................................... iv List of Tables ............................................................................................................................ vi List of Figures .......................................................................................................................... vii List of Appendices ..................................................................................................................... x Introduction ........................................................................................................................... 1 1.1 Technology Readiness Levels (TRLs): from NASA to Industry ....................................... 1 1.2 Issues with Expertly Assessed Technology Readiness Levels .......................................... 3 1.3 Thesis Outline ................................................................................................................. 4 Automating Technology Readiness Level Assessment using Bibliometrics ............................ 6 2.1 Introduction .................................................................................................................... 6 2.2 Background .................................................................................................................... 6 2.3 Bibliometric Method for Assessing Technological Maturity (BIMATEM) ...................... 9 3 Evaluating Bibliometrics for the Automation of TRL Assessments .......................................14 3.1 Introduction ...................................................................................................................14 3.2 Methods .........................................................................................................................14 3.2.1 Database Selection .............................................................................................15 3.2.2 Technology Selection .........................................................................................17 3.2.3 Forming Search Queries .....................................................................................19 3.2.4 Calculating S values ...........................................................................................22 3.2.5 Calculating ATS Values .....................................................................................24 3.2.6 TRL Estimation ..................................................................................................26 3.3 Results and Discussion ..................................................................................................27 iv Gaps in Automating Technology Readiness Level Assessments Using Bibliometrics ............31 4.1 Flawed Mapping of Bibliometric Indicators to Technology Readiness Levels ................31 4.2 The Biased Public Perception Around Technologies ......................................................35 4.3 Inaccurate Technology Trends .......................................................................................38 4.3.1 The Hype Cycle .................................................................................................38 4.3.2 The Logistic Growth Curve ................................................................................41 4.4 Maturity is dependent on Technology Application and Context .....................................44 4.5 Unclear Benchmarking of Maturity ................................................................................47 4.6 Forming Queries Without Expert Opinion ......................................................................50 Conclusions and Recommendations ......................................................................................53 5.1 Introduction ...................................................................................................................53 5.2 Future Work: Modifications to Bibliometric Technology Readiness Level Assessments ..................................................................................................................53 5.2.1 Designing a Machine Learning Model ................................................................53 5.2.2 Surveying Practitioners and Updating Query Designs .........................................55 5.3 Limitations ....................................................................................................................56 5.4 Conclusion .....................................................................................................................57 Bibliography .............................................................................................................................59 Appendices................................................................................................................................64 Appendix A: List of Queries ......................................................................................................64 Appendix B: Technology Record Collection..............................................................................72 1. Scopus ..............................................................................................................................72 2. USPTO .............................................................................................................................74 3. Factiva ..............................................................................................................................77 4. ATS Calculation ...............................................................................................................79 5. TRL Estimation ................................................................................................................80 v List of Tables Table 2.1: Technology life cycle stages and their bibliometric indicators (adapted from [25])..... 7 Table 2.2: TRL for each technology life cycle stage and the matched technology record database (adapted from [15]). ..................................................................................................................