Assessing Bibliometrics for the Automation of Technology Readiness Level Assessments

Safa Faidi

A thesis submitted in conformity with the requirements for the degree of Master of Applied Science Department of Mechanical & Industrial Engineering University of Toronto

Assessing Bibliometrics for the Automation of Technology Readiness Level Assessments

Safa Faidi

Master of Applied Science

Department of Mechanical & Industrial Engineering University of Toronto

2021 Abstract

Technology Readiness Levels (TRL) are used to assess technology maturity, an important factor in technology investment and onboarding decisions. Expertly assessed TRLs are time- consuming, financially demanding, and subject to bias. While quantitative TRL assessment tools have been suggested, their accuracy is yet to be confirmed. To address this, we explore an existing method that used prevailing bibliometric models of maturity to assess TRLs. We automated and replicated the method and investigated its reliability by estimating the TRLs of 23 technologies. We show that the results of the automated method do not agree with expertly assessed TRLs and we offer hypotheses as to why bibliometrics fail to provide accurate technology maturity estimates. We argue that the single indicator model commonly used to assess technology life cycle stages must be revised. Finally, we suggest machine learning and

Natural Language Processing tools to replace outdated models of maturity and automate TRL assessments.

Acknowledgments

The experiences and knowledge I gained during these last two years at UofT are unforgettable.

I would like to start by thanking my supervisor and mentor, Alison Olechowski. Alison, I’ve learned so much from you. You taught me how to be a better leader (and how to manage an excellent meme slack channel). I am very grateful for the opportunity to be a part of Ready Lab and the friendships I’ve made there. Thank you for your constant support and guidance, especially through the past few challenging months.

I would also like to thank Florian Wolf for his input on this project. Florian is the founder and CEO of Mergeflow, an exciting analytics company based in Germany, and I’ve had the privilege of having him as an industry partner on this project. His professional expertise and feedback on this work have influenced its direction greatly. Thank you, Florian.

I would also like to thank my committee member, Greg Jamieson, for his time and feedback on this research project.

I’ve made lifelong friends at the University of Toronto. I was lucky to be a part of the amazing ILead Grad team, where we connected graduate students together. The community we built there will be something I cherish forever.

Thank you to my wonderful parents for all they’ve done to get me here, and a big thank you to my sisters, and a special one for my little sister Ameera, for being my number one supporters.

Finally, I would like to honor the memory of my good friend Faizan Butt who unfortunately left us too soon. I had the pleasure of meeting Faizan when we were both grad students at UofT, and our friendship will be something I always treasure.

Thank you all.

iii

Table of Contents

Acknowledgments ...... iii

Table of Contents ...... iv

List of Tables ...... vi

List of Figures ...... vii

List of Appendices ...... x

Introduction ...... 1

1.1 Technology Readiness Levels (TRLs): from NASA to Industry ...... 1

1.2 Issues with Expertly Assessed Technology Readiness Levels ...... 3

1.3 Thesis Outline ...... 4

Automating Technology Readiness Level Assessment using Bibliometrics ...... 6

2.1 Introduction ...... 6

2.2 Background ...... 6

2.3 Bibliometric Method for Assessing Technological Maturity (BIMATEM) ...... 9

3 Evaluating Bibliometrics for the Automation of TRL Assessments ...... 14

3.1 Introduction ...... 14

3.2 Methods ...... 14

3.2.1 Database Selection ...... 15

3.2.2 Technology Selection ...... 17

3.2.3 Forming Search Queries ...... 19

3.2.4 Calculating S values ...... 22

3.2.5 Calculating ATS Values ...... 24

3.2.6 TRL Estimation ...... 26

3.3 Results and Discussion ...... 27 iv

Gaps in Automating Technology Readiness Level Assessments Using Bibliometrics ...... 31

4.1 Flawed Mapping of Bibliometric Indicators to Technology Readiness Levels ...... 31

4.2 The Biased Public Perception Around Technologies ...... 35

4.3 Inaccurate Technology Trends ...... 38

4.3.1 The Hype Cycle ...... 38

4.3.2 The Logistic Growth Curve ...... 41

4.4 Maturity is dependent on Technology Application and Context ...... 44

4.5 Unclear Benchmarking of Maturity ...... 47

4.6 Forming Queries Without Expert Opinion ...... 50

Conclusions and Recommendations ...... 53

5.1 Introduction ...... 53

5.2 Future Work: Modifications to Bibliometric Technology Readiness Level Assessments ...... 53

5.2.1 Designing a Machine Learning Model ...... 53

5.2.2 Surveying Practitioners and Updating Query Designs ...... 55

5.3 Limitations ...... 56

5.4 Conclusion ...... 57

Bibliography ...... 59

Appendices...... 64

Appendix A: List of Queries ...... 64

Appendix B: Technology Record Collection...... 72

1. Scopus ...... 72

2. USPTO ...... 74

3. Factiva ...... 77

4. ATS Calculation ...... 79

5. TRL Estimation ...... 80

List of Tables

Table 2.1: Technology life cycle stages and their bibliometric indicators (adapted from [25])..... 7

Table 2.2: TRL for each technology life cycle stage and the matched technology record database (adapted from [15])...... 9

Table 3.1: Technology list with assessed TRLs [12]...... 18

Table 3.2: Query concept mapping for “Speech synthesis” ...... 20

Table 3.3: Query concept mapping for “Millimetric radio waveforms” ...... 21

Table 3.4: Query inclusion operators ...... 21

Table 3.5: The S values for the training set and the resulting ATS value for each database ...... 25

Table 3.6: Technologies with estimated bibliometric TRL and expert assessed TRL ...... 28

Table 3.7: Standard deviation and prediction interval for training set’s S values ...... 29

Table 4.1: Technology life cycle stage, the equivalent Technology Readiness Level, R&D stage and bibliometric indicator (adapted from [10], [15], [25])...... 32

Table 4. 2: Survey response for agreeing (yes) /disagreeing (no) with the TRL 9 assessment of 21 SSTR technologies ...... 48

Table 4.3: A summary of the design and search methodologies for queries in literature ...... 51

List of Figures

Figure 1-1: Technology Readiness Levels (adapted from [2])...... 1

Figure 1-2: Horizon 2020 Technology Readiness Level definitions [8] ...... 2

Figure 2-1: Technology life cycle stages of the logistic growth curve and their respective bibliometric indicator (adapted from [15], [25])...... 8

Figure 2-2: TRL estimation through technology publications [15] ...... 10

Figure 2-3: Comparing S values of each database to ATS values of each database to determine TRL (adapted from [15])...... 13

Figure 3-1: Flowchart showing an overview of the data collection and calculation process for the training set. The Input from training set shown on the bottom right can be found in Figure 3-2 and discussed in section 3.2.5...... 23

Figure 3-2 Flowchart showing an overview of the data collection and calculation process for the training set...... 26

Figure 3-3: Flowchart describing TRL estimation through comparing standard error of regression values (S) to the absolute standard error of S value (ATS) to determine the technology life cycle stage...... 27

Figure 3-4: TRL assignment using the ATS values calculated for each database using the training set...... 27

Figure 3-5: Mature (TRL 8-9) patent application records logistic fits ...... 30

Figure 4-1: The Watts and Porter bibliometric indicator model (adapted from [10]). Note that the magnitude of peaks is irrelevant...... 32

Figure 4-2: (a) Illustration of the expected bibliometric trend for emerging, growing, and mature technologies according to the Watts and Porter model, versus (b) the real bibliometric trend for Nano-battery, LIDAR, and Z-wave from Scopus, USPTO, and Factiva. Technology Readiness Level assessment is from the 2011 SSTR report...... 34

vii

Figure 4-3: (a) Illustration of the bibliometric trend for emerging technologies according to the Watts and Porter model, versus (b, c) the real bibliometric trend for Proton exchange membrane (PEM) fuel cell, and Piezoelectric energy harvesting from Scopus, USPTO, and Factiva. Technology Readiness Level assessment is from the 2011 SSTR report...... 36

Figure 4-4: (a) Illustration of the expected bibliometric trend for a growing technology according to the Watts and Porter model, versus (b) the real bibliometric trend for self-driving car from Scopus, USPTO, and Factiva collected from 1976 to 2020...... 37

Figure 4-5: Hype curve is the sum of Hype level and S-curves [29]...... 39

Figure 4-6: Gartner’s hype cycle and its five phases [29]...... 39

Figure 4-7: News and media record plots for mature (TRL 9) technologies collected from 1976 to 2020...... 40

Figure 4-8: The rate of growth bell curve modeling the life cycle stages and the resulting cumulative growth curve (i.e. S-curve/logistic growth curve), tm is the point of symmetry or inflection point (adapted from [27])...... 41

Figure 4-9:Mature (TRL 8-9) technology plots of their cumulative scientific publication records (source: Scopus) fit to logistic growth curve model. Technology Readiness Level assessment is based on the SSTR report in 2011...... 42

Figure 4-10: Mature (TRL 8-9) technology plots of their cumulative patent application records (source: USPTO) fit to logistic growth curve model. Technology Readiness Level assessment is based on the SSTR report in 2011...... 43

Figure 4-11: Applications of Computer Vision with varying TRLs assessed in 2020 (adapted from [62]) (*in less controlled environments like busy railway stations (i.e. pose variations, low quality/resolution, bad lighting, etc.)) ...... 45

Figure 4-12: Self-Driving Car TRL according to the 5 levels of SAE assessed in 2020 (adapted from [62])...... 46

viii

Figure 4-13: Tradeoff between precise searches (i.e. searching in the highlight section fields) and sensitive searches (i.e. searching in all-text fields) [36]...... 51

Figure 5-1: Machine learning model to predict Technology Readiness Levels using eight bibliometric indicators: venture capital, market estimates, technology licensing, research projects, clinical trials, scientific publications, patents, and news and media...... 55

List of Appendices

Appendix A: List of Queries ...... 64

Appendix B: Technology Record Collection...... 72

Introduction 1.1 Technology Readiness Levels (TRLs): from NASA to Industry

Innovative technologies have revolutionized the world. Industries are constantly on the look-out for emerging technologies that could give them a competitive advantage. Today, an important task of an R&D manager is deciding whether they should hold, invest, or disinvest in a technology [1]. These decisions heavily depend on understanding the technical readiness of technologies, a metric that is often challenging to quantify. How do we measure technical readiness? When faced with this question in the 1970s, the National Aeronautics and Space Administration (NASA) engineers developed the Technology Readiness Levels (TRLs): a nine- level scale that quantifies readiness and maturity of technologies [2].

Figure 1-1: Technology Readiness Levels (adapted from [2]).

The Technology Readiness Level scale was developed by NASA’s researcher Stan Sadin in 1974 and initially consisted of seven maturity levels. The scale continued to be revised until it reached its final form in the 1990’s consisting of nine maturity levels shown in Figure 1-1. At TRL 1, the scale starts at a technology’s most basic scientific form and progresses to TRL 9 where a technology is successful in its operational environment. Research suggests a significant inverse relationship between Technology Readiness Levels and risks of technology adoption [3]. A

1 2 technology of lower TRL has a high perceived risk while a technology of higher TRL has a low perceived risk.

The scale’s initial maturity level definitions were centered around NASA’s space missions and flight operations. However, the scale started to move to other industries, like the US Department of Defense (DoD) in the early 2000’s. Today, the TRL scale can be seen in industries like Biomedical and Manufacturing. These industries adopted their own TRL definitions to reflect their operations and applications [4],[5]. In addition, TRLs are seen in government grants and funding applications. For example, the Clean Growth in the Natural Resource Sectors Program (CGP) in Canada dedicated $ 155 million to support clean technology research and currently uses the TRL scale to assess applicant’s technology readiness and eligibility for a grant [6]. The EU’s large Horizon 2020 project also adopted the TRL scale as its main tool for assessing applicants for funding [7]. The revised general TRL definitions used by the Horizon 2020 program can be seen in Figure 1-2.

Figure 1-2: Horizon 2020 Technology Readiness Level definitions [8]

Technology Readiness Levels are often used in technology roadmapping. Technology roadmapping is a planning process led by industry experts that gives decision-makers, like R&D managers, the means to evaluate and select strategic alternatives for achieving technological

3 advantages [9]. By defining TRLs (i.e. maturity status), decision-makers can have a holistic picture of the risk levels for adopting and acquiring technologies.

One common way of assessing maturity in a technology roadmap report is Delphi: a method that uses the collaboration and input of an expert panel in the form of a questionnaire or survey. Delphi is currently the most popular and used method in technology roadmapping [10], [11]. For example, in 2011 the Government of Canada created the Soldier System Technology Roadmap (SSTR) report that assessed the TRL of different technologies to investigate potential investment opportunities that can impact the competitiveness of Canada’s Department of National Defense (DND). The assessment took place through several workshops with over 1,550 participants from government, academia, companies and research organizations, both national and international. The TRL assessments of technologies were done collaboratively during in-person working sessions or by delegating the tasks to content experts [12].

The Technology Readiness scale is a powerful and popular tool. Its ability to quantify and measure maturity gives industry professionals the means to make confident business decisions. The TRL scale has made its way to various industries since its initial development for NASA’s space missions. Today, the scale is used in government innovation funding programs, and technology roadmapping projects. Consequently, this wide adoption of the Technology Readiness Level scale has made it the subject of scrutiny by both researchers and practitioners over the last few decades.

1.2 Issues with Expertly Assessed Technology Readiness Levels

Previous studies highlighted the subjectivity of the TRL assessment as a challenge worth addressing [13]–[16].

Olechowski et al. conducted 19 interviews with TRL practitioners from seven industries including space, defense, and electronics to uncover challenges of TRL assessments [13]. The practitioners interviewed had ranging experiences with the TRL scale, some with over eight years of experience and others with less than two. The practitioners were either responsible for, directly involved in, or used the outputs of TRL assessments. The interviews highlighted 15 TRL challenges across three categories: system complexity, planning and review, and assessment

4 validity. The challenges were then ranked based on criticality through a survey of 113 industry practitioners.

One of the challenges highlighted was the subjective nature of the TRL assessment. One practitioner noted that the TRL assessment can often be influenced by a stakeholder’s power or interest. Another related challenge was the oversimplification of the scale’s definitions, which summarizes a technology’s complex maturity journey in nine simple levels. Although this gives the scale the power of quantifying maturity in a concise manner, practitioners reported that the TRL’s descriptions are often too general to be useful. We believe that this is closely related to the subjectivity of the TRL assessment challenge. The low-level detail of each level’s description invites personal interpretations and thus adds to the biased, and possibly inaccurate, assignment of an appropriate TRL.

As discussed in the previous section, the Soldier System Technology Roadmap (SSTR) report’s TRL assessments were done by expert collaboration over seven workshops that extended from June 2009 to September 2010. Expert opinion assessments are often used in most technology roadmapping projects; however, they are time-consuming and financially demanding. Technology Readiness Level assessments done through expert opinion collaborations may create subjective TRL results that are irreproducible outside the expert group. As a result, the automation of the Technology Readiness Level assessment was the main motivation of this research.

1.3 Thesis Outline

The gaps in the application of Technology Readiness Levels highlighted by previous researchers motivated us to search for methods that can automate TRL assessments.

The thesis consists of six chapters. In chapter 1, we introduced the Technology Readiness Level scale along with the reported subjectivity of the assessments. In chapter 2, we will first introduce bibliometrics as a technology assessment and forecasting tool and explore one method developed by researchers that uses bibliometrics to estimate the TRL of a technology. Next, in chapter 3, we will show how we automated and replicated the method discussed in chapter 2 and used it to estimate the TRLs of 23 technologies to address the following research question: is bibliometrics an accurate tool to estimate TRLs?

The journey of addressing this research question revealed a series of gaps and challenges that we believe complicate a total automation of the TRL assessment. These complications will be discussed in chapter 4. Finally, in chapter 5, we propose recommendations for improving the precision of bibliometrics in assessing Technology Readiness Levels. Although our recommendations require further development and testing, they may be promising in improving assessment results compared to the method presented in chapter 2. Chapter 5 will also discuss the limitations of our research and conclude this thesis.

Automating Technology Readiness Level Assessment using Bibliometrics 2.1 Introduction

Previous research has shown that the Technology Readiness Level assessment may be affected by personal bias as users tend to use their own personal expertise in deciding on a maturity level. To address this challenge, researchers have attempted to rely on quantitative data to create an objective maturity assessment method as a substitute to the traditional qualitative expert opinion. In particular, Lezama-Nicolás et al. presented a framework called the Bibliometric Method for Assessing Technology Maturity (BIMATEM) that uses technology records from scientific publication, patent, and news and media databases to estimate TRLs of technologies [15]. The authors base their proposed method on a popular technology assessment and forecasting model developed by Watts and Porter [25]. Since its publication in 1997, the Watts and Porter bibliometric indicator model has been used by several technology assessment and forecasting literature.

In this chapter, we will first introduce the use of bibliometrics for technology maturity assessment and forecasting and describe the Watts and Porter bibliometric indicator model. We will then present the reader with the detailed steps of Lezama-Nicolás et al.’s bibliometric method for estimating TRLs.

2.2 Background

Bibliometrics is defined as the statistical measurement of texts, including but not limited to publications and books [17]. Historically, bibliometric methods have been used to assess a researcher’s, research group’s, institution’s or journal’s impact [18]. For example, bibliometric data can be publication counts to assess the productivity of a researcher in the academic field or citation counts to assess the impact of an academic paper.

Bibliometric methods can also be applied in the field of technology assessment and forecasting through the analysis of technology records. For example, Rodriguez-Salvador et al. used bibliometric records from scientific publications and patents to study the research and technological output of bioprinting technologies to determine the overall knowledge landscape

6 7

[19]. Bibliometrics can also be used to determine emerging technologies, which are often crucial in investment and research opportunities. Bengisu et al. used counts of bibliometric records from scientific publications and patents to forecast emerging technologies [20]. Moro et al. used scientific publication and patent documents to determine emerging technologies in the field of renewable energy using text mining [21],[22].

In addition, technology forecasting research uses counts of bibliometric records to predict the growth trajectory of technologies. Cho et al. used technology forecasting models to estimate the future growth of OLED technologies by collecting scientific publications and patent records since 1990 [23]. Bibliometrics can also help researchers compare technology progress across different countries. Lokuhitige et al. collected scientific publications and patent records from five leading IoT technology countries to compare the technology’s maturity progress [24].

Bibliometric technology forecasting and assessment research often use the technology life cycle (TLC) indicator model proposed by Watts and Porter [25]. The model uses the three technology life cycle stages, emerging, growing, and mature, and pairs them to a unique bibliometric indicator that can be used to assess the maturity status of a given technology (Table 2.1).

Table 2.1: Technology life cycle stages and their bibliometric indicators (adapted from [25])

TLC Stage Bibliometric Indicator

Emerging Science and engineering publications

Growing Patents

Mature News and media

The technology life cycle stages are often modeled using the logistic growth curve (or S-curve) [26]. The mathematical model of the logistic growth curve is shown in (1) where k is the upper limit of growth and a and b are the initial stage of diffusion and velocity of diffusion respectively.

푘 (1) 푟(푡) = 1 + 푎푒−푏(푡−푡0)

The logistic equation was first introduced in 1838 by Belgian mathematician Pierre-Francois Verhulst to describe the self-limiting growth of a population [27]. The logistic equation models the law of natural growth which can often be simplified to periods of birth, growth, maturity, decline and death of any system. These sets of periods are often referred to as the life cycle of a system. This makes the logistic growth model attractive and universal since all systems experience the same life cycle periods. For example, the Technology Readiness Levels also describe the birth (TRL 1-5), the growth (TRL 6-7) and the maturity (TRL 8-9) of a technology [28]. Although the TRLs do not model decline and death, they are the natural projection of any technology as time passes after it has reached peak maturity.

Figure 2-1: Technology life cycle stages of the logistic growth curve and their respective bibliometric indicator (adapted from [15], [25]).

The ability of the logistic equation to simplify any system into the life cycle periods makes it a versatile method to study the behaviors of different systems from the growth of crime to the topic of interest in this thesis: the elaboration of inventions and diffusion of innovations [27]. Figure 2- 1 shows the logistic growth curve with respect to the Watts and Porter model.

2.3 Bibliometric Method for Assessing Technological Maturity (BIMATEM)

The Bibliometric Method for Assessing Technology Maturity was developed by Lezama-Nicolás et al. [15]. The authors apply the existing technology life cycle and bibliometric indicator model developed by Watts and Porter to a framework they believe can estimate the TRL of technologies. The BIMATEM assumes that each TRL can be matched to a technology life cycle (TLC) stage and estimated using the bibliometric indicator of that stage (Table 2.2).

Table 2.2: TRL for each technology life cycle stage and the matched technology record database (adapted from [15]).

TLC Stage TRL Databases Statistical Approximation

Emerging 3 Scientific publications Logistic growth curve (S-curve)

4-5 Engineering publications Logistic growth curve (S-curve)

Growing 6-7 Patents Logistic growth curve (S-curve)

Mature 8-9 News and media records Hype-type evolution (Gartner’s Hype cycle)

As shown in Table 2.2, the authors pair each TLC stage with a TRL bracket. Emerging technologies are described as being TRL 3-5 where the technology is still undergoing proof of concept (TRL 3), or laboratory (TRL 4) and relevant-environment (TRL 5) component validation. The authors note that although TRL 1 and TRL 2 technologies are also considered emerging, they cannot be linked to a scientific publication database because most journals require proof of concept in order to be published [15]. Growing technologies are described as TRL 6-7 where their system prototype was validated in a relevant environment (TRL 6) or the target environment (TRL 7). Mature technologies are described as TRL 8-9 where the actual system was completed and qualified (TRL 8) or tested and proven in an operational environment (TRL 9).

For scientific and engineering publication databases, the authors use Science Citation Index (SCI) for proof of concept research (TRL 3) and INSPEC for applied and engineering research

(TRL 4-5). For patents, the authors use the commercial database Patseer and Factiva for collecting news and media records. The authors use two statistical approximation curves to study technology records: the S-curve (1) for scientific publications, engineering publications and patents and the hype-type evolution curve for news and media records. The hype-type evolution model is an adaptation of Gartner’s hype cycle that claims to model the news and media records of mature technologies over time. Gartner, a technology research and consulting firm, never provided a mathematical representation of the curve but explain that it is an extension of the original S-curve [29].

The model was mathematically estimated by Campani and Vaglio [30] and is shown in (2) where r(t) represents the S-curve, j is the proportionality constant, and t’=t-t* where t* is the modifier of delay. We say “estimated” because there is no mathematical proof that (2) accurately models the hype-cycle curve which will be further explained in sections 3.2.4 and 4.3.1. The S-curve and hype-type evolution curve can be seen in Figure 2-2.

푑푅(푡) 푎푏푘푒푏(푡−푡0) 푗푘 (2) 퐻(푡) = + 푗푅(푡′) = + 푑푡 [푎 + 푒푏(푡−푡0)]2 1 + 푎푒−푏(푡′−푡0)

Figure 2-2: TRL estimation through technology publications [15]

Figure 2-2 shows the proposed method for estimating TRLs using technology records. Records collected from each database are fitted to the statistical approximation curves (S-curves for scientific publications, engineering publications and patents and hype-type evolutions for news and media records) using non-linear regression. The goodness of fit is measured using the standard error of regression (S) value. The lower the S value, the better the model describes the data or technology records. S values for mature technologies are used to create a threshold to determine whether technologies have passed a TLC stage.

The following detailed steps explain the BIMATEM process of estimating TRLs of technologies using bibliometric records:

Step 1: Calculating the Acceptance Threshold for S Value (ATS)

Step 1.1: Selecting Mature Technologies and Designing Queries

Lezama-Nicolás et al. used the following 10 technologies to calculate ATS values: Cloud Computing, Data Mining, Location-Aware technology, Microelectromechanical systems, Organic Light emitting diodes, RFID (radio frequency identification), Smartphone, Speech recognition, Text-to-speech, and Wireless local area network. The authors suggest that these technologies are mature (TRL 8-9). The queries were designed and searched in each database (see steps 2.1 and 2.2).

Step 1.2: Finding the Standard Error of Regression (S)

The search query designed from step 1.1 is searched on each database: from Science Citation Index (SCI), INSPEC, Patseer and Factiva. The technology records collected from SCI, INSPEC, and Patseer are fit to the S-curve, while records collected from Factiva are fit to the hype-type evolution curve. Curve fitting is done using non-linear regression and the standard error of regression (S) value is calculated and recorded. The S value describes the average distance that the observed data fall from the regression line. In other words, the S value tells us how well the technology records fit the curve model. A lower S value means that the observed data are closer to the fitted line which indicate a good fit.

Step 1.3: Calculating the Acceptance Threshold for S (ATS) Value

The standard error of regression for each technology in each database is used to calculate the ATS value. The authors believe that this value acts as a threshold or benchmark that dictates whether a technology has passed the technology life cycle stage. The authors calculated ATS by finding the upper limit of the 95% prediction interval of the S values for each database. At the end of this step, four ATS values are recorded: ATSSCI, ATSINSPEC, and ATSPatseer, ATSFactiva. The ATS values act as a threshold for the goodness of fit of a technology’s records at each TLC stage.

Step 2: Estimating TRL of Technologies

Step 2.1: Technology Selection and Design of the Search Query

The technology selection stage is where the technology under assessment is chosen and all its associated synonyms and related keywords are defined. The search query is designed based on the stated keywords with the help of operators like “AND”, “OR”, and/or wildcards to widen the search while keeping it specific. In their paper, the authors state that their queries were designed with the help of experts.

Step 2.2: Retrieval of Results

The queries are used to search each database. The authors note that in this stage, the time periods for the searches need to be consistent (years or months; years are recommended).

Step 2.3: Technology Maturity Assessment

Figure 2-3 shows the summary of the technology maturity assessment. Each S value is compared to the ATS value of each TLC stage. The authors’ assumption is that if a technology has completed its maturity through a TLC stage, there would be a lower error in the fit of the record counts in the appropriate dataset to the statistical approximator (S-curve or hype-type evolution).

For example, the S value of SCI is calculated based on the fit of its data to the S-curve. If the recorded SSCI is less than or equal to the ATSSCI value (calculated in step 1.3), the authors suggest this might signal that the technology has passed this TLC stage. However, it must first be assessed using the criteria of the next TLC stage. Alternatively, if the recorded SSCI is larger than

the ATSSCI, this suggests that the technology record’s fit is worse than the average mature technology at that stage and so it should be assigned to the previous TLC stage.

Figure 2-3: Comparing S values of each database to ATS values of each database to determine TRL (adapted from [15]).

To test their method, the authors applied BIMATEM to seven Additive Manufacturing technologies to estimate their TRLs. However, the authors did not have any other source of TRL assessments of these technologies to authenticate their results. In addition, much of the BIMATEM’s steps were done manually and with a small dataset (17 technologies).

In our next chapter, we show how we automated and replicated the BIMATEM’s framework and we evaluated its TRL output using a bigger dataset of 45 technologies with expertly assessed TRLs.

Evaluating Bibliometrics for the Automation of TRL Assessments 3.1 Introduction

The accuracy of bibliometrics in assessing Technology Readiness Levels remains unknown. Lezama-Nicolás et al.’s work (BIMATEM, previously discussed in section 2.3) is currently the only published method that assesses TRLs using bibliometric data, however little information is known about its reliability. This chapter’s aim is to investigate the following research question: is the use of counts of bibliometric records an appropriate TRL indicator? We address this question by replicating and evaluating the BIMATEM framework using the bibliometric records of 45 technologies. The TRLs of these technologies were previously expertly assessed and published in the 2011 Soldier System Technology Roadmap (SSTR) report [12].

In this chapter, we discuss the methods used to replicate the BIMATEM (section 3.2). We then discuss the results which show that for all 45 technologies, there was disagreement between BIMATEM's TRL estimates and the SSTR expert TRL estimates. (section 3.3).

3.2 Methods

This section provides details on the methodology used to estimate the TRL of technologies using Lezama-Nicolás et al.’s bibliometric framework. We incorporated most of the steps of the original method with an exception to the choice of databases, query formation method, and the technology trends used. This section discusses the five steps of the BIMATEM method: database selection, technology selection, forming search queries, calculating standard error of regression values (S), calculating acceptance threshold for S values (ATS), and TRL estimation.

The databases chosen were Scopus for scientific and engineering publications, the United States Patent and Trademark Office (USPTO) for patent applications, and Factiva for news and media records. Records were collected from 1976 to 2011. This was because USPTO’s available archive collection is up to 1976 and the TRL assessment year of the SSTR report is 2011. Forty- five Technologies were selected from the 2011 Soldier System Technology Roadmap (SSTR) report developed by the Department of National Defense (DND). The technologies were used to

14 15 create two datasets: a training set (22 technologies) that will be used to calculate ATS, and a test set (23 technologies) that will be used to measure the accuracy of the BIMATEM.

After collecting technology records from each database for all 45 technologies, we applied non- linear logistic curve (S-curve) regression and the S values for each technology were recorded. For the training set, the S values were used to calculate the ATS for each database. The ATS values were used to assess whether a technology in the test set has passed a technology life cycle (TLC) stage: emerging (TRL 1-5), growing (TRL 6-7), and mature (TRL 8-9) by comparing each ATS of a database to each technology’s S value for the same database. The BIMATEM’s assumption here is that if a technology has completed its maturity through a TLC stage, we would expect it to have a lower error in the fit of the record counts to the S-curve compared to a mature technology at the same TLC stage. In other words, a technology is only considered to have passed a TLC stage if it meets the threshold (ATS) of the respective database.

We decided to fully automate the data collection and analysis process using Python 3 on Jupyter Notebook version 6.0. We believe this automated script will allow researchers to quickly and confidently collect bibliometric records. Appendix B of this thesis provides the full code used to collect data and calculate results. This section will provide an overview of the method used, but specific details of the Python script and the flowcharts of the code can be found in the appendix.

In the following sections, we will explain the details of each of the five steps described above.

3.2.1 Database Selection

In their methodology, Lezama-Nicolás et al. paired each technology life cycle stage to a unique dataset as suggested by Watts and Porter in 1997 [25]: emerging phase (TRL 3-5) is paired with scientific and engineering publications, growing phase (TRL 6-7) is paired with patents, and the mature phase (TRL 8-9) is paired with news and media records.

For our analysis, scientific and engineering publications, patent applications, and news and media records were collected from Scopus, USPTO, and Factiva respectively. To collect data automatically, we accessed the Application Programming Interface (API) of Scopus and USPTO using Python. Because Factiva does not offer an API, news and media records were collected directly from Factiva’s search engine.

Using an API allows for a more automated and large-scale data collection method. API calls are an attractive alternative to web scraping because they are significantly faster. In addition, web scraping is sensitive to website changes which force constant edits in the code and is prone to possible IP security blockage. Python 3 was used as the program language to access APIs. The software used was Jupyter Notebook version 6.0. Separate scripts were created for each database in order to adjust to each API’s data structure and query requirements.

Lezama-Nicolás et al. divided the emerging stage into two databases: Science Citation Index to study basic research (TRL 3) and INSPEC to study applied research (TRL 4-5). Scopus combines both basic research and applied research in their database of records, and for this reason we assigned the emerging stage in total (TRL 1-5) to Scopus. We chose Scopus over Science Citation Index and INSPEC due the availability of an API structure. Scopus data coverage includes 78 million items including records from journals, books and book series, conference proceedings and trade publications across 16 million author profiles and 70,000 institutional profiles since 1970 [31]. Scopus offers a RESTful API data structure that can be accessed using an API key and an HTTP GET request method. We accessed the academic subscription available through the University of Toronto Library for the full Scopus API access. However, a free API key can also be generated without a subscription but with limited access to data [32]. We used Python’s requests package to access the Scopus API. For a detailed review of the Python code used to collect data, please refer to Appendix B.

The BIMATEM analyzed patent records from the commercial software Patseer. For our analysis, we chose to analyze data from a publicly available patent source: the USPTO. The USPTO database includes full text patents issued from 1976 to present [33]. Collecting application patents over granted patents is a better reflection of the technology’s growth per given year since there exist a lag of 2-3 years between a patent filing and a granted patent [24].

We accessed the USPTO patent application data through PatentsView’s API, a patent data visualization and analysis platform that houses US patent data directly from the USPTO. The API is accessed through an HTTP GET request method where the query string can only be a single JSON object containing all the required parameters [34].The single JSON object requirement complicates the manual design of queries and could lead to a higher chance of human error. For this reason, we created an automated process for generating USPTO queries in

Python. The query formation code as well as the data collection script for the PatentsView API can be found in Appendix B.

We collected news and media records from Factiva’s database. Factiva offers coverage from 36,000 global news, editorially selected social media content, websites, and blogs with 35 years of archives [35]. Unlike Scopus and USPTO, Factiva’s data collection was not done using an API due to its unavailability. We collected news and media records directly from Factiva’s online search engine by downloading query search results as CSV files that are read, cleaned, and analyzed by Python. The detailed outline of the Python script used can also be found in Appendix B.

3.2.2 Technology Selection

The Soldier Systems Technology Roadmap (SSTR) is a capstone report and action plan created in 2011 that summarizes the needed advancement in Canadian soldier systems technologies. The report identifies the current state of soldier systems technologies by reporting assessments of their current maturity using the TRL scale, highlighting the key R&D priorities, and providing recommendations on an action plan to move technologies forward on the TRL scale. The SSTR project was led by partners from industry, academia, and government.

The roadmap document contains six technical domains: power and energy, weapons effect, C4I1, sensing, survivability/mobility/sustainability, and human and system integration. Each technical domain lists technologies with their TRL assessed in 2010-2011. The assessment took place through several workshops with over 1,550 participants from government, academia, companies and research organizations, both national and international [12]. The assessment was carried out without the use of automated or semi-automated TRL assessment tools. The results were all collected based on qualitative analysis (expert opinion).

The SSTR report was used as a source to validate the BIMATEM TRL assessments. Forty-five technologies from the SSTR report were chosen and are shown in Table 3.1. Sixteen technologies are emerging (TRL 1-5), six technologies are growing (TRL 6-7) and 23

1 Command, Control, Communications, Computer and Intelligence

18 technologies are mature (TRL 8-9). Twenty-two mature technologies (TRL 8-9) were used as a training set and 23 technologies (22 with TRLs ranging from 1-7 and 1 with TRL 9) as a test set. section 3.2.4 and 3.2.5 will cover details about the different calculations carried out for each.

Table 3.1: Technology list with assessed TRLs [12]. Dataset Type Technology SSTR TRL 3D Audio 5 Adaptive RF-filters 6 Brain Computer Interface 5 Cognitive Radio 5 Computer Vision 5 Connectors for E-textiles 3 Electro-textile Shielding 5 Electromechanical Energy Harvester 6 Li-air Battery 3 Test Set Li/Cfx Battery 7 Micro Reformation for Fuel Cells 5 Millimetric Radio Waveforms 4 Mobile Ad Hoc Networks (MANETs) 5 Nano-battery 1-2 Ni-Zn Battery 7 Optical LIDAR 7 PEM Fuel Cell 4 Piezoelectric Material 4 Pyroelectric Material 4 Short Wave Infrared (SWIR) Sensor 9 Software Defined Radio 7 Thermoelectric Material 4 Ultrasonic Ranging 5 Accelerometer 9 Actigraph Monitor 9 Automatic Speech Recognition 8 Blood Pressure Sensor 9 Bluetooth 9 Bone Conduction Hearing Aid 9 ECG Biosensor 9 Electromechanical Actuator 9 Gyroscope Sensor 9 Training Set Human Exoskeleton 9 Hydro Energy Harvesting 9 Li-ion Battery 9 Low Power Processors 9 Noise Cancelling Technology 9 SPO2 Biosensor 9

Solar Energy Harvesting 9 Speech Synthesis 8 UAV Autopilots 9 Ultra-wideband Technology 9 Wind Energy Harvesting 9 Z-wave 9 Zigbee 9

The 45 technologies shown in Table 3.1 were selected from every domain of the SSTR report (power and energy, C4I3, sensing, survivability/ mobility/ sustainability, and human and system integration) except the “weapon affect” domain due to conflict with personal ethical beliefs of the researcher.

Each section was scanned for usable technology names. Some technologies were not usable due to their vague titles. For example, titles like “Environmental Protection of E-textiles” were avoided due to their broad definition. Environmental protection does not convey the specific method or technology used and so creating a query to accurately capture “Environmental Protection of E-textiles” is not possible without additional information.

Technologies that were assessed a large range of TRLs in the SSTR report were also excluded. For example, “Magnetic Wave Communications” was assessed as TRL 6-9. This range is too large to include since the technology could either be a growing technology or a mature technology.

The technology names outlined in Table 3.1 are shown as they appear in the SSTR report and were used to design search queries. The following section will describe the steps used to form queries for each technology title.

3.2.3 Forming Search Queries

Previous literature, including Lezama-Nicolás et al.’s work, relied heavily on the use of expert opinion to create and tailor search queries for bibliometric research (more on this in section 4.6). The use of expert opinion however makes query design near impossible to reproduce and challenging for researchers without depth of expertise in the field of that technology. There are

3 Command, Control, Communications, Computer and Intelligence

20 currently no methods in the literature that provide a uniform way of creating trusted queries. This thesis offers steps that standardized the query formation process specifically for this research.

Each search query was formed by following a repeatable method inspired by concept mapping. Concept mapping was suggested to us by the University of Toronto’s Library experts and is also documented on their online “Searching the Literature Guide” [36]. Concept mapping allows for a repeatable and reproducible method of forming queries. The method consists of first identifying the different concepts in a research question and then breaking down each concept to brainstorm the different ways it can be described. We found that most of our technologies were single concept as they are titles, but with a few exceptions.

To decide whether a technology was a single concept, we searched each technology title in Google’s search engine. From the search results, we mainly analyzed encyclopedias and scientific or technical pages. A technology was considered to have multiple concepts if none of the search results returned its exact title. For example, searching “speech synthesis” returns results with the exact technology title and its related synonyms (i.e. Text to speech) (Table 3.2). However, when searching for “millimetric radio waveforms” no results return this exact title. The technical and scientific search results show that the technology is often referred to in a combination of ways: “extremely high frequency radio waves”, “radio frequency signals in the millimeter band”, etc. For this reason, this technology was separated into two concepts to accommodate the different ways to which it can be referred (Table 3.3).

Table 3.2: Query concept mapping for “Speech synthesis”

A B C

Concept and “speech synthesis” N/A N/A Synonym OR “text-to-speech” OR “text to speech”

Table 3.3: Query concept mapping for “Millimetric radio waveforms”

A B C

Concept and “extremely high “radio wave*” OR N/A Synonym frequency” or “radio frequenc*” “millimet*”

We selected synonyms through multiple revisions based on reviewing Google search results for other relevant keywords that can also describe the technology. For example, “brain computer interface” Google search results showed that the technology can also be referred to as "brain- computer interface", "neural-control interface", "mind-machine interface", "direct neural interface", or "brain-machine interface”.

Wildcards were used to account for the various ways terms can be referred to. For example, “millimetric” can also be referred to as millimeter. Using a wildcard (millemet*) includes both variations of the term (Table 3.3).

Each database had its own unique rules for forming queries. For example, the USPTO does not include plurals of terms automatically, they must be added, while Scopus and Factiva automatically include plurals of terms. These rules can all be reviewed at each database’s query formation page [37],[34],[38]. Once the concept mapping table is created (Tables 3.2 and 3.3), the synonyms for each concept are combined with an “OR” Boolean operator. If multiple concepts exist, they are combined with the “AND” Boolean operator (Table 3.4).

Table 3.4: Query inclusion operators

Operator Type Operator Example

Inclusion operator AND A AND B A B

OR A OR B A B

We avoided the use of proximity operators (i.e. NEAR #) for the purpose of making the query formation method as close to automated as possible. The use of proximity operators would require us to select the average length of words between each term and we found that to be outside the intended scope of making the query formation method reproducible and without subjective input.

A list of all the queries used for each of the 45 technologies can be found in Appendix A.

3.2.4 Calculating S values

The BIMATEM framework’s statistical calculations were all performed with Minitab. However, we decided to use Python for our analysis. The curve fit process was carried out using Python’s SciPy package. SciPy offers multiple sub packages for engineering, science, and mathematics. We used SciPy’s optimize package for curve fitting bibliometric records to the logistic curve (S-curve) model (1) where k is the upper limit of growth, a and b are the initial stage of diffusion and velocity of diffusion respectively.

푘 (1) 푟(푡) = 1 + 푎푒−푏(푡−푡0)

Lezama-Nicolás et al.’s BIMATEM uses the logistic growth curve (S-curve) to fit scientific and engineering publications and patents, and the hype curve to fit news and media records. However, there currently exists no mathematical modeling or validation of the hype cycle, as we will explain in section 4.3 of this thesis. We found the mathematical model used by the BIMATEM paper to be inaccurate in representing the news records of our selected technologies. More specifically, the paper that the BIMATEM method bases the mathematical model from fit two types of records to different parts of the hype curve: scientific and engineering publications were fitted to the bell- shaped curve and patents to the S-curve [30]. The model does not present a mathematical equation of the hype curve, but rather represents the different elements that make up the hype curve, the sum of a bell-shaped curve and an S-curve. We found the model used to be inaccurate when we performed non-linear regressions and for this reason, we chose to analyze news and media data using S-curves as well.

The SciPy optimize package can be used for both linear and non-linear regression problems. Because we are optimizing a non-linear model, we used the curve_fit function of the optimize package. We selected the Trust Region Reflective algorithm with initial guess values for each of the model’s parameters to 0.5 with a constraint of (0, ∞) and (-∞, ∞) for a and b respectively.

The Standard error of regression (S) is the average distance that the observed values fall from the regression line. In other words, the S value tells us how good the technology records fit the logistic curve model. A higher S value means a worse fit and a lower S value means a better fit.

The S value was calculated using Python’s Sklearn metrics package. We used the package to calculate the mean squared error value (MSE) and calculated the S value by finding the square root of MSE. Equation 3 below shows the mathematics used to find the S value where n is the total number of terms, 푦푖 it the observed value of the variable, and 푦̅푖 is the predicted value of the variable.

1 (3) √ ∑푛 (푦 − 푦̅ )2 푛 푖=1 푖 푖

Non-linear regression was carried out for all 45 technologies (training set and test set). For the training set, the collected S values were used to calculate the acceptance threshold for S value (ATS) which will be discussed in the following section. Figure 3-1 shows an overview of the process followed to collect the S values for the test set to estimate the TRL of each technology (section 3.2.6).

3.2.5 Calculating ATS Values

We calculated ATS values with Python using the technology records of the training set (22 mature TRL 8-9 technologies). The BIMATEM suggests that each database (i.e. TLC stage) can have a threshold value that can be used to determine whether a technology has passed that stage. The authors used 10 technologies to calculate the ATS value, however, their mature technologies were evaluated in 2018 and so we believe they may not be an appropriate indicator of mature technologies in 2011. Because our test set was TRL assessed in 2011, the technologies used to calculate the threshold values (ATS) must be mature technologies in 2011. For this reason, we chose to use TRL 8-9 technologies from the SSTR report as our training set.

After collecting the standard error of regression (S) for each technology (section 3.2.4), Grubbs outlier test was applied to each S value using Python’s smirnov_grubbs library with an alpha value of 0.5. There were no outliers found.

The ATS values were calculated using the upper bound 95% prediction interval calculation shown in (4) where Xn is the mean, Sn is the standard deviation, and Ta is the 100(1-p/2)th percentile of Student’s t-distribution with n-1 degrees of freedom.

1 (4) 푋̅ + 푇 푆 √1 + ( ) 푛 푎 푛 푛

The S values for each mature technology along with the calculated ATS values can be found in Table 3.5 below.

Table 3.5: The S values for the training set and the resulting ATS value for each database Technology Scopus (TRL 1-5) USPTO (TRL 6-7) Factiva (TRL 8-9) Standard error of regression (S) value Accelerometer 5.18 3.66 5.55 Actigraph Monitor 5.12 1.02 3.5 Automatic Speech Recognition 3.8 2.12 1.95 Blood Pressure Sensor 7.02 4.78 4.42 Bluetooth 2.81 4.02 3.39 Bone Conduction Hearing Aid 5.57 7.63 3.73 ECG Biosensor 3.97 4.88 3.47 Electromechanical Actuator 4.8 4.7 4.98 Gyroscope Sensor 5.86 3.24 3.92 Human Exoskeleton 5.73 9.36 4.43 Hydro Energy Harvesting 8.61 6.2 4.6 Li-ion Battery 4.25 4.6 5.59 Low Power Processors 3.42 3.69 3.66 Noise Cancelling Technology 3.46 5.65 5.9 SPO2 Biosensor 7.35 8.66 4.32 Solar Energy Harvesting 7.19 3.13 3.7 Speech Synthesis 4.52 5.56 2.26 UAV Autopilots 2.22 2.17 2.86 Ultra-wideband Technology 2.35 6.92 1.52 Wind Energy Harvesting 6.09 3.4 3.38 Z-wave 6.46 2.99 3.63 Zigbee 4.23 no patents found 3.81 S Value Average 5.0 4.68 3.84 Acceptance Threshold for S 8.62 9.29 6.23 value (ATS)

Figure 3-2 below is an overview of the data collection method used to calculate the ATS values. These values are used as an input in the TRL classification process as was outlined in Figure 3-1.

Figure 3-2 Flowchart showing an overview of the data collection and calculation process for the training set.

3.2.6 TRL Estimation

The TRL estimation process is summarized in Figure 3-3. The pre-calculated ATS values (section 3.2.5) for each database were used as a threshold to check whether the technology has passed the specific life cycle stage: emerging (TRL 1-5), growing (TRL 6-7), and mature (TRL 8-9). The BIMATEM explains that the threshold value calculated for each database acts as an indicator for its respective TLC stage. For example, the ATS for Scopus is used as a threshold to determine whether the technology has passed or is still at the emerging stage.

The S value for each technology determines how good the fit of the records is to the logistic growth curve (S-curve). If an S value is less than the calculated ATS value for the database (i.e. TLC stage), this indicates that there is a lower error between the data and the S-curve, and thus, the authors suggest that the technology has possibly passed that stage but should first be evaluated with the ATS of the next stage, using the S value for that database. On the other hand, if the S value is greater than the ATS value for that database, there is a larger deviation between the S-curve and the technology’s record which is an indication that the technology has not

27 matured through a full S-curve and has not passed that stage. This TRL estimation method was automated in Python and the description of the code is provided in Appendix B.

Figure 3-3: Flowchart describing TRL estimation through comparing standard error of regression values (S) to the absolute standard error of S value (ATS) to determine the technology life cycle stage.

3.3 Results and Discussion

The TRL of each technology from the test set was estimated using Python with the method described in the previous section. Figure 3-4 below summarizes the TRL assignment process and Table 3.6 shows the resulting estimates.

Figure 3-4: TRL assignment using the ATS values calculated for each database using the training set.

Table 3.6: Technologies with estimated bibliometric TRL and expert assessed TRL Technology Estimated Expert Assessed Match Bibliometric TRL via SSTR TRL BIMATEM 3D Audio 8-9 5 No Adaptive RF-filters 8-9 6 No Brain Computer Interface 8-9 5 No Cognitive Radio 8-9 5 No Computer Vision 8-9 5 No Connectors for E-textiles N/A 3 N/A Electro-textile Shielding 6-7 5 No Electromechanical Energy Harvester 1-2 6 No Li-air Battery 8-9 3 No Li/Cfx Battery 1-2 7 No Micro Reformation for Fuel Cells 8-9 5 No Millimetric Radio Waveforms 8-9 4 No Mobile Ad Hoc Networks (MANETs) 8-9 5 No Nano-battery 3-5 1-2 No Ni-Zn Battery 8-9 7 No Optical LIDAR 8-9 7 No PEM Fuel Cell 8-9 4 No Piezoelectric Material 8-9 4 No Pyroelectric Material N/A 4 N/A Short Wave Infrared (SWIR) Sensor 1-2 9 No Software Defined Radio 8-9 7 No Thermoelectric Material 8-9 4 No Ultrasonic Ranging 8-9 5 No

As shown in Table 3.6, the BIMATEM did not result in a TRL estimate for Connectors for E- textiles or Pyroelectric Material. The logistic growth curve (S-curve) could not be fit to the Factiva results for Connectors for E-textiles because the technology only returned two records. The logistic growth curve could also not be fit to the USPTO results because the technology only returned one record. For this reason, these two technologies received no TRL estimate.

The results summarized in Table 3.6 show that none of the 23 technologies’ bibliometric TRL estimates matched their expertly assigned SSTR TRL assessments. Interestingly, about 70 percent of the test set was estimated to be at a TRL 8-9. From Figure 3-4, we can see that TRL 8- 9 is the last assessment stage of the BIMATEM. This means that 70 percent of technologies had S values less than or equal to the ATS values of every database. In other words, the fit errors of

29 the test set technologies were lower than the training set (mature) technologies, meaning their data more closely followed an S-curve trajectory and thus considered more mature in the BIMATEM technique.

Table 3.7 summarizes the mean, standard deviation and prediction interval for each S value for every database. The ATS values of Scopus, USPTO, and Factiva were 8.62, 9.29, and 6.22 respectively. However, as seen in Table 3.7, the prediction interval for each of the databases is large, as is also reflected in the standard deviations. For example, the USPTO’s recorded S values have a standard deviation of 2.16. This explains the large prediction interval where the lower bound is at 0.08 while the upper bound is recorded to be 9.29.

Table 3.7: Standard deviation and prediction interval for training set’s S values Scopus S Values USPTO S Values Factiva S Values Mean 5.0 4.68 3.84 Standard Deviation 1.70 2.16 1.12 95% Prediction Interval (1.38, 8.62) (0.08, 9.29) (1.47, 6.22)

The large standard deviations of the S values of the USPTO database can be further explained by studying the logistic curve fits of the patent application records for some technologies. Figure 3- 5 shows the plots for six technologies from the training set. Upon visual inspection, the real data scatter plots deviate from the S-curve shape which is characterized as a concave curve followed by a convex curve with an inflection point as a transition.

Figure 3-5: Mature (TRL 8-9) patent application records logistic fits

Not only do we fail to see bibliometric evidence of a classic S-curve shape, but we see large deviations in the S-curve error values across the technologies, even when experts assess them to be equally mature. According to the logic of the BIMATEM model, we would expect all the record patterns for the mature technologies in Figure 3-5 to have low error when fit with a logistic growth curve, and therefore to look like an S-curve. Instead, we see increasing, but not S-curve, trends. These findings indicate that it is inaccurate to assume all mature technologies follow a similar technology trend that can be used to create a universal threshold for passing technology life cycle stages. As we see from both the data and plots above, technologies tend to vary significantly in the way they behave bibliometrically as they mature. This challenge as well as other TRL bibliometric automation challenges will be discussed in chapter 4.

Gaps in Automating Technology Readiness Level Assessments Using Bibliometrics

In chapter 3, we estimated the TRLs of 23 technologies from the 2011 SSTR report using a bibliometric method developed by Lezama-Nicolás et al. [15]. The goal was to address whether the use of bibliometric records is an appropriate and accurate TRL indicator. The results showed that the bibliometric method did not produce TRLs that matched the assessments of the SSTR report. This discrepancy can be attributed to several reasons that explain why bibliometrics is not an accurate estimator of maturity and therefore rendering it challenging in the use for the automation of Technology Readiness Level assessments. These challenges will be identified and explored in this chapter.

4.1 Flawed Mapping of Bibliometric Indicators to Technology Readiness Levels

The Watts and Porter model proposed in 1997 suggests that bibliometric indicators can be used to reveal the life cycle stage (i.e. maturity level) of a technology. The model is summarized in Table 4.1 and Figure 4-1. The model suggests that the number of records from the respective bibliometric source is expected to rise to a peak as the technology matures through the TLC stage, and then decrease signaling the start of the next TLC stage. By collecting technology records from each bibliometric indicator source, the assumption is that technology life cycle stages can be estimated by monitoring the rise and fall of the number of technology records.

Table 4.1: Technology life cycle stage, the equivalent Technology Readiness Level, R&D stage and bibliometric indicator (adapted from [10], [15], [25]).

TLC Stage Technology Readiness Level R&D Stage Bibliometric Indicator

Emerging4 TRL 3 Basic Research Science publications

TRL 4-5 Applied Research Engineering publications

Growing TRL 6-7 Development Patents

Mature TRL 8-9 Application News and media

Figure 4-1: The Watts and Porter bibliometric indicator model (adapted from [10]). Note that the magnitude of peaks is irrelevant.

4 A reminder that TRL 1 and TRL 2 technologies are also in the emerging life cycle stage, however they cannot be linked to a scientific publication database because most journals require proof of concept in order to be published [15]

The pairing of bibliometric indicators to life cycle stages has been used in many previous technology forecasting and maturity assessment research [24],[39],[40]. The literature suggests that as technologies advance through their TLC (and TRLs), their bibliometric records would show clear patterns of this maturity change. As technologies progress from being emerging technologies, to growing technologies, and finally to mature technologies, their record activity is expected to manifest in scientific and engineering publications, patents, and news and media in this order over time respectively [25].

Understanding which technology record databases give insightful input about the maturity status of a technology is crucial before attempting to automate the Technology Readiness Level assessment using bibliometrics. Our previous paper titled “Identifying Gaps in Automating the Assessment of Technology Readiness Levels” addressed the following research question: is the literature correct in pairing specific datasets to specific levels of maturity? [41]. We investigated the technology record activity of 15 emerging (TRL 1-5) nanotechnologies in scientific and engineering publications, patents, and news and media databases. The results showed that some emerging technologies exhibit significant patent activity which contradicts the pre-existing literature’s assumption that emerging technologies exhibit the most activity in scientific and engineering publications.

We found similar patterns to our previous paper in the bibliometric data collected for the SSTR technologies. Figure 4-2 shows the expected bibliometric trend for each technology life cycle stage according to Watts and Porter, and the number of scientific and engineering publications (Scopus), patent applications (USPTO), and news and media (Factiva) records collected for three SSTR technologies from 1976 to 2011. A reminder that in this thesis, we combined the bibliometric indicators for basic research and applied research of the emerging TLC stage into one indicator by collecting records for both R&D stages from Scopus.

(a) (b)

As illustrated in Figure 4-2, we would expect an emerging technology, or TRL 1-5, to be undergoing active research which would be reflected by an increase in its scientific and engineering publications, possibly some minor patent activity as researchers file their findings, and insignificant news and media records as the technology is far from commercialization. Surprisingly, Nano-battery showed significantly higher number of news and media records compared to scientific and engineering publications and patent applications at only TRL 1-2.

A growing technology, or TRL 6-7, is undergoing prototype development and testing. According to Watts and Porter, scientific and engineering publications are expected to decrease as more patents are being filed. We would also expect to see minor news and media activity because the technology is yet to be demonstrated in its final configuration. However, LIDAR, a TRL 7 technology, shows an increasing trend in scientific and engineering publications and a similar trend in news and media activity. Both bibliometric activities are significantly higher than patents.

Finally, a growing technology, or a TRL 8-9, is in its final technical configuration and is expected to show an increase in its news and media records, a decrease in its patent applications as no new developments are made, and insignificant scientific and engineering publications. Contrary to expectations, Z-wave, a TRL 9 technology, shows an increase in its patent applications, and although we see an expected increase in news and media records, it was significantly lower than patent activity.

These findings corroborate the conclusions made in our previous work that pairing bibliometric indicators to specific Technology Readiness Levels is inaccurate. The way technologies are represented in bibliometric databases is complex as they depend on multiple inputs. Funding, bias, and industry size are all factors that can influence the magnitude and distribution of technology records in bibliometric databases. Therefore, we should re-think the model on which we base bibliometric analysis of maturity, beyond Watts and Porter’s original model.

A further analysis of this would be to find the years at which each a technology has achieved a new Technology Readiness Level (i.e. TLC stage) and monitor how bibliometric records change over time.

4.2 The Biased Public Perception Around Technologies

As discussed in the section above, the pairing of technology maturity stages to specific technology record databases is applied in many technology forecasting and assessment research. According to Watts and Porter, news and media reporting are expected to peak only when technologies are mature and closer to becoming market ready. However, what this theory excludes is the biased public perception and reporting around some technologies.

For example, Figure 4-3 shows the expected bibliometric trend, according to the Watts and Porter model, for an emerging technology and the plots of scientific and engineering publications, patent applications, and news and media records collected from 1976 to 2011 for two SSTR technologies from the power and energy sector: Proton exchange membrane (PEM) fuel cell, and Piezoelectric energy harvesting.

(a) (b) (c)

The bibliometric model suggests that technologies in the emerging TLC stage (TRL 1-5) are undergoing early R&D and so will have an increasing trend in the number of scientific and engineering publications. We would expect to see little to no activity for patents as the technology is yet to have a prototype developed and tested. We would also expect to see no news and media activity at this early stage of maturity. Piezoelectric energy harvesting showed more conforming behavior than PEM fuel cells. As expected, we see that both Piezoelectric energy harvesting and PEM fuel cells have a continuous rising trend in scientific and engineering publications, but what was surprising was the increasing trend in news and media activity of PEM fuel cells. One possible explanation for this phenomenon can be attributed to the bias around fuel cells. With the increasing concern for climate change and the depletion of fossil fuels, the research for allocating environmentally safe energy sources is continuously on the rise. For example, the Office of Energy Efficiency and Renewable Energy in the United States dedicated 63.2 million dollars in funding to support hydrogen and fuel cell technology research and development projects in 2010. The increased need for clean technologies drives the high levels of public discussion around fuel cells, evidenced in its news and media record activity.

This is especially noteworthy when compared to technologies that may have less interest around them, like Piezoelectric energy harvesting.

Public bias can also be seen in other technologies like self-driving cars. Self-driving can operate without any human input or without the presence of a human altogether [42]. Google (Waymo), Tesla, Apple, and BMW are among the companies currently developing self-driving cars. There are no commercially available self-driving cars yet as they are still undergoing road testing which, according to the Technology Readiness Level definitions, places self-driving cars at a TRL 6-7 [2], [43].

Self-driving cars are often pitched to the public as having the ability to eliminate vehicle accidents and save millions of lives [44],[45],[46]. There is no doubt that a technology like self- driving cars would change the world as we know it, which is why it has gained massive news & media coverage (Figure 4-4) specifically between 2010 and 2020, coinciding with Google’s Waymo project launch in 2009 [47]. As seen in Figure 4-4, the record activity of self-driving cars in news and media is significantly higher than scientific and engineering publication and patent application activity, contrary to what is proposed by Watts and Porter. Technologies like self-driving cars promise a safer future and consequently attract more bias and buzz from the public. (a) (b)

The bibliometric indicator model suggests that technologies only start to demonstrate significant news and media activity as they reach TRL 8-9 (the mature TLC stage) and that technologies of

38 lower TRLs are not usually publicized or discussed outside the scope of publications and patents because they have not been demonstrated to be consumer ready. However, what we see from the examples covered in this section, and what we know to be true based on our own interactions with mainstream news, is that reporting on technologies can sometimes start very early in a technology’s life cycle. The Watts and Porter model does not account for bias and it could be hypothesized that it is only applicable to technologies that are not prone to public interest. These findings further support our conclusion from section 4.1 that we should re-think the model on which we base bibliometric analysis of maturity, beyond Watts and Porter.

4.3 Inaccurate Technology Trends

The technology forecasting and assessment literature uses technology trends to model and forecast technologies as they mature through their life cycles. Two popular technology trends are Gartner’s hype cycle, and the S-curve (logistic growth curve). Previous research has used the hype cycle [48],[49],[50] and the logistic growth curve [20], [51]–[54] to predict technology behavior or maturity status using bibliometric records. But we ask, how accurate are these models in representing a technology’s bibliometric behavior? And can we confidently continue using these trends to make important decisions about technology maturity? In this section, we address both questions by presenting other research’s critiques of the models and real-life examples of technologies.

4.3.1 The Hype Cycle

The hype cycle was first developed in 1995 by Gartner, a technology research and consulting firm and is currently used to model the maturity of emerging technologies over time. Gartner claims that the hype cycle adds another dimension to the S-curve by including human attitudes towards technology, or as they eventually called it “hype”. Figure 4-5 shows how the hype cycle is a result of adding the classic S-curve often used to model technology maturity, and a bell- shaped curve that represents the hype of human attitude towards technologies. The bell-shaped curve represents Gartner’s Innovation Trigger and Peak of Inflated Expectations phases while the S-curve represents Gartner’s Trough of Disillusionment, Slope of Enlightenment and Plateau of Productivity phases (Figure 4-6).

Figure 4-5: Hype curve is the sum of Hype level and S-curves [29].

Figure 4-6: Gartner’s hype cycle and its five phases [29].

Although the hype cycle is widely used for making important investment decisions in industry, Gartner fails to provide empirical or theoretical motivation for the cycle. Gartner claims that the mathematical representation of its hype cycle is the summation of a bell-shaped curve and an S- curve. However, although both curves share the same independent variable (x-axis/time), their dependent variable (y-axis) is different. The S-curve’s dependent variable is the cumulative sum of technology records while the dependent variable of the bell-shaped curve, or the hype of human attitude towards technologies as Gartner describes it, remains unclear. Gartner uses “expectations” and “visibility” interchangeably when labeling their y-axis. They explain

40 expectations as “expected future value of an innovation” and visibility as “technology presence rate on media channels, conferences as well as in interpersonal conversations”, but as Dedehayir and Steinert explain, both of these variables lack an operational definition and solid quantification [29]. Dedehayir and Steinert’s paper also pointed to the lack of consensus in previous literature work about whether technologies empirically show hype cycle conforming behavior. The authors reported a misalignment between how technologies are placed on the hype cycle and the results obtained from empirical analysis of news publication counts of technologies. When investigating our own data for this thesis, we studied whether the news and media data from Factiva exhibit hype cycle conforming behavior as suggested in the methodology of Lezama-Nicolás et al. [15]. Figure 4-7 shows the plot for nine mature technologies. Four of the technologies are TRL 9 assessed by the SSTR in 2011 and four technologies were described as mature technologies by Lezama-Nicolás et al. in 2018. Each technology’s non-cumulative news and media records from Factiva were plotted from 1976 to 2020, our assumption being that if a technology was mature in 2011 and 2018, it would remain so now.

Figure 4-7: News and media record plots for mature (TRL 9) technologies collected from 1976 to 2020.

By inspection, we see that none of the technologies in Figure 4-7 exhibited a hype cycle conforming behavior as outlined by Gartner. Some technologies, like the Accelerometer, exhibited a continuous increase over time while others, like Speech Synthesis, exhibited multiple peaks and troughs. Therefore, it can be hypothesized that Gartner’s hype cycle is not an accurate estimate of news and media records for all technologies. It is possible that some technologies exhibit this behavior, but without any experimental verification of the hype cycle, it is difficult to empirically estimate its accuracy in modeling news and media activity. No previous scientific work has been able to present an accurate mathematical model for the hype cycle and until we are presented with one, its validity remains questionable.

4.3.2 The Logistic Growth Curve The Technology Life cycle stages are often modeled using the logistic growth curve (S-curve) [26]. The logistic equation models the law of natural growth which can often be simplified to periods of birth, growth, maturity, decline and death of any system. These sets of periods are often referred to as the life cycle of a system (Figure 4-8). The logistic growth curve has previously been used to study scientific publication and patent data for technologies through curve fitting techniques [20], [51]–[54]. The S-curve is used to plot cumulative data and is characterized by a convex curve and a concave curve with an inflection point midway (tm) as seen in Figure 4-8.

The logistic growth curve continues to be used for technology forecasting and maturity assessment literature with very limited criticism. Magee et al. present one of the few studies to report patent activity following exponential growth and not S-curve conforming behavior [55]. Figures 4-9 and 4-10 show nine plots of cumulative scientific and engineering publication and application patent activity for mature (TRL 8-9) technologies from 1976 to 2020. None of the technology plots appeared unequivocally like an S-curve. They also showed no distinct inflection points after which the shape of the cumulative growth takes a convex form towards plateauing. Patent application cumulative data for some technologies (Bluetooth, Accelerometer, Electromechanical actuator, Lithium-ion battery, and Automatic speech recognition) showed a slight plateau, however this cannot be confirmed until we collect data for consecutive years ahead. The lack of the plateau effect in any of the technologies is noteworthy. After reaching maturity, the rate of growth of bibliometric records is expected to decline, translating to a plateau of cumulative bibliometric records over time [27]. However, this is not observed in our data.

We hypothesise that the lack of the plateau effect is due to mature technologies continuously being used in the development of new emerging technologies. For example, our search of references to Bluetooth in Scopus today in 2020 reveals the following headlines: Bluetooth in COVID-19 contact tracing apps [56], Bluetooth in emerging wearable technologies [57], Bluetooth in Internet of Things applications [58]. Mature technologies like Bluetooth play an integral role in many of our everyday technologies. The discussion of Bluetooth in scientific publications and patents will continue to rise as long as the technology can be used to connect emerging technologies to existing systems.

The lack of clear S-curve features in any of the technologies plotted in Figures 4-9 and 4-10 should be further explored. A future direction of this work could potentially investigate methods to study the shapes of logistic growth curves for different technologies to recognize possible patterns. In addition, some technology forecasting and assessment research have used another variation of the logistic growth curve called the bi-logistic growth model[52], [54]. The bi- logistic growth model was developed by Rockerfeller University and suggests that some

44 technologies go through multiple “pulses” or logistic growth curves in their lifetime [59]. However, special techniques and software must be applied to determine if technologies follow this behavior.

4.4 Maturity is dependent on Technology Application and Context

In their paper, Lezama-Nicolás et al. chose the following mature technologies for their training set: Cloud computing, Datamining, Location Aware Technology, MEMS, Organic Light Emitting Diodes, Radio-frequency Identification, Smartphone, Speech Recognition, Text to Speech, and Wireless area connection [15]. However, definitively classifying these technologies as mature is challenging. Technologies have various applications, each with varying levels of maturity. For example, Cloud Computing has reached maturity as a platform for cloud storage but is still an emerging technology when discussing its role in Internet of Things [60]. Location aware technologies is a broad term that encompasses several types of technologies each with unique maturity levels. For example, GPS and Wi-Fi are TRL 9 technologies as they have successfully demonstrated their ability in their operational environment and are currently commercially available [2]. However, current research at MIT is developing a location aware technology that uses low power wireless signals to locate tumors for targeted cancer treatments [61]. This type of a location aware technology is still in its research phase (TRL 1-5) and cannot be labeled as mature.

Let’s look at another example: Computer Vision (CV) is an application of Artificial Intelligence. The SSTR report assessed Computer Vision at a TRL 5 without specifying an application. However, this assessment is questionable because the maturity of CV is very dependent on its application as shown in Figure 4-11. For example, CV can be used in applications like facial recognition and text recognition. Today, your iPhone unlocks with the face ID feature, and Google Photos lets you search by things, people, and places, both features use facial recognition or CV. However, facial recognition in CCTV systems is still undergoing testing. CV text recognition also has varying levels of maturity. For example, Optical Character Recognition systems are a feature of many Portable Document Format (PDF) software like Adobe Acrobat. However, CV text recognition is still in its early maturity phase for deciphering non-human readable media like ancient tablets.

In section 4.2, we discussed how public bias around self-driving cars complicates estimating its maturity using bibliometric indicators. One other complication of automating technology assessments for technologies like self-driving cars is the ambiguity behind the term. SAE international defined 6 levels of automation grouped into two categories [63]:

The human monitors the driving environment: • Level 0: the driver controls the vehicle manually without any automated aid • Level 1: the vehicle provides some automated assistance like monitoring speed through cruise control • Level 2: the vehicle can perform steering and acceleration, but the driver monitors all tasks and can take over anytime The automated system monitors the driving environment: • Level 3: the vehicle can perform most driving tasks, but human override is still required • Level 4: the vehicle performs all driving tasks under specific circumstances (requires geofencing) and human override is optional • Level 5: the vehicle performs all driving tasks under all circumstances and zero human attention or interaction is required

SAE international in 2014 reported that media and legislative references to “self-driving cars” often includes some or all 6 levels of automation [64]. Tesla’s current Autopilot technology is really only a level 2 automation (TRL 9), while Google’s Waymo is at a Level 4 automation (TRL 5) [62]. However, both companies use the term “self-driving” to describe their technologies. The following quote is taken directly from Tesla’s website:

“Autopilot and Full Self-Driving Capability are intended for use with a fully attentive driver, who has their hands on the wheel and is prepared to take over at any moment. While these features are designed to become more capable over time, the currently enabled features do not make the vehicle autonomous.” [65].

Figure 4-12 shows the European Commission’s TRL assessment of all 6 SAE automation levels.

Figure 4-12: Self-Driving Car TRL according to the 5 levels of SAE assessed in 2020 (adapted from [62]).

As seen in Figure 4-12, the European Commission in 2020 also used the term “self-driving cars” to refer to all levels of automation. An ambiguous term complicates the bibliometric assessment of Technology Readiness Levels. Each driving automation level is assessed at a unique TRL. Collecting bibliometric records for “self-driving cars” would not return an accurate reflection of the true maturity status of the technology.

Without including application or context, technology maturity becomes difficult to estimate. Previous studies in the technology assessment and forecasting literature have excluded

47 application from their technology scopes. Technology Readiness Levels describe very different life cycle stages and as we demonstrated through a few examples, they can change dramatically depending on the specified scope of the technology.

4.5 Unclear Benchmarking of Maturity

Understanding how mature technologies (TRL 8-9) behave in technology record databases is crucial for automating the Technology Readiness Levels using bibliometrics. Mature technologies can act as a benchmark to compare and assess technologies of lower maturity. Lezama-Nicolás et al. used a dataset consisting of 10 mature technologies as their training set to determine the benchmark value that can be used to determine if other technologies meet this value and are indeed “mature” [15]. Determining which mature technologies to use in a training set becomes a crucial step in an automated assessment method. If the training set is not accurate or is not truly representative of mature technologies, the method’s accuracy becomes debatable.

In chapter 3, we listed the mature technologies we used to assess the accuracy of Lezama- Nicolás et al.’s method in estimating Technology Readiness Levels using bibliometrics. We used 22 TRL 8-9 technologies assessed by the SSTR report in 2011. To investigate the accuracy of the assessment of these mature technologies, we created an expert interview through a questionnaire that was approved by the University of Toronto’s Research Ethics Board.

With the questionnaire, we asked the expert participant to agree or disagree with the SSTR assessment of 21 TRL 9 technologies (Table 4.2). The participant was given the definition of a TRL 9 technology and the definition of a mature technology:

• A technology is at a TRL 9 if the actual system was proven in an operational environment. A TRL 9 is often used to describe a mature technology. • A mature technology is defined in the technology life cycle model as a technology that has been sufficiently developed to meet its required performance and is well understood and fully controlled

The questionnaire was given to an expert in the field of assessing technology maturity through their professional line of work and was asked to provide reasoning for when they disagreed with the TRL assessment of the SSTR report.

Table 4.2 lists the results of the survey. The expert disagreed with the technology readiness assessment of solar energy harvesting, wind energy harvesting, and hydro energy harvesting. The following reasoning was given for each of the technologies:

“as far as I know, I can't buy consumer devices yet that have solar energy harvesting integrated.” - Solar energy harvesting TRL assessment response

“unlike big wind turbines, small-scale wind energy harvesting is not an off-the-shelf product yet, as far as I know”- Wind energy harvesting TRL assessment response

“still rather experimental, to my knowledge”- Hydro energy harvesting TRL assessment response

The expert also left the following response under the assessment of low power processors:

“I found "low power processor" more challenging to assess. While there are low power processors in the market, there are none so far for applications that currently still use very high-power processors (e.g. GPUs for visualization or machine learning). So I would only give them a "partial TRL 9" assessment.”- Low power processors TRL assessment response

Table 4. 2: Survey response for agreeing (yes) /disagreeing (no) with the TRL 9 assessment of 21 SSTR technologies

Technology Survey Response: Is the Technology Mature?

Solar Energy Harvesting No

Bluetooth Yes

Accelerometer Yes

ECG Biosensor Yes

Noise Cancelling Technology Yes

Electromechanical Actuator Yes

Low Power Processors Yes (partially)

Li-ion Battery Yes

Bone Conduction Hearing Aid Yes

Gyroscope Sensor Yes

Actigraph Monitor Yes

Zigbee Yes

Wind Energy Harvesting No

SpO2 Biosensor Yes

Human Exoskeleton Yes

Short Wave Infrared Sensor Yes

UAV Autopilots Yes

Z-wave Yes

Hydro Energy Harvesting No

Blood Pressure Sensor Yes

Ultra-wideband Technology Yes

The questionnaire responses revealed that our expert assessed the maturity of technologies based on their market availability. However, it is important to note that commercially available technologies and TRL 9 technologies are not necessarily the same. The Technology Readiness Levels were not created for the intention of assessing market readiness, but rather assess technological capabilities. In fact, previous research created market readiness levels as an extension to Technology Readiness Levels [66]. The market readiness levels were used as part of

50 a framework that would demonstrate the overall process it takes technologies to move from achieving TRL 9, to getting the needed regulatory approvals, and finally becoming market ready.

Technology Readiness Levels alone cannot be used to describe the market readiness of technologies. The lack of consensus about what makes a technology “mature” is also a challenge of automating Technology Readiness Level assessments. Is defining maturity the subject of market-readiness or technical-readiness?

A future direction of this work would be to distribute the questionnaire to other experts, including researchers, and compare the responses of participants based on their line of work and expertise.

4.6 Forming Queries Without Expert Opinion

One main step of automating Technology Readiness Level assessments is the design of a search query. The search query should be designed to maximize the relevant bibliometric records returned on a technology. This is particularly challenging since there usually exists multiple variations of a single technology name. For example, “Brain computer interface” can also be referred to as “neural-control interface”, “mind-machine interface”, “direct neural interface” or “brain-machine interface”. Finding reliable sources for technology synonyms is important.

In addition, the search method can impact the quality of bibliometric records returned. For example, searching in all-text versus the highlight section (title, abstract, and keywords) can return more noise. Figure 4-13 shows the tradeoff between precise searches (i.e. searching in title, abstract, and keywords) and sensitive searches (i.e. searching in all-text). Although a precise search can avoid noise in comparison to a sensitive search, it can miss some relevant publications and records about the technology. Alternatively, a sensitive search will return most of the relevant technology records but with the tradeoff of capturing many irrelevant results.

Figure 4-13: Tradeoff between precise searches (i.e. searching in the highlight section fields) and sensitive searches (i.e. searching in all-text fields) [36].

Authors that collected bibliometric technology records for their research often failed to report the methodology they followed to form and search for queries. Table 4.3 summarizes the method for designing and searching for queries in databases for five technology assessment and forecasting papers.

Table 4.3: A summary of the design and search methodologies for queries in literature

Literature work Method for designing Method for searching queries queries

Lezama-Nicolás et Literature review and Searched in title field for scientific publications al. (2018) [15] expert validation (title and abstract if necessary), title field for patents, and headline field for news.

Rodríguez-Salvador Literature review and Searched in title, abstract, and keyword field for et al. (2017) [19] expert validation scientific publications, title, abstract, and claim field for patents. No mention of type of patents.

Bengisu et al. Iterative process and No mention of fields. Patents collected were (2006) [20] expert validation based on issue date.

Moro et al. (2020) Expert validation Searched in title, abstract and keyword field for [22] both scientific publications and patents. No mention of type of patents.

Lokuhitige et al. No mention of how Searched in all text for both scientific (2017) [24] queries were designed publications and patents. Patents collected were based on application date.

The most common method of creating queries in these works was expert generation. However, these papers often lacked details of the expert, their training, and their scope of expertise. Furthermore, the papers did not provide additional details about the query creation process the expert used. Automating technology assessments is difficult without a repeatable and reproducible query forming method. Forming queries using expert opinion limits the potential of having an industry-wide technology assessment tool. How can we create a tool that can be available to individuals with varying levels of knowledge of a technology? There is also lack of consensus on the search method followed. Some authors used the all-text fields, while others used the highlights field for their searches. Some collected patent applications to account for the lag time in receiving patent approvals while others collected granted patents.

Future work must focus on best practices of forming and searching queries that would result in the highest quality of technology records and reproducible results. This grants accurate decisions about the relationship between the trends in bibliometric records and the Technology Readiness Levels. Until then, this remains a big challenge.

Conclusions and Recommendations 5.1 Introduction

In this final chapter, we will discuss future directions of this work, its limitations, and a conclusion of the findings. The future work section will outline potential modifications to bibliometric Technology Readiness Level assessments that we believe, with further research, can be developed into a trusted TRL automation tool.

5.2 Future Work: Modifications to Bibliometric Technology Readiness Level Assessments

In chapter 4, we highlighted some of the challenges and gaps of using bibliometrics for the automation of Technology Readiness Level assessments. In this section, we offer future work directions and recommendations for addressing each gap.

5.2.1 Designing a Machine Learning Model

In our previous chapter, we showed how pairing bibliometric indicators to technology life cycle stages results in inaccuracies. Previous research typically limits its analysis to studying scientific and engineering publications, patents, and news and media records. However, other research has criticized the use of single indicators to estimate a technology life cycle stage [39], [67], [68]. We believe we should expand beyond the existing single indicator model by adding more relevant technology record sources that can be important signals of maturity.

One of the highlights of this research project was our collaboration with Mergeflow, an analytics company based in Germany that aids scientists, engineers, product managers, and investors to discover and track emerging technologies. Their database includes data from: venture capital, market estimates, clinical trials, technology blogs, technology licensing, and funded research projects. Through this collaboration and access to the Mergeflow database, we gained better understanding of how we can expand beyond the indicators suggested by Watts and Porter.

For example, in addition to scientific publications, funded research project data may be an indicator that a technology is in its early Technology Readiness Level (TRL 1-5). Funding sources with publicly available data include organizations such as the Small Business Innovation Research program and the National Science Foundation. These organizations typically provide

54 the first funding a business or research group receives and is often at the earliest stages of developing a technology. On the other hand, venture capital investments are often larger in value and happen at later stages of maturity. This is typically at the growing phase (TRL 6-7) or mature phase (TRL 8-9) [69]. The initiation of clinical trials can signal a shift in Technology Readiness Levels since medical devices start undergoing clinical trials as they move from TRL 5 to TRL 6 [5]. Technology licensing records can be collected from technology transfer offices of universities and R&D organizations. This data can be an indicator of a growing technology (TRL 6-7). In addition to these databases, Mergeflow has an algorithm that can extract market estimates from web-available contents and provide technology market segments and market size in USD. The availability of market segments can signal that a technology is now mature (TRL 8- 9).

These databases can be used as additional bibliometric indicators of maturity. However, as we showed in chapter 4, technologies are complex and often behave unexpectedly in their technology record activity and so we cannot label Technology Readiness Levels to these databases without further investigation. Future research must understand the relationship between Technology Readiness Levels and each of the additional maturity indicators we outlined. One way we recommend is the use of supervised machine learning. Figure 5-1 shows the machine learning model we suggest. Each Technology Readiness Level bracket (emerging (TRL 1-5), growing (TRL 6-7), mature (TRL 8-9)) will have features (i.e. data from each of the new bibliometric indicators suggested by this thesis) that will be used to train a machine learning algorithm. This form of supervised machine learning will output a prediction model that can then estimate the Technology Readiness Levels of technologies.

Another challenge discussed in chapter 4 is the influence of public bias on the number of news and media records of a technology. This can often make technologies appear as being higher on the TRL scale (more mature), when they are not. This challenge could also be addressed using the machine learning model suggested above by updating it to include a measurement of public bias. One potential way we can measure public bias is by comparing the ratio of scientific publications to social media tech blogs. We hypothesize that a small ratio of scientific publications to technology blogs can signal bias. This input can be used to detect bias so that the algorithm updates its assessment accordingly.

In addition, the use of a machine learning algorithm to estimate TRLs would substitute the traditional technology trend models of hype cycles and S-curves. As we showed in chapter 4, these models do not often accurately capture bibliometric record activity. Instead of relying on curve estimate techniques (like non-linear regression), we can use classifiers and clusters to estimate Technology Readiness Levels [70].

Although we offer a basic machine learning model, we believe that with further development, testing, and refinement, it could result in a promising automated TRL assessment tool.

5.2.2 Surveying Practitioners and Updating Query Designs

In chapter 4 we argued that there is a lack of consensus on benchmarking the definition of mature technologies. We showed the result of a questionnaire where our expert assessed the maturity of technologies based on their market readiness, not their technical readiness. We

56 believe a wider distribution of the questionnaire to practitioners (the end users of an automated TRL assessment tool) can help us understand how experts view and assess maturity.

Another gap of automating TRL assessments discussed in chapter 4 is the lack of a repeatable and reproducible method of designing search queries. We recommend future research to focus on providing best practices for technology forecasting and assessment researchers to accurately design queries. In addition, the design of a search query is particularly challenging when addressing the issue of defining context and application of technologies. Previously in this thesis, we showed how technology names can often be used to refer to several applications, each with unique TRLs. For this reason, collecting technology records using keyword search queries may results in significant noise. Technology records should accurately reflect the document’s context and application in describing a technology, which is challenging to capture using basic search query methods.

To assess relevancy of results, Natural Language Processing tools (NLP) can be used. Natural Language Processing is a branch of Artificial Intelligence where computers can interpret and “understand” contents of documents. NLP-enhanced bibliometrics can understand the full text of records collected to assess relevancy [71]. Previous research has used NLP methods to collect relevant patents for technology assessment and forecasting research [72], [73]. We can use similar methods in collecting all bibliometric records to ensure that context and application for technologies are defined in the scope of the document collected. NLP tools can collect records based on their meaning, beyond keyword occurrences only.

5.3 Limitations

To develop an automated tool that accurately measures the maturity of technologies, we need a large database of accurate TRL assessments in order to build a trusted machine learning model. However, there is currently a lack of publicly, specifically recently, available Technology Readiness Level assessments. In addition, we found no roadmap reports that updated their TRL assessments. Having multiple TRL assessments for the same set of technologies over time will create a more accurate prediction model. Understanding how bibliometric records change with a technology’s maturity is valuable input that could allow us to better understand the relationship between bibliometric indicators and maturity levels. These challenges limit our ability to use machine learning tools.

Another limitation of estimating Technology Readiness Levels using bibliometrics is the grouping of levels. The TRL scale was created with 9 levels, each signaling a specific maturity status. Grouping Technology Readiness Levels into categories takes away from the precision of the scale. Bibliometric indicators may lack the sensitivity to accurately distinguish TRLs within the same category (for example: differentiate a TRL 6 from a TRL 7 in the growing phase). Therefore, this must be investigated further.

There is also a lack of empirical evidence in the literature to support the mapping of Technology Readiness Levels to life cycle stages. For example, one can argue that component validation in a laboratory environment (TRL 4) or a relevant environment (TRL 5) can warrant a technology to have the growing life cycle status and not emerging as the previous literature has estimated. The ambiguous assignment of TRLs to life cycle stages are a limitation of using bibliometric indicators.

Although expertly assessed TRLs invite bias, they also invite discussions, disagreement and learning which ultimately leads to a better understanding of the technology at hand. An automated tool would replace the need for an in-person discussion which might limit the exchange of ideas and therefore the collaborations between experts.

Finally, although a multi-input bibliometric indicator model would certainly be better than a single indicator model, it still assumes that all technologies follow a similar bibliometric pattern. However, non-commercial technologies, for example, might have no patents or venture capital data available when compared to a commercial technology. Therefore, assuming that a universal automated TRL tool is applicable to all industry technologies is flawed. The outputs of the automated assessment may still require expert evaluation.

5.4 Conclusion

In this thesis, we evaluated the use of bibliometrics as a Technology Readiness level indicator. We did so by applying a method from the literature that used non-linear regression on technology records from publications, patents, and news and media to estimate TRLs. The method’s TRL estimates were compared to expertly assessed TRLs for the same technologies and we found no agreement between the two. We attribute these results to challenges that complicate the overall automation of Technology Readiness Level assessments using

58 bibliometric indicators. We elaborate on these challenges in detail: the inaccurate mapping of technology record sources to levels of maturity, the unavoidable public bias surrounding some technologies, the inaccurate existing technology trends used to model bibliometric records, the dependency of maturity on the scope of the technology, the lack of consensus on benchmarking a definition for mature technologies, and the lack of trusted and accurate search query design methods.

We challenged the Watts and Porter model of pairing bibliometric indicators to technology life cycle stages and we argue that the inherent complexity of technologies is translated into their unique and unpredictable bibliometric patterns. Therefore, the reduction of maturity into three life cycle stages each with a single bibliometric indicator is flawed. We must rethink the original bibliometric indicators used to gain an understanding of the readiness of technologies. State-of- the-art machine learning algorithms can be implemented to update outdated models of maturity. In addition, using NLP tools to collect technology records could result in more accurate TRL estimates.

Expertly assessed Technology Readiness Levels run the risk of personal bias and irreproducible results, consequently prolonging project timelines and becoming financially demanding. The automation of the TRL assessment would allow practitioners to confidently assess maturity and minimize risks in technology onboarding and investment decisions. At present, several questions remain unanswered about factors that affect bibliometric records and their relationship to Technology Readiness Levels. However, we are optimistic that with new advances in AI and machine learning, we can better understand and model bibliometric indicators towards an automation of TRL assessments. Our aim is that by highlighting the possibility of machine learning as a promising TRL automation tool, we’ve inspired future work on this topic.

Bibliography

[1] T. Albert, M. G. Moehrle, and S. Meyer, “Technological Forecasting & Social Change Technology maturity assessment based on blog analysis,” Technol. Forecast. Soc. Chang., vol. 92, pp. 196–209, 2015. [2] NASA, NASA System Engineering Handbook SP-2016-6105 Rev2, Rev2 ed. 2016. [3] DW Engel, A. Dalton, K. Anderson, C. Sivaramakrishnan, and C. Lansing, “Development of Technology Readiness Level ( TRL ) Metrics and Risk Measures,” 2012. [4] M. J. Ward, S. T. Halliday, and J. Foden, “A readiness level approach to manufacturing technology development in the aerospace sector : an industrial approach,” vol. 226, pp. 547–552, 2011. [5] A. Naveau, R. Smirani, S. Catros, H. de Oliveira, J. C. Fricain, and R. Devillard, “A bibliometric study to assess bioprinting evolution,” Appl. Sci., vol. 7, no. 12, pp. 1–16, 2017. [6] N. R. Canada, “Government of Canada,” 02-Nov-2020. [Online]. Available: https://www.nrcan.gc.ca/climate-change/canadas-green-future/clean-growth- programs/20254. [Accessed: 25-Nov-2020]. [7] European Commission, “Horizon 2020 Funding & Tender Opportunities ,” Funding & tenders. [Online]. Available: https://ec.europa.eu/info/funding- tenders/opportunities/portal/screen/support/faq/2890. [Accessed: 01-Nov-2021]. [8] D. Magagna and L. Margheritini, Workshop on identification of future emerging technologies in the ocean energy sector, no. March. 2018. [9] Industry Canada, Technology Roadmapping in Canada: A Development Guide. 2007. [10] J. P. Martino, “A review of selected recent advances in technological forecasting,” Technol. Forecast. Soc. Change, vol. 70, no. 8, pp. 719–733, 2003. [11] C. Y. Hung, W. Y. Lee, and D. S. Wang, “Strategic foresight using a modified Delphi with end-user participation: A case study of the iPad’s impact on Taiwan’s PC ecosystem,” Technol. Forecast. Soc. Change, vol. 80, no. 3, pp. 485–497, 2013. [12] Industry Canada, “Soldier Systems Technology Roadmap: Capstone Report and Action Plan,” Government of Canada. 2011. [13] A. L. Olechowski, K. Tomaschek, S. D. Eppinger, and N. Joglekar, “Technology readiness levels : Shortcomings and improvement opportunities,” Syst. Eng., vol. 23, no. 4, pp. 1–14, 2020. [14] M. Sarfaraz, B. J. Sauser, and E. W. Bauer, “Using system architecture maturity artifacts to improve technology maturity assessment,” in Procedia Computer Science, 2012, vol. 8, pp. 165–170. [15] R. Lezama-Nicolás, M. Rodríguez-Salvador, R. Río-Belver, and I. Bildosola, “A bibliometric method for assessing technological maturity: the case of additive manufacturing,” Scientometrics, vol. 117, no. 3, pp. 1425–1452, 2018. [16] J. K. Barry L. Britt, Michael W. Berry, and Murray Browne, Mary Ann Merrell, “Document Classification Techniques for Automated Technology Readiness Level

Analysis,” J. Am. Soc. Inf. Sci. Technol., vol. 59, no. 4, pp. 675–680, 2008. [17] T. U. Daim, G. Rueda, H. Martin, and P. Gerdsri, “Forecasting emerging technologies: Use of bibliometrics and patent analysis,” Technol. Forecast. Soc. Change, vol. 73, no. 8, pp. 981–1012, 2006. [18] A. Agarwal et al., “Bibliometrics: Tracking research impact by selecting the appropriate metrics,” Asian J. Androl., vol. 18, no. 2, pp. 296–309, 2016. [19] M. Rodríguez-Salvador, R. M. Rio-Belver, and G. Garechana-Anacabe, “Scientometric and patentometric analyses to determine the knowledge landscape in innovative technologies: The case of 3D bioprinting,” PLoS One, vol. 12, no. 6, 2017. [20] M. Bengisu and R. Nekhili, “Forecasting emerging technologies with the aid of science and technology databases,” Technol. Forecast. Soc. Change, vol. 73, no. 7, pp. 835–844, 2006. [21] A. Moro, E. Boelman, G. Joanny, and J. Lopez-Garcia, “A bibliometric-based technique to identify emerging photovoltaic technologies in a comparative assessment with expert review,” Renew. Energy, vol. 123, pp. 407–416, 2018. [22] A. Moro, G. Joanny, and C. Moretti, “Emerging technologies in the renewable energy sector: A comparison of expert review with a text mining software,” Futures, vol. 117, no. May 2019, p. 102511, 2020. [23] Y. Cho and T. Daim, “OLED TV technology forecasting using technology mining and the Fisher-Pry diffusion model,” Foresight, vol. 18, no. 2, pp. 117–137, 2016. [24] S. Lokuhitige and S. Brown, “Forecasting Maturity of IoT Technologies in Top 5 Countries Using Bibliometrics and Patent Analysis,” in Proceedings - 2017 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, CyberC 2017, 2017, no. October, pp. 338–341. [25] R. J. Watts and A. L. Porter, “Innovation Forecasting,” Technol. Forecast. Soc. Change, vol. 56, no. 1, pp. 25–47, 1997. [26] H. Ernst, “The Use of Patent Data for Technological Forecasting: The Diffusion of CNC- Technology in the Machine Tool Industry,” Small Bus. Econ., vol. 9, no. 4, pp. 361–381, 1997. [27] D. Kucharavy, R. De Guio, and S. C. T. Conference, “Application of S-Shaped Curves,” pp. 81–88, 2008. [28] M. R. Salvador, R. L. Nicolás, R. M. R. Belver, and A. R. Andara, “Lessons Learned in Assessment of Technology Maturity,” in Engineering Digital Transformation, 2019, pp. 103–110. [29] O. Dedehayir and M. Steinert, “Technological Forecasting & Social Change The hype cycle model : A review and future directions,” Technol. Forecast. Soc. Chang., vol. 108, pp. 28–41, 2016. [30] M. Campani and R. Vaglio, “technological research impact leading to hype-type,” pp. 75– 83, 2015. [31] Elsevier, “Scopus Factsheet,” 2019. [32] Elsevier, “Elsevier Developers,” Elsevier Developer Portal. [Online]. Available:

https://dev.elsevier.com/sc_apis.html. [Accessed: 25-Nov-2020]. [33] United States Patent and Trademark Office, “Search for patents,” United States Patent and Trademark Office - An Agency of the Department of Commerce, 31-Aug-2020. [Online]. Available: https://www.uspto.gov/patents-application-process/search-patents. [Accessed: 01-Nov-2020]. [34] PatentsView USPTO, “API Query Language,” PatentsView Query Module API, 2020. [Online]. Available: https://api.patentsview.org/query-language.html. [Accessed: 01-Nov- 2020]. [35] D. Jones, “Dow Jones- General Product.” [36] University of Toronto, “Searching the Literature: A Guide to Comprehensive Searching in the Health Sciences: Steps to Searching the Literature,” Research guides, 2020. [Online]. Available: https://guides.library.utoronto.ca/comprehensivesearching. [Accessed: 01-Nov- 2020]. [37] “Elsevier Developers,” Elsevier Developer Portal. [Online]. Available: https://dev.elsevier.com/sc_affil_search_tips.html. [Accessed: 01-Jan-2021]. [38] D. Jones, “Search Builder Quick Reference Guide Factiva.” [39] L. Gao et al., “Technology life cycle analysis method based on patent documents,” Technol. Forecast. Soc. Change, vol. 80, no. 3, pp. 398–407, 2013. [40] S. Jun, “Technological Forecasting & Social Change A comparative study of hype cycles among actors within the socio-technical system : With a focus on the case study of hybrid cars,” Technol. Forecast. Soc. Chang., vol. 79, no. 8, pp. 1413–1430, 2012. [41] S. Faidi and A. Olechowski, “IDENTIFYING GAPS IN AUTOMATING THE ASSESSMENT OF TECHNOLOGY READINESS LEVELS,” in Design 2020, 2020, pp. 1–11. [42] T. Holstein, G. Dodig-Crnkovic, and P. Pelliccione, “Ethical and social aspects of self- driving cars,” arXiv, 2018. [43] Waymo, “Waymo Safety Report,” 2020. [44] J. Adorney, “Self-Driving Cars Are a Way Better Solution to Drunk Driving Than Sobriety Checkpoints: Julian Adorney,” FEE Freeman Article, 30-Jan-2019. [Online]. Available: https://fee.org/articles/self-driving-cars-are-a-way-better-solution-to-drunk- driving-than-sobriety-checkpoints/. [Accessed: 01-Jan-2021]. [45] “Self-Driving Vehicles Will Change The World In Some Unexpected Ways,” Forbes, 15- Dec-2020. [Online]. Available: https://www.forbes.com/sites/quora/2019/02/08/self- driving-vehicles-will-change-the-world-in-some-unexpected-ways/?sh=4658cfec6030. [Accessed: 01-Jan-2021]. [46] L. Albrecht, “How driverless cars could save lives even sooner than we think,” MarketWatch, 21-Mar-2018. [Online]. Available: https://www.marketwatch.com/story/how-driverless-cars-could-save-lives-even-sooner- than-we-think-2017-11-07. [Accessed: 01-Jan-2021]. [47] Waymo, “Journey,” Waymo. [Online]. Available: https://waymo.com/journey/. [Accessed: 01-Jan-2021].

[48] F. Alkemade and R. A. A. Suurs, “Patterns of expectations for emerging sustainable technologies,” Technol. Forecast. Soc. Change, vol. 79, no. 3, pp. 448–456, 2012. [49] A. Ruef and J. Markard, “What happens after a hype? How changing expectations affected innovation activities in the case of stationary fuel cells,” Technol. Anal. Strateg. Manag., vol. 22, no. 3, pp. 317–338, 2010. [50] H. Järvenpää and S. J. Mäkinen, “Empirically detecting the Hype Cycle with the life cycle indicators: An exploratory analysis of three technologies,” 2008 IEEE Int. Conf. Ind. Eng. Eng. Manag. IEEM 2008, pp. 12–16, 2008. [51] K. D. G. Daiha, R. Angeli, and S. D. De Oliveira, “Are Lipases Still Important Biocatalysts ? A Study of Scientific Publications and Patents for Technological Forecasting,” pp. 1–20, 2015. [52] M. Rezaeian, H. Montazeri, and R. C. G. M. Loonen, “Science foresight using life-cycle analysis, text mining and clustering: A case study on natural ventilation,” Technol. Forecast. Soc. Change, vol. 118, pp. 270–280, 2017. [53] S. M. Zanjirchi, M. Rezaeian, and N. Jalilian, “Scientometrics Four decades of fuzzy sets theory in operations management : Application of life-cycle , bibliometrics and content analysis,” no. April, 2019. [54] D. Kucharavy and R. De Guio, “Application of logistic growth curve,” Procedia Eng., vol. 131, pp. 280–290, 2015. [55] C. L. Magee, S. Basnet, J. L. Funk, and C. L. Benson, “Quantitative empirical trends in technical performance,” Technol. Forecast. Soc. Change, vol. 104, pp. 237–246, 2016. [56] L. Maccari and V. Cagno, “Do we need a contact tracing app?,” Comput. Commun., vol. 166, no. November 2020, pp. 9–18, 2021. [57] P. Escobedo et al., “Wireless wearable wristband for continuous sweat pH monitoring,” Sensors Actuators, B Chem., vol. 327, no. August 2020, 2021. [58] G. Shan and B. H. Roh, “Performance Model for Advanced Neighbor Discovery Process in Bluetooth Low Energy 5.0-Enabled Internet of Things Networks,” IEEE Trans. Ind. Electron., vol. 67, no. 12, pp. 10965–10974, 2020. [59] P. Meyer, “Bi-logistic growth,” Technol. Forecast. Soc. Change, vol. 47, no. 1, pp. 89– 102, 1994. [60] D. M. Smith, “Hype Cycle for Cloud Computing , 2011,” Gart. Inc., Stamford, vol. 71, no. July, pp. 1–74, 2011. [61] J. Heimgartner, “GPS Tracking Goes Inside the Human Body,” Engineering.com. [Online]. Available: https://www.engineering.com/DesignerEdge/DesignerEdgeArticles/ArticleID/17469/GPS- Tracking-Goes-Inside-the-Human-Body.aspx. [Accessed: 01-Jan-2021]. [62] E. Commission, “Assessing Technology Readiness Levels for Artificial Intelligence,” 2020. [63] “The 6 Levels of Vehicle Autonomy Explained,” Synopsys Automotive, 2020. [Online]. Available: https://www.synopsys.com/automotive/autonomous-driving-levels.html. [Accessed: 01-Jan-2021].

[64] J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems - SAE International, 2014. [Online]. Available: https://www.sae.org/standards/content/j3016_201401/. [Accessed: 01-Jan-2021]. [65] “Autopilot and Full Self-Driving Capability,” Tesla, Inc, 09-Jun-2020. [Online]. Available: https://www.tesla.com/support/autopilot. [Accessed: 01-Jan-2021]. [66] P. H. Kobos, L. A. Malczynski, L. T. N. Walker, D. J. Borns, and G. T. Klise, “Timing is everything: A technology transition framework for regulatory and market readiness levels,” Technol. Forecast. Soc. Change, vol. 137, no. July, pp. 211–225, 2018. [67] R. Haupt, M. Kloyer, and M. Lange, “Patent indicators for the technology life cycle development,” Res. Policy, vol. 36, no. 3, pp. 387–398, 2007. [68] C. Lee, J. Kim, O. Kwon, and H. G. Woo, “Stochastic technology life cycle analysis using multiple patent indicators,” Technol. Forecast. Soc. Change, vol. 106, pp. 53–64, 2016. [69] G. Adam et al., “Measuring Technology Readiness for Investment: Accelerating Technology Development and Improving Innovation Performance,” no. March, p. Report of a project undertaken by Manufacturing Te, 2017. [70] Sunil RayI am a Business Analytics and Intelligence professional with deep experience in the Indian Insurance industry. I have worked for various multi-national Insurance companies in last 7 years., “Commonly Used Machine Learning Algorithms: Data Science,” Analytics Vidhya, 23-Dec-2020. [Online]. Available: https://www.analyticsvidhya.com/blog/2017/09/common-machine-learning-algorithms/. [Accessed: 02-Jan-2021]. [71] I. Atanassova, M. Bertin, and P. Mayr, “Editorial: Mining Scientific Papers: NLP-enhanced Bibliometrics,” Frontiers, 10-Apr-2019. [Online]. Available: https://www.frontiersin.org/articles/10.3389/frma.2019.00002/full. [Accessed: 02-Jan- 2021]. [72] H. Park, K. Kim, S. Choi, and J. Yoon, “A patent intelligence system for strategic technology planning,” Expert Syst. Appl., vol. 40, no. 7, pp. 2373–2390, 2013. [72] A. Moeller and M. G. Moehrle, “Completing keyword patent search with semantic patent search: introducing a semiautomatic iterative method for patent near search based on semantic similarities,” Scientometrics, vol. 102, no. 1, pp. 77–96, 2015.

Appendices

Appendix A: List of Queries

The following appendix contains a list of all the queries for each of the 45 SSTR technologies used to collect data. Queries were formed by using concept mapping. It’s important to note that all records for the following queries were collected in October and November of 2020.

1. Micro reformation for fuel cells

A B C

Concept and Micro reform* OR micro-reform* OR “fuel N/A Synonym microreform* cells”

2. PEM fuel cell

A B C

Concept and “Proton-exchange membrane” OR “polymer “fuel N/A Synonym electrolyte membrane” cells”

3. Pyroelectric material for power and energy harvesting

A B C

Concept and Synonym “Pyroelectricity” OR “pyroelectric material” “harvest*” N/A

4. Piezoelectric material for power and energy harvesting

A B C

Concept and Synonym “Piezoelectricity” OR “piezoelectric material” “harvest*” N/A

5. Thermoelectric material for power and energy harvesting

A B C

Concept and Synonym “Thermoelectric material” OR “thermoelectric” “harvest*” N/A

6. Electromechanical energy harvester

A B C

Concept and “electromechanical energy” OR “electromechanical “harvest*” N/A Synonym power”

7. Li-air battery

A B C

Concept and Synonym “Lithium-air battery” OR “Li-air battery” N/A N/A

8. Li/Cfx battery

A B C

Concept and “Lithium carbon monofluoride” OR “Li/CFx” OR “battery” N/A Synonym “lithium CFx” OR “Li-CFx”

9. Ni-zn battery

A B C

Concept and “Ni-zn battery” OR “Nickel zinc battery” OR “Nizn N/A N/A Synonym battery”

10. Nano-battery

A B C

Concept and Synonym “Nanobattery” OR “nano-battery” OR “nano battery” N/A N/A

11. Electro-textile shielding

A B C

Concept and “E-textile” OR “Electronic textiles” OR “electro- “shield*” N/A Synonym textile”

12. Connectors for e-textiles

A B C

Concept and “E-textile” OR “Electronic textiles” OR “electro- “connector” N/A Synonym textile”

13. Cognitive radio

A B C

Concept and Synonym “Cognitive radio” N/A N/A

14. Millimetric radio waveforms

A B C

Concept and “Extremely high frequency” or “Radio wave*” OR “radio N/A Synonym “millimet*” frequenc*”

15. Adaptive RF-filters

A B C

Concept and Synonym “radio frequency” OR “radio-frequency” filter* adaptive

16. Automatic speech recognition

A B C

Concept and Synonym “speech recognition” OR “speech to text” N/A N/A

17. Speech synthesis

A B C

Concept and “Speech synthesis” OR “text-to-speech” OR “text to N/A N/A Synonym speech”

18. Brain computer interface

A B C

Concept and "brain computer interface" OR "brain-computer interface" OR N/A N/A Synonym "neural-control interface " OR "mind-machine interface" OR " direct neural interface " OR " brain-machine interface"

19. Ultrasonic ranging

A B C

Concept and “Ultrasonic rang*” OR “Ultrasonic module” OR N/A N/A Synonym “Ultrasonic sensor”

20. Mobile ad hoc networks (MANETs)

A B C

Concept and "mobile ad hoc networks" OR "MANET" OR "MANETS" OR N/A N/A Synonym "mobile ad-hoc networks" OR "mobile ad hoc network" OR "mobile ad-hoc network"

21. Software defined radio

A B C

Concept and Synonym “Software defined radio” Or “software-defined radio” N/A N/A

22. Solar energy harvesting

A B C

Concept and Synonym “Solar energy” OR “solar power” “harvest*” N/A

23. 3D audio

A B C

Concept and "3D audio" OR "3-D audio" OR "three dimensional N/A N/A Synonym audio"

24. Optical LIDAR

A B C Concept and “Optical radar” OR "light detection and ranging" OR N/A N/A Synonym "LIDAR"

25. Li-ion battery

A B C

Concept and synonym "Li-ion battery” OR “Lithium-ion battery” N/A N/A

26. Low power processors

Concept and “Ultra-low-voltage processor” OR “low power processor” OR “ULV synonym processor”

27. Bluetooth

Concept and synonym “bluetooth”

28. Accelerometer

Concept and synonym “accelerometer”

29. ECG Biosensor

A B

Concept and "ECG sensor” OR “EKG sensor” OR "ECG biosensor” OR “EKG N/A synonym biosensor” OR “heart rate sensor” OR “heart rate biosensor”

30. Electromechanical actuator

Concept and synonym "Electro-mechanical actuator” OR “electromechanical actuator”

31. Noise cancelling technology

Concept and “noise cancellation” OR “active noise reduction ” OR “Active noise synonym control”

32. Bone Conduction Hearing Aid

A B

Concept and "Bone conduction hearing aid” OR “Bone-anchored hearing N/A synonym aid”

33. Gyroscope Sensor

Concept and synonym “Gyroscope” OR “gyro sensor"

34. Zigbee

Concept and synonym “Zigbee”

35. Actigraph Monitor

Concept and synonym “Actigraph” OR “actimetry sensor”

. 36. SPO2 biosensor

A B

Concept and “SPO2 sensor“ OR “oxygen saturation sensor” OR “SPO2 N/A synonym biosensor“ OR “oxygen saturation biosensor”

37. Human exoskeleton (powered exoskeleton)

Concept and “powered exoskeleton” OR “power armor” OR “powered armor” OR synonym “powered suit” OR “exoframe” OR “hardsuit” OR “exosuit” OR “human exoskeleton”

38. Short wave infrared sensor (SWIR)

Concept and “Short wave infrared sensor” OR “Shortwave infrared sensor” OR “SWIR synonym sensor” OR “Short-wavelength infrared sensor”

39. UAV autopilots

Concept and “UAV autopilots” OR “ unmanned aerial vehicle” OR “uncrewed aerial synonym vehicle”

40. Z-wave

Concept and synonym “Z-wave” OR “Z wave”

41. Blood pressure sensor

A B

Concept and synonym “blood pressure sensor” OR “blood pressure biosensor” N/A

42. Ultra-wideband technology

Concept and “Ultra-wideband” OR “ultra wideband” OR “ultra-wide band” OR synonym “ultraband”

43. Wind energy harvesting

A B

Concept and synonym "Wind energy" OR "wind power" "harvest*”

44. Hydro energy harvesting

A B

Concept and synonym "hydro energy" OR "hydro power" "harvest*”

45. Computer vision

A B

Concept and synonym "computer vision” N/A

46. Self-driving Car

A B

Concept and "self driving car” OR “self driving vehicle” OR “autonomous vehicle” N/A synonym OR “autonomous car” OR “driverless car” OR “driverless vehicle” OR “fully automated vehicle” OR “fully automated car”

Appendix B: Technology Record Collection

This appendix provides details on the Python scripts created to collect and clean data from three databases: Scopus, USPTO, and Factiva. Scopus was used to collect scientific and engineering publications, USPTO was used to collect patent applications and Factiva to collect news and media records for. It also provides details on Python’s ATS calculation and TRL estimation. A list of the technologies and an overview of the method and results can be found in chapter 3.

The full code can be accessed on this GitHub Repository: https://github.com/SafaFaidi- byte/MASc-Thesis

1. Scopus

Figure 1 outlines the Scopus data collection process using Python. A CSV file is initially created consisting of two columns: technology name and query (Table 1). This file is manually created and must follow the Scopus query language requirements.

The CSV file is read by the Python script as a Pandas DataFrame (df_0). Technology names and queries are extracted from df_0 as two lists. The list containing the queries (querylist) is then passed to the for loop and placed in the RESTful API request. The for loop will loop through every query and every year in the selected range. The request will generate a response body containing the number of documents for each year. The total number of documents for each query is extracted from the API response body and stored in a list (TotalResults_list). The year used for each request is also recorded in a list (year_list). The document counts collected

(TotalResults_list) for each year (year_list) for every technology (techlist) along with the query used (querylist) are all combined as one DataFrame (df). In order to prepare the data for logistic curve fitting, some data manipulation is first required. This includes removing document count values of 0, calculating the cumulative sum for the document count values, and normalizing those values. Once all these additional columns are added to the dataframe, it is passed to the logistic curve fitting function that uses the scipy.optimize package to find the best fit. The standard error of regression (S) is then calculated and recorded for each technology in a final CSV file.

In order to minimize any potential data losses, copies of some of the DataFrames discussed above are saved as CSV copies in a database on the researcher’s secure hard drive (Figure 1).

Figure 1. Scientific Publication data collection process for Scopus

Table 1. Scopus CSV file format example

Technology Query

Technology name Technology Query (Appendix A)

2. USPTO

The API is accessed through an HTTP GET request method where the query string can only be a single JSON object containing the parameters that determine the criteria for the query. The single JSON object requirement complicates the manual design of queries and could lead to a higher chance of human error. For this reason, an automated process for generating USPTO queries was created in Python.

The Python script will read a CSV file input as DataFrame df_0. The CSV file format, shown in Table 2, contains a single column with the technology name and multiple columns containing keywords that will eventually make up the query. These multiple columns are divided based on whether the query will contain an “AND” operator, an “OR” operator or a combination of both.

The blocks on the right of Figure 2 represent the different elements that satisfy PatentsView query requirements. X represents the mandatory criterion that combines keywords and the patent date range. Y represents an optional criterion that is only used if keywords have an OR operator. Z represents the mandatory criterion for searching every keyword in patent title and patent abstract. Each of these blocks can be combined in different formats depending on one of the following cases: query contains only “OR” operator(s), query contains only “AND” operator(s), or query contains both “OR” and “AND” operator(s). J represents the empty string that will contain the final automated query.

The flowchart shown in Figure 2 describes the various formats J can have according to the three cases outlined. Each query (J) for every technology will be appended to a final list: querylist.

Figure 2. USPTO query automation process

Table 2. USPTO CSV file format

Technology OR_1 OR_2 AND_1 AND_2 ANDOR_1 ANDOR_2

Technology Query Query "" "" Query "" name keyword_1 keyword_2 keyword_3

Figure 3 shows the flowchart outlining the full data collection method. Once querylist is complete, it will be passed to USPTO’s RESTful API using a for loop. The for loop will also loop through each specified year. The following steps used to analyze the response body, create the DataFrame df, and apply logistic curve fitting are exactly similar to Scopus’s data collection.

The standard error of regression is also calculated and recorded for each technology in a final CSV file. Copies of some of the DataFrames are also saved as CSV copies in a database on the researcher’s secure hard drive to avoid any potential data losses.

Figure 3. Patent data collection process for USPTO

3. Factiva

News and media data were collected from Factiva’s database. Factiva’s software allows a direct download of query search results as a CSV file. To prepare for data collection, a CSV file is manually created containing two columns: one with technology names and another of their queries (Table 3). Each query is manually copied onto Factiva’s search engine and the resulting CSV file (Figure 4) is manually downloaded to a database on the researcher’s computer.

Table 3. Factiva input CSV file format

Technology Query

Technology name Technology query (Appendix A)

Figure 4. Factiva’s CSV file output format

Figure 5 outlines the method used to manipulate and analyze Factiva’s data. The Python script automatically reads all downloaded CSV files from the database mentioned above and saves each as a Python Dictionary in order to extract the following: document count (TotalResults_list), years (yearlist), and query (querylist2).

One limitation of manually copying queries is the increased risk of human error that could result in inaccurate data. The script was designed to check for these potential errors by comparing querylist2 to querylist1, which contains the original queries from Table 3 (df_0). If mismatches between the two exist, the program flags the specific query(s).

If no mismatches exist between querylist1 and querylist2, the document counts collected (TotalResults_list) for each year (yearlist) for every technology (techlist) along with the query used (querylist2) are all combined as one DataFrame (df). The same data manipulation used in both Scopus and USPTO’s Python scripts are also applied here. Once all data manipulation is completed, the DataFrame (df) is passed to the hype curve fitting function that also uses the scipy.optimize package to find the best fit. The standard error of regression is then calculated and recorded for each technology in a final CSV file.

In order to minimize any potential data losses, copies of some of the DataFrames discussed above are saved as CSV copied onto a database on the researcher’s secure hard drive.

Figure 5. News data collection process for Factiva

4. ATS Calculation

ATS calculation was done on Python using the Technology records of the training set (22 mature TRL 8- 9 technologies). The three CSV files containing the S values and technology names are imported as three DataFrames: df_Scopus, df_USPTO, and df_Factiva. Grubbs outlier test is applied to each S value column of every DataFrame using Python’s smirnov_grubbs library with an alpha value of 0.5. Any outliers are removed from the respective DataFrame. The ATS value is calculated using Equation 2.

1 푋̅ + 푇 푆 √1 + ( ) 푛 푎 푛 푛

Equation 2: Upper Bound 95% Prediction Interval calculation where Xn is the mean, Sn is the standard deviation, and Ta is the 100(1-p/2)th percentile of Student’s t-distribution with n-1 degrees of freedom.

5. TRL Estimation

TRL assignment was automated using Python as seen in Figure 6. The pre-calculated ATS values were used to compare each technology in the three DataFrames listed on the right side of Figure 6. Multiple if statements were used to develop an automated TRL assessment using each technology’s recorded S value across the three databases: Scopus, USPTO, and Factiva.

The automated process uses a combination of a for loop and if statements to assign a TRL according to each technology’s different S values. The first S value to be analyzed is the Scopus S value (S_Scopus). If S_scopus is larger than the ATS value for Scopus (ATS_Scopus), then the TRL assigned is “1-2”. If this statement is false, and S_Scopus is less than or equal to ATS_Scopus, then another set of if statements are evaluated, this time using the S and ATS values for USPTO. If S_USPTO is larger than ATS_USPTO, then the TRL assigned is “3-5”. However, if S_USPTO is less than or equal to ATS_USPTO, Factiva’s S and ATS values are to be analyzed. If S_Factiva is larger than ATS_Factiva, then the TRL assigned is “6-7”. If this statement is false, and S_Factiva is less than or equal to ATS_Factiva, a TRL of “8-9” is assigned.

Figure 6. Automated TRL assignment on Python Jupyter Notebook