Paper ID #29557

The effectiveness of TRIZ from the perspective of comprehensive benefits of technological

Prof. Wei YAO, School of Public Affairs, Zhejiang University Dr. Chu Zhaowei, ZheJiang University

Chu Zhaowei, Ph.D candidate at Institute of China’s Science Technology and Policy, School of Public Affairs, Zhejiang University. He holds BS degree in Material Physics from Nanjing University of Science and Technology in 2015, and MS degree in Education from Zhejiang University in 2017. He is currently interested in engineering education, global competence and . Dr. hu shunshun, zhejiang university

Hu Shunshun is a PhD student in the Institute of China’s Science, Technology and Education Policy at Zhejiang University in Hangzhou, Zhejiang. He received a BA in Marketing from the Nanjing Univer- sity of Chinese Medicine in 2015, and a MA in Educational Economic and Management from College of Public Administration, Nanjing agricultural University in 2018. He is currently interested in higher engineering education, engineering education policy, and emerging engineering education. Mr. Bifeng ZHANG, Zhejiang University

Bifeng ZHANG is a PhD student at Zhejiang University in Hangzhou, Zhejiang, China. He received his BE degree from Beijing University of Posts and Telecommunications and MBA degree from Zhejiang University. His research focuses on engineering education and systematic innovation.

c American Society for Engineering Education, 2020 The Effectiveness Assessment of TRIZ：From the Perspective of

Comprehensive Benefits

Abstract: TRIZ is a method that identifies contradictions in a given problem and then searches for solutions. There are still some controversies on the effectiveness of TRIZ. This study focuses on three questions: (1) What is the effectiveness of TRIZ? (2) Is there any difference in the effectiveness of different TRIZ tools? (3) What are the implications for engineers to apply TRIZ in R&D activities. These questions provide an opportunity to enhance the effectiveness of TRIZ, and even improvements on the method itself.

According to the questionnaire survey on more than 300 engineers and assessment on comprehensive benefits, regression analysis was performed, to find out the contribution of TRIZ to comprehensive benefits.

Through the research above, we can draw following conclusions: (1) There are sharp differences in the effectiveness of different TRIZ tools. (2) Applying of TRIZ has more significant economic and intellectual benefits and relatively poor social benefits. (3) Engineers should attach importance to the comprehensive benefits, giving priority to effective TRIZ tools, and select TRIZ tools according to the type of problem.

1. Introduction TRIZ is the acronym of the phrase "Theory of Inventive " in Russian. Researches on TRIZ began from G. Altshuller, when he found that innovation is not a random process relying solely on uncontrolled flash of insight. He found that engineering breakthroughs depend on the objective trends or principles which can be learned and repeated [1]. G. Altshuller and engineers around him believe once these principles can be identified and codified. Engineers could make the process of invention more predictable and quicker. According to the survey by European TRIZ Association in 2009, TRIZ had been used in R&D, product , patent solutions, and engineering management over more than 39 countries nowadays [2]. Existing literatures indicate that TRIZ contributes in three aspects: (1) TRIZ can help engineers improve novelty and diversity of solutions [3], and also help them to find out root cause of technical problems [4]. (2) TRIZ can improve engineers’ , and steadily promote their ability to solve engineering problems [5]-[7]. (3) TRIZ has significant psychological value, such as increasing engineers’ confidence when they are solving engineering problems [8], and also place engineers into real engineering environment [9].

With the extensive applying of TRIZ, scholars still want to verify the effectiveness of TRIZ, because TRIZ is not a scientific theory through peer review: Firstly, what is the effectiveness of TRIZ? Some surveys show that the overall efficiency of TRIZ is relatively low [10], and the existing TRIZ literature "exaggerate the value of TRIZ without exception" [11]. Current researches on the effectiveness of TRIZ mainly focus on its impact on individual’s creativity, which cannot explain the macro benefits. However, macro impacts on economic and social benefits are the key to dispel these doubts. Secondly, are there any differences about in the effectiveness of different TRIZ tools? Research shows that benefits from applying TRIZ depends on personal talents and cannot produce systematic solutions [12]. The problem-solving process cannot be repeated, and results will vary from person to person [1].

In summary, this research focuses on three questions: (1) What is the effectiveness of TRIZ? (2) Is there any difference in the effectiveness of TRIZ tools? (3) what is the implication for engineers to apply TRIZ in R&D activities. In order to answer these questions, we collected relevant data through questionnaire. Based on factor analysis and regression analysis, comprehensive benefits of TRIZ and the effectiveness of TRIZ will be evaluated, and suggestions on applying of TRIZ will be given for engineers.

2. Background 2.1 TRIZ tools and procedures to solve problems TRIZ provides a large number of tools or methods helping engineers to solve inventive problems. As showed in Figure 1, the process of problem-solving by TRIZ is mainly divided into four steps: problem describing; problem analyzing; problem solving and scheme forming. Tools provided by TRIZ are mainly used in second and third step.

Figure 1 The process of problem-solving by TRIZ

2.2 The logic of TRIZ to solve problems Although there are many TRIZ tools with each own usage, the logic behind these tools strictly follows the problem-solving model, as shown in Figure 2. The traditional problem solving is to directly find out Specific Solutions for Specific Problems, while TRIZ converts a Specific Problem into Typical Problems first, then find out Typical Solutions for these Typical Problems, through these Typical Solutions, and finally obtains many Specific Solutions. Wang provided a typical TRIZ tool (Technical Contradiction and 40 Invention Principles) as an example to explain how TRIZ could help engineers solve inventive problems [13]: Specific problem: At high speed, the air flow over the top of a car tends to create a low-pressure area at the rear and thus the car is reducing traction and losing stability. Typical problem: Increasing traction and therefore the downward force (improving parameter #15–force) of a car at high speed, and do not want to increase its weight (worsening parameter #1–weight of a moving object). Typical solutions: We can find inventive principles #8,#1,#9,#13,#37,#28,#31,#35,#18 in the 2003 Contradiction Matrix, and finally select the inventive principle #8 “counterweight” (to compensate for the weight of an object, make it interact with the environment–e.g. use aerodynamic, hydrodynamic, buoyancy and other forces). Specific solutions: The spoilers used in airplanes for descending are adopted here on cars to reduce lift and thus improve traction while not increasing the car’s weight.

This example illustrates the logic of TRIZ. It starts with mapping the problem from a specific scenario to a set of contradicted features (force and weight), and then it maps generic inventive principles to specific solutions (the counter weight principle yields the solution of spoilers.) Obviously, most specific problem can be solved by TRIZ based on above-mentioned process and this will help engineers a lot.

Figure 2 Problem-solving model for TRIZ

3. Literature Review: Comprehensive Benefits Assessment The comprehensive benefits refer to the sum of direct and indirect benefits in engineering activities. The primary purpose of engineering activities is to obtain economic benefits, such as obtaining profits, maintaining market share, reducing costs, improving performances of products [14], improving resource utilization, improving quality and productivity, etc. [15]. Engineering activities should also pay attention to social benefits. Social benefits include promoting personal development, improving quality of life, promoting social welfare, improving resource utilization, etc. [16]. Intellectual benefits are also important in engineering activities. Intellectual benefits usually include obtaining inventive patents [17], and promoting technological accumulation [18]. Finally, negative benefits should be considered [19-20]. At present, the most commonly assessment system for comprehensive benefits is Oslo Manual assessment developed by OECD [21], which considers economic, social, and environmental impacts in production, delivery and business organization. Based on the researches above and the objective of this study, assessment index for comprehensive benefits were established as shown in Table 1.

Table 1 Assessment index for comprehensive benefits by using TRIZ

Benefits indicators Indicator interpretation / items in Questionnaire Source

Direct profits TRIZ can increase direct profits Siegel, Oslo Manual

Potential profits TRIZ can increase future profits Siegel, Oslo Manual

Market demands growth TRIZ can boost product demands and enlarge market Oslo Manual

share

Increasing in TRIZ can improve the convenience of product Siegel, Clausing productivity manufacturing, and increase productivity

Cost reduction TRIZ can reduce the total costs of product design, use and Siegel

maintenance

Improving performances TRIZ can improve the main performances of products Tewksbury、Siegel

R&D time reduction TRIZ can shorten R&D time, including faster product Oslo Manual

designing, manufacturing and time to market

Improving technology TRIZ can improve technology maturity, including Baldwin preparation reducing product failure rates, increasing reliability, etc.

Technology TRIZ can solve the key problems and promote the Tewksbury accumulation development of related technologies

Improving social total TRIZ can increase employment rate and improves social Tewksbury welfare total welfare

Improving quality of life TRIZ can improve the quality of life of customers and Oslo Manual enhance their happiness

Improving resource TRIZ can reduce waste and improve overall resource Tewksbury 、 Oslo utilization utilization Manual

Inventive Patent TRIZ can increase the number of patents for invention Campbell authorized

Technical tips TRIZ can increases the number of technical tips Gretschmann

Reducing environmental TRIZ can reduce negative impacts and hazards on Sun, Oslo Manual hazards environment such as pollution

Reducing safety risks TRIZ can reduce harmful risks during the production, Tohidi, Oslo Manual

transportation and use of products, including potential

threats, to environment and people

4. 4.1 Questionnaire design and distribution The purpose of this survey is to find out the effectiveness of TRIZ and its impact on comprehensive benefits. The questionnaire contains two parts: the first part is assessment on TRIZ tools. According to related literature [12], there are 16 TRIZ tools in total, and the respondents were asked to evaluate these 16 tools in turn. The second part is assessment on comprehensive benefits. The specific indicators are derived from Table 1. The respondents were asked to evaluate the comprehensive benefits after using TRIZ. The questionnaire is a Likert-type 5-point scale, and 1-5 indicates the effect or benefits "Extremely Little, Little, Average, Large, Extremely Large." Engineers with TRIZ experience were asked to answer this questionnaire throughout the country. Finally, 348 questionnaires were collected, of which 284 were valid. The Cronbach ’s α coefficient of all items is 0.955.

4.2 Descriptive statistics of samples The industries of respondents cover multiple industries such as “Machinery and Transportation Engineering”, “Information and Electronic Engineering” and so on. The industries distribution of respondents is shown in Figure 3. The organization of respondents is shown in Table 2. 58.8% units have more than 1,000 people and 14.79% have 501-1000 people, indicating that TRIZ is mainly applied in large and medium- sized units. As for the nature of the units surveyed, state-owned enterprises, private enterprises, and research institutes account for 33.47%, 34.275，29.84%respectively, and foreign-funded enterprises accounts for 2.42%.

Figure 3 Industries of samples

The basic characteristics of samples are shown in Table 2. As we can see, men account for 73.95% and women account for 26.41%. In terms of academic qualifications, undergraduates account for 35.04%, masters account for 41.6%, PhD accounts for 14.81%, and junior colleges and others account for 8.55%. As for the time to learn TRIZ, 41.93% respondents have learned TRIZ less than 5 years. In short, according to the basic characteristics, the samples basically meet the demands for random sampling.

Table 2 Basic characteristics of samples

Variable Category Samples Proportion size >1000 167 58.80% Unit size 501-1000 42 14.79% 251-500 19 6.69% <250 56 19.72% State-owned enterprise 83 33.47% Nature Private Enterprise 85 34.27% Research Institutes 74 29.84% Foreign-funded 6 2.42% enterprises Gender Male 209 73.59% Female 75 26.41% Specialties and others 21 8.55% Undergraduate 87 35.04% Education Master degree 103 41.6% PhD 37 14.81% Time to >10 years 41 16.53% learn TRIZ 5-10 years 58 23.39% <5 years 104 41.93%

4.3 Descriptive statistics on the effects of TRIZ tools In the questionnaire, the respondent is asked to evaluate the effect of each tool. The score 1-5 are divided into "Minimal effect", "Small effect", "Medium effect", "Large effect", "Great effect". As shown in Table 3, three tools with the highest scores in the survey are technical contradiction and 40 invention principles, physical contradiction and separation principles, and functional analysis, which reached 4.15, 4.11, and 4.09 respectively. The mean value of these three tools is all greater than 3, and the standard deviations are all less than 1.

The survey on the effects of TRIZ shows that: Firstly, all TRIZ tools can positively promote innovation, and a small standard deviation indicates that there are less disputes on the value of these tools; Secondly, the assessment on effects of TRIZ tools are very different. Technical Contradiction and 40 Invention Principles, Physical Contradiction and Separation Principles, Functional Analysis, Causal Effects Chain Analysis, these four tools have the highest evaluation, all greater than 4, but the evaluations on STC, Method of Little Men, Goldfish Method, ARIZ, other TRIZ tools are all lower than 3.51.

Table 3 Mean and standard deviation of survey

Variables Standard Mean Deviation Cause Effect Chains Analysis 4.15 0.800 Resource Analysis 4.11 0.801 Functional Analysis 4.09 0.808 Nine screen method 4.08 0.845 System Trimming 3.92 0.874 Technical Contradiction and 40 Invention Principles 3.85 0.770 Physical Contradiction and Separation Principle 3.85 0.943 Su-Field Modelling and 76 Standard Solutions 3.83 0.825 Database of Effects 3.78 0.843 IFR 3.73 0.889 Laws of Systems Evolution 3.68 0.849 STC 3.55 0.914 Method of Little Men 3.51 0.859 Goldfish Method 3.51 0.900 ARIZ 3.39 0.893 Other TRIZ tools 3.35 0.858

5. Data Processing and analysis 5.1 Factor analysis on the effect of TRIZ tools The KMO and Bartlett test were performed on the items about the effectiveness, and the KMO value is 0.933. The p-value of the Bartlett spherical test is 0.000, indicating that the data is suitable for factor analysis. The results of the factor analysis are shown in Table 4. Three common factors are identified. The factor loadings are between 0.494 and 0.845. The cumulative interpretation of the overall variance value reached 65.682%.According to the characteristics of tools included in common factors, and refer to Zlotin's classification on TRIZ tools[22]:(1) Analytical tools, which are tools to analyze definition, and build problem models; (2) Knowledge-based tools, which are tools that provide systematic knowledge and methods; (3) Psychological operators, which are tools to improve problem-solving processes and inspire ideas. These three common factors are named as analytical tools, knowledge-based tools, psychological operators, respectively.

Table 4 Factor analysis on the effect of TRIZ tools

Common factors Number Tools 1 2 3 1 Cause Effect Chains Analysis 0.755 2 Resource Analysis 0.774 3 Functional Analysis 0.776 4 Nine screen method 0.597 5 System Trimming 0.559 6 Technical Contradiction and 40 Invention Principles 0.816 7 Physical Contradiction and Separation Principle 0.816 8 Su-Field Modelling and 76 Standard Solutions 0.640 9 Database of Effects 0.494 10 IFR 0.563 11 Laws of Systems Evolution 0.533 12 STC 0.707 13 Method of Little Men 0.818 14 Goldfish Method 0.845 15 ARIZ 0.813 16 Other TRIZ tools 0.728

5.2 Factor analysis of TRIZ comprehensive benefits Firstly, the KMO and Bartlett tests were performed on the items about the comprehensive benefits. The KMO value is 0.925, and the p-value of the Bartlett test is 0.000, indicating that the data is suitable for factor analysis. The results of the factor analysis are shown in Table 5. A total of three common factors are identified. The factor loadings are between 0.558 and 0.893. The cumulative interpretation of the overall variance value reached 65.878%. According to the characteristics of indicators included in factor analysis, referring to Yang [23] who separate comprehensive benefits into three categories as economy, intellectual and society, these three common factors 1, 2, and 3 are named as economic benefits, intellectual benefits, and social benefits.

Table 5 Factor analysis of comprehensive benefits

Common factors Indictors 1 2 3 Direct profits 0.689 Potential profits 0.663 Market demands growth 0.776 Increasing in productivity 0.731 Cost reduction 0.644 Improving performances 0.713 R&D time reduction 0.642 Improving technology preparation 0.558 Technology accumulation 0.870 Improving social total welfare 0.893 Improving quality of life 0.576 Improving resource utilization 0.603 Inventive Patent authorized 0.799 Technical tips 0.708 Reducing environmental hazards 0.668 Reducing safety risks 0.705

5.3 Regression analysis (1) Regression model Regarding analytical tools, knowledge-based tools, and psychological operators as independent variables x1, x2, x3, and economic, intellectual, and social benefits as the dependent variables y1, y2, y3, relations among them conform to the following multiple linear regression model:

Model 1: Impact of TRIZ tools on economic benefits: y1=α1+β11x1+β12x2+β13x3+μ1

Model 2: Impact of TRIZ tools on intellectual benefits: y2=α2+β21x1+β22x2+β23x3+μ2

Model 3: Impact of TRIZ tools on social benefits: y3=α3+β31x1+β32x2+β33x3+μ3

Among them, α1, α2, α3 represent the constant terms of the three models; βi=(βi1,βi2,βi3)

(i=1,2,3) are the regression coefficientsμ1,μ2,μ3 are the random errors, which meet the normal distribution.

(2) Results We use SPSS 25.0 to test these three multiple linear regression models. The analysis table of variance of the regression equation is shown in Table 6. The p values are all 0.000, indicating that the linear relationship of the three linear regression models are all significant.

Table 6 Analysis of variance of multiple linear regression equations

Model Sum of squares DOF Mean square F Significance 1 Return 51.349 3 17.116 20.689 0.000b Residual 231.651 280 0.827 Total 283.000 283 2 Return 31.695 3 10.565 11.771 0.000b Residual 251.305 280 0.898 Total 283.000 283 3 Return 60.253 3 20.084 25.246 0.000b Residual 222.747 280 0.796 Total 283.000 283

The significance test table for regression coefficients is shown in Table 7. In Model 1, the regression coefficients of analytical tools, knowledge-based tools, and psychological operators are 0.138 (P <0.05), 0.280 (P <0.01), and 0.290 (P <0.01). These results show all three tools can significantly improve economic benefits. In Model 2, the regression coefficients of analytical tools, knowledge-based tools and psychological operators are 0.049 (not significant), 0.315 (P <0.01), 0.100 (P <0.05). The results show that the use of analytical tools have no significant impact on intellectual benefits, and the use of knowledge-based tools and psychological operators can significantly improve intellectual benefits. In Model 3, the regression coefficients of analytical tools, knowledge-based tools, and psychological operators are 0.335 (P <0.01), 0.259 (P <0.01), and 0.183 (P <0.01). The results show that all three tools can significantly improve the social benefits.

Table 7 Significance test for regression coefficients

S C Collinearity test Model Beta t p VIF EF (Constant) 0.000 1.000 1 Analytical tools 0.138 2.552 0.011** 1.000 1.000 Knowledge-based tools 0.280 5.177 0.000*** 1.000 1.000 Psychological Operators 0.290 5.362 0.000*** 1.000 1.000 (Constant) 0.000 1.000 2 Analytical tools 0.049 0.878 0.381 1.000 1.000 Knowledge-based tools 0.315 5.601 0.000*** 1.000 1.000 Psychological Operators 0.100 1.780 0.076* 1.000 1.000 (Constant) 0.000 1.000 3 Analytical tools 0.335 6.324 0.000*** 1.000 1.000 Knowledge-based tools 0.259 4.882 0.000*** 1.000 1.000 Psychological Operators 0.183 3.450 0.001*** 1.000 1.000

Annotation:***,**,*mean significant level are 1%､5%､10%.

6. Discussion 6.1 Benefits from TRIZ applying Based on the analysis above, we can find the applying of TRIZ could create decent comprehensive benefits. Three types TRIZ tools are all very effective in improving economic and social benefits, and knowledge-based tools and psychological operators are significantly effective in improving intellectual benefits. Technical contradiction and 40 invention principles, physical contradiction and separation principles, functional analysis, and cause effects chains analysis are the best tools. In contrast, the assessment on STC, method of little men-goldfish method, ARIZ, and several other psychological operators are poor. On the whole, analytical tools have disadvantages such as incomplete analysis problems, selection bias, and limited thinking. The existence of psychological operators has limitations such as unclear guidance and low efficiency [24]. These tools may mislead engineers in engineering design, so further optimization on these tools or integrated with other tools are necessary.

6.2 Advices to engineers Most important of all, engineers who have experience on TRIZ should fully consider the comprehensive benefits as social benefits could be neglected in most cases. To avoid this situation, Darrell Mann [25] has improved the technical contradiction based on classic TRIZ and increased its general engineering parameters from 39 to 48, and 8 of 9 new engineering parameters are directly related to social benefits. The main purpose is to transform social requirements into engineering parameters, so as to better improve social benefits. In addition, engineers could increase final benefits by selecting properly tools, such as the applying of scientific effects, which are conducive to obtain more patents. Given the various comprehensive benefits, all aspects of benefits should be considered and coordinated.

Engineers who have little experience on TRIZ should give priority on effective TRIZ tools and try to overcome their disadvantages. Some TRIZ tools have shortcomings, such as high economic benefits of psychological operators but poor assessment to stimulate creativity. Related concepts and rules need to be clarified. For example, the definition of “Past-Present-Future” of Multi Screen is vague for users. For analytical tools, the cause analysis process should be divided into internal causes (attributes) and external causes (conditions), and other analytical tools can also be introduced for integration to overcome the shortcomings of original tools. In short, by overcoming the disadvantages of these fewer effective tools in a targeted manner, the overall efficiency of TRIZ will be improved.

Last but most important, engineers need to select TRIZ tools flexibly according to actual problem. Different tools have their own advantages for different types of comprehensive benefits. Therefore, it should be guided by practical requirements and final results, and comprehensively select tools according to the characteristics of problems. For example, when economic benefits are demanded highly, the applying of psychological operators should be paid more attention to.

7. Conclusions This study focuses on the effectiveness of TRIZ and its usages for engineers. Through questionnaires, factor analysis, and regression models, questions are roughly answered. However, this study failed to analyze the root cause for the differences of tools. The subsequent analysis can be performed from engineering creativity and its mechanism. In addition, the assessment indicators used in this article need to be further improved.

Acknowledgements Research reported in this paper is funded by National Natural Science Foundation of China（No.71974172）: Evaluation, Systematic Development Mechanism and Modes of Engineering Creativity for Manufacturing Power Strategy, and funded by Humanities

and Social Sciences Fund of Chinese Ministry of Education：“Systematic Developing

Path of Engineering Creativity” (No. 17JDGC038)

References: [1] Valeri Souchkov, “A brief history of TRIZ”, April 2015. [Online]. Available: http://www.xtriz.com/BriefHistoryOfTRIZ.pdf. [Accessed April. 15, 2019]. [2] Denis Cavallucci, “World Wide status of TRIZ perceptions and uses a survey of results”, Sep 2009. [Online]. Available: http://etria.eu/documents/TRIZ_survey_V2.pdf. [Accessed April. 15, 2019]. [3] Noe Vargas Hernandez. Experimental Assessment of TRIZ Effectiveness in Idea Generation [A]. American Society for Engineering Education. ASEE Proceedings 2012[C]. IEEE,2012:25.612.1-25.612.13. [4] Birdi K, Leach D, Magadley W. Evaluating the impact of TRIZ creativity training: an organizational field study[J]. R & D Management, 2012, 42(4):315-326. [5] Haines-Gadd L. Does TRIZ Change People? Evaluating the Impact of TRIZ Training within an Organization: Implications for Theory and Practice[J]. Procedia Engineering, 2015, 131:259-269. [6] Harlim J, Belski I. On the effectiveness of TRIZ tools for [J]. Procedia Engineering, 2015, 131:892-898. [7] Harlim J, Belski I. Learning TRIZ: Impact on Confidence when Facing Problems[J]. Procedia Engineering, 2015, (131):95-103. [8] Ogot M, Okudan G. Integrating systematic creativity into first-year engineering design curriculum[J]. International Journal of Engineering Education, 2006, 22(1):109-115(7). [9] Elvin Shields. Fostering Creativity in the Capstone Engineering Design Experience[A]. American Society for Engineering Education. ASEE Proceedings 2007[C]. IEEE,2007:12.756.1-12.483.10. [10] IlevbareI M, Probert D, Phaal R. A review of TRIZ, and its benefits and challenges in practice[J]. Technovation, 2013,33(2-3):30-37. [11] Chechurin L. Research and Practice on the Theory of Inventive Problem Solving (TRIZ)[M]. London, Springer, 2016:2-5.

[12] Spreafico C, Russo D. TRIZ Industrial Case Studies: A Critical Survey ☆[J].

Procedia Cirp, 2016, 39:51-56. [13] Shih-Liang (Sid) Wang. Developing A TRIZ Design Tools to Enhance Engineering Design Courses [A]. American Society for Engineering Education. ASEE Proceedings 2012[C]. IEEE, 2007:12.483.1-12.483.8. [14] Siegel D S, Phan P. Advances in the Study of Entrepreneurship, Innovation & Economic Growth. Evaluating the social returns to innovation: An application to university technology transfer[J]. Advances in the Study of Entrepreneurship Innovation & Economic Growth, 2009, 19(1):171-187. [15] Clausing D, Fey V, “Effective Innovation—Benefits”, May 2004. [Online]. Available: https://asmedigitalcollection.asme.org/ebooks/book/120/chapterabstract/23236/Ef fective-Innovation-Benefits?redirectedFrom=fulltext. [Accessed Dec. 15, 2019]. [16] Campbell, B. How to judge the value of innovation[J]. BMJ, 2012, 344 :1457-1457. [17] Gretschmann K, Schepers S. Revolutionising EU Innovation Policy [M]. London: Springer,2016: 978-1057. [18] Baldwin A, Betts M, School H O, et al. A Framework for Measuring It Innovation Benefits[J]. Electronic Journal of Information Technology in Construction, 2000,5(6):1-7. [19] Sun Yan. Introduction into the Social Assessment of Science and Technology[J]. Studies in Philosophy of Science of Technology, 2012, 29(2):92-96. [20] Tohidi H, Jabbari MM. Providing a Framework for Measuring Innovation within Companies[J]. Procedia Technology, 2012, 1:583-585. [21] OECD. Oslo Manual 2018: Guidelines for Collecting, Reporting and Using Data on Innovation (4th Edition), The Measurement of Scientific, Technological and Innovation Activities[M]. Paris: OECD Publishing, 2018:85-102. [22] Boris Zlotin, Alla Zusman, Len Kaplan, “TRIZ Beyond Technology: The Theory and Practice of Applying TRIZ to Non-Technical Areas”, Jan 2001. [Online]. Available: https://pdfs.semanticscholar.org/c7d7/653f22df8c3e448b261e2a45a54c2b137cb6. pdf. [Accessed April. 15, 2019]. [23] Yang Liexun. Review of Research on R&D Project Assessment [J]. Journal of Management Engineering in China, 2002, 16 (2): 60-65. [24] Yao Wei, Han Xu. A Theory driven Comparison and Enlightenment of TRIZ, USIT and ASIT[J]. Science & Technology Progress and Policy, v.35; No.448 (12): 7- 14. [25] Darrell Mann, Simon Dewulf, Boris Zlotin, Alla Zusman, “"Matrix 2003" Publication Announcement of the Japanese Edition and Q&A Documents for the English Edition”, Apr 2005. [Online]. Available: https://www.osakagu.ac.jp/php/nakagawa/TRIZ/eTRIZ/elinksref/eSKIBooks/eMa trix2003/eMatrix2003-050329.html#PaperMann. [Accessed Dec. 15, 2019].