Research Policy 44 (2015) 34–49

Contents lists available at ScienceDirect

Research Policy

jo urnal homepage: www.elsevier.com/locate/respol

The evolving state-of-the-art in transfer research:

Revisiting the contingent effectiveness model

a,∗ b c

Barry Bozeman , Heather Rimes , Jan Youtie

a

Center for Organizational Research and Design, Arizona State , United States

b

Department of Public Administration and Policy, University of Georgia, United States

c

Enterprise Institute and School of Public Policy, Georgia Institute of Technology, United States

a r t i c l e i n f o a b s t r a c t

Article : The purpose of our study is to review and synthesize the rapidly evolving literature on technology transfer

Received 2 December 2013

effectiveness. Our paper provides a lens into relatively recent work, focusing particularly on empirical

Received in revised form 18 March 2014

studies of US technology transfer conducted within the last 15 years. In doing so, we update and extend

Accepted 23 June 2014

the Contingent Effectiveness Model of Technology Transfer developed by Bozeman (2000). Specifically,

Available online 28 July 2014

we include the growing interest in social and public value oriented technology transfer and, thus, the

contingent effectiveness model is expanded to consider this literature. We categorize studies according

Keywords:

their approaches to measuring effectiveness, draw conclusions regarding the current state of technology

Technology transfer

transfer evaluation, and offer recommendations for future studies.

Public policy

Research © 2014 Elsevier B.V. All rights reserved. Theory

1

1. Introduction technology transfer. Using the same structure facilitates com-

parison of the pre- and post-2000 technology transfer literature.

Technology transfer continues to be a popular topic among Further, we categorize studies according their approaches to mea-

not only researchers, but also among managers and entrepreneurs suring effectiveness, draw conclusions regarding the current state

trolling the academic literature and hoping for usable knowledge. of technology transfer evaluation, and offer recommendations for

As is the case for so many popular research topics, especially those future studies.

addressed by the researchers from numerous, diverse disciplines, Since Bozeman’s previous study, the broader technology

the research findings and theory developments in technology transfer literature has been expanding rapidly in several major

transfer evolve rapidly. Our paper provides a lens into relatively directions. First, there have been many studies of

recent work, focusing particularly on the last 15 years. laboratory and research centers, especially those located European

In 2000, Bozeman published in this journal a comprehensive nations. During the period covered in Bozeman’s earlier review,

state of the art review of domestic technology transfer literature. the majority of studies focused on US laboratories and research

Our study updates and extends this review, with an emphasis, centers. A second trend is that the vast majority of the post-2000

although not an exclusive one, on the US technology transfer pol- technology transfer literature focuses on transfer from university

icy and program context, and research about these policies, and settings or from multi-organizational research centers or consortia

programs. In doing so, the paper employs a modestly revised (many of which are anchored by or housed entirely in ).

version of the Contingent Effectiveness Model of Technology Trans- A third trend is that non-linear technology transfer mechanisms

fer used in the earlier paper. The model has by this time been have been put forth and analyzed to a greater extent. Bradley

adapted or applied directly in scores of analyses or evaluations of et al. (2013) developed rich descriptions and sets of literature

around these non-linear mechanisms. These authors highlight four

such non-linear mechanisms: (1) reciprocal relationships among

The research was supported by the U.S. National Institute of Standards and Tech-

nology under a under a subcontract from VNS Group, Inc. The opinions expressed in

1

this monograph are the authors’ and do not necessarily reflect the opinions of any The Contingent Effectiveness model has been used in application or as a concep-

government agency, or Arizona State University, the University of Georgia, Georgia tual framework in a wide variety of articles, ranging from industrial ecology to higher

Tech, or VNS Group, Inc. to transfer of vaccines (see for example Ramakrishnan, 2004;

Corresponding author. Tel.: +1 4806866336. Albors et al., 2006; Albors-Garrigos et al., 2009; Mohammed et al., 2010; Kitagawa

E-mail address: [email protected] (B. Bozeman). and Lightowler, 2013; Hendriks, 2012).

http://dx.doi.org/10.1016/j.respol.2014.06.008

0048-7333/© 2014 Elsevier B.V. All rights reserved.

B. Bozeman et al. / Research Policy 44 (2015) 34–49 35

university–industry- and government actors (Etzkowitz and transfer, how they are doing it, what is being transferred and to

Leydesdorff, 2000); (2) “multiversity” approaches in which many whom.

sub-units and programs of the university can interact with The term “contingent” is key in both the original and revised

companies in diverse ways (Kerr, 2001); (3) model because of the assumption that technology transfer by

(Chesbrough, 2003) in which the university can both acquire definition includes multiple parties and these parties generally

and distribute unused ; and (4) open source have multiple goals and, ergo multiple effectiveness criteria. Effec-

approaches (such as the Creative Commons) in which knowledge tiveness is considered in terms of multiple criteria including (1)

transfer extends to collaborators through standards creation and out-the-door (was anything transferred?), (2) market impact, (3)

tacit knowledge sharing and for which the technology transfer , (4) political advantage, (5) development

office can serve as a broker. of scientific and technical human capital, and (6) opportunity cost

This current review of technology transfer evaluation stud- considerations. The revised model, shown in Fig. 2, adds an addi-

ies focuses chiefly on empirical research, including qualitative tional effectiveness criterion: public value.

research, and has a US orientation, albeit not an exclusive one.

Our primary data source was articles pertaining to evaluation of 2.1. The addition of the public value criterion

technology transfer programs or policies that appeared in scholarly

journals concerning technology, policy and management such as The addition of the Public Value criterion arises from the recog-

The Journal of Technology Transfer, Research Policy, Organization nition that transfer agents, particularly public sector transfer agents

Science, Technovation, Research Evaluation, The Journal of Higher but others as well, are housed within agencies and organizations

Education, Evaluation and Program Planning, Regional Studies, Tech- that are themselves in pursuit of broad public-interest goals. Thus,

nological Forecasting and Social Change, Minerva, R&D Management, their endeavors are motivated, influenced, and directed by ever-

and International Journal of Technology Management. We emphasize changing constellations of public values (Jørgensen & Bozeman,

post-2000 literature in this paper except in those cases where 2007). For example, each federal laboratory operates within a

an allusion to previous literature is necessary for clarifying our federal agency or department which in turn functions under the

understanding of research trajectories. To this end, we have also auspices of a mission to further some aspect of the public inter-

been guided by several technology transfer literature reviews (e.g. est. As a case in point, part of the mission of the U.S. Department

Adomavicius et al., 2008; Agrawal, 2003; Tran and Kocaoglu, 2009; of Agriculture (USDA) is to “promote agriculture production sus-

Protogerou et al., 2012; Bradley et al., 2013). tainability that better nourishes Americans while also helping feed

Two criteria for exclusion of articles were applied to this search. others throughout the world; and to preserve and conserve our

First, much of the technology transfer literature continues to focus Nation’s natural resources” (USDA, 2013). Consequently, one means

on international relations and owner , often focused by which to judge USDA technology transfer successful is if the

on relationships between nations and sometimes groups of firms. transfer in some way furthers the agency’s mission. Indeed, many

We do not examine the international technology transfer litera- federal agencies include narratives of technology transfer success

ture, though we do consider cross-national transfers among peer stories in their annual reports, and often these success stories cen-

2

firms. Second, we also do not include the great many single case ter on social impacts of agency technology transfer activities. In

study papers resident in the gray literature in this review, albeit we this way, evidence demonstrates tacit acceptance by practitioners

acknowledge the importance of such gray literature in the field of that public value is an important criterion for evaluating technology

assessing the effectiveness of technology transfer. Evaluations that transfer activity in some realms.

appear in gray literature only are not always publicly accessible, Importantly, the Public Value criterion counterbalances some

although there are a few exceptions to this rule where gray liter- of the emphasis on economic impacts of technology transfer. To

ature is included because it is accessible and it elucidates findings this end, it is comparable to notions of responsible innovation,

across programs administered by multiple organizations. Within which take into consideration equity and inequality; sustainability,

the parameters of this search, we acknowledge that despite our health and safety; and the improvement of quality of life through

best efforts to canvas this post-2000 literature, we may well have addressing societal needs or grand challenges. The expectation

missed some evaluations. that and innovation will produce economic growth

Finally, and as we see below, the present study includes the is not new; however, inclusion of the Public Value criterion in

growing interest in social and public value oriented technology the Contingent Effectiveness Model acknowledges the fact that

transfer. Thus, the contingent effectiveness model is expanded to economic impacts are sometimes not the best measure of well-

consider this literature. being. For example, if economic impacts are in aggregate favorable

but exacerbate inequalities then such an outcome may not in

some circumstances be desired. There are three reasons to give

2. The revised contingent effectiveness model of greater attention to public values in thinking about S&T policy.

technology transfer First, public values are more likely to encompass outcomes that

are ultimately important to most people. For example, despite its

Fig. 1 shows the original Contingent Effectiveness Model of pervasiveness as an instrumental concern, few people care about

Technology Transfer. The revised model is nearly identical to the economic growth for its own sake. Instead, they care about better

original. Both models identify five categories of technology transfer health, more or better leisure, safety, educational opportunity,

effectiveness determinants or contingencies, including: (1) char- or increased likelihood of obtaining a satisfying job. Economic

acteristics of the transfer agent, (2) characteristics of the transfer growth is prized because it is seen as enabling these first order

media, (3) characteristics of the transfer object, (4) demand envi- values. Second, public science and technology are supported by tax

ronment, and (5) characteristics of the transfer recipient. These dollars, under tax systems that include in most nations progressive

dimensions are not entirely exhaustive but are broad enough to elements and promotion of equity. Thus, a rationale for infusing

include most of the variables examined in studies of government public values in science, technology and innovation policy is that

technology transfer activities. The arrows in the model indicate

relations among the dimensions (broken lines indicate weaker

links). In a nutshell, both models maintain that the impacts of tech- 2

See the following for examples: http://ttc.nci.nih.gov/about/success.php;

nology transfer can be understood in terms of who is doing the http://spinoff.nasa.gov/index.html; http://techtransfer.energy.gov/energy.

36 B. Bozeman et al. / Research Policy 44 (2015) 34–49

Fig. 1. Contingent effectiveness model of technology transfer.

Fig. 2. Revised contingent effectiveness model of technology transfer.

B. Bozeman et al. / Research Policy 44 (2015) 34–49 37

those values are by definition broader values and, by implication, For this reason, if no other, it warrants special attention. But, as we

ones more likely to affect all or most citizens. see below, it also has the merit of practical utility and convenience

A third reason for systematic inclusion of public values in of measurement.

science, technology and innovation policy is that without direct The primary assumption of the Out-the-Door criterion for tech-

attention they are easily set aside or ignored. We can say that nology transfer effectiveness is that the technology transfer agent

science, technology and innovation policy values, and indeed all (e.g. the federal laboratory) has succeeded once the technology

values expressed in major policies, are both dynamic and “stage has been converted into a transfer mechanism, either formal or

dependent.” That is to say, public policies evolve in stages (Rose, informal, and another party has acquired the technology. The orga-

1993; John, 1998), though not necessarily in fixed sequence. In nization acquiring the technology may or may not have put it to

most instances, these stages include (1) agenda-setting, (2) policy use. Thus, the organization receiving the intellectual property (IP)

design(s), (3) policy choice, (4) policy implementation and (usually may do so reflexively or because there is a directive to do so, with

but not always), (5) policy assessment or even systematic evalua- an intent to use the IP or not, or even with an intent to quash the

tion. Particularly in science and (Burgess et al., technology so that it is not available for rivals. Neither the motive

2007; Bozeman and Sarewitz, 2005), values are important at every nor the uses of the IP are considered in the Out-the-Door criterion.

stage, but they are changeable and not always in predictable ways. As suggested by the label, the goal is getting the IP out the door.

Values change as a result of learning, in other cases they fall aside Within this general concept of the Out-the-Door model we can

for lack of advocacy, and in still others they fall under the weight distinguish three sets of significantly different results revealed by

of new values injected by other self-interested parties in political three different sets of indicators. In the first place we have the case

processes (Beierle and Konisky, 2000). of the “Pure Out-the-Door” in which there is no indication that any-

Table 1 describes the public value criterion along with other thing has occurred with research to the IP except for its transfer.

effectiveness criteria developed previously. The table also briefly Second, there is “Out-the-Door with Transfer Agent Impacts.” In some

reviews the advantages and disadvantages of each effectiveness cases it is clear that the transferring organization has benefited

criterion. from the activity even if no one else ever does. Thus, if a federal

laboratory obtains licensing revenue, that is a sort of impact. That

3. Technology transfer effectiveness research type of impact might not be related to the primary goals of the

US Stevensen-Wydler Act or the US Technology Transfer Act, but

This study reviews and discusses current research on technology it is an impact and one than provides benefit. Third, there is “Out-

transfer effectiveness in light of the Revised Contingent Effective- the-Door with Transfer Partner Impacts.” In most cases public policy

ness model. To enable analysis according to the elements of the focuses not on enriching technology transfer partners but rather

model, we develop a table that reports findings and recommenda- on broader social and economic impacts. Nonetheless, if partners

tions from the scholarly literature. Since the table is quite large we benefit then certainly that qualifies as an external benefit, though

include it as an appendix (Appendix A) to this paper but we draw usually a relatively narrow one.

from the table in our discussion below. Among the surprisingly few academic studies examining data

pertaining to technology transfer success, in either a federal labora-

3.1. “Out-the-Door” criterion for technology transfer effectiveness tory or a university setting, the vast majority employ Out-the-Door

measures (see for example, Thursby et al., 2001; Siegel et al., 2003;

Technology transfer research gives disproportionate attention Anderson et al., 2007; Park et al., 2010; Heisey and Adelman, 2011).

to what is referred to as the “Out-the-Door” technology transfer A typical approach is Jaffe and Lerner’s (2001). The authors examine

effectiveness criterion. This criterion is most often used by both patenting results for 23 Department of Energy federally-financed

scholars and practitioners and, in many cases, the only one used. research and development centers (FFRDC’s, i.e. US public research

Table 1

Technology transfer effectiveness criteria.

Effectiveness criterion Key question Theory base Major advantage and disadvantage

“Out-the-Door” Was technology transferred? Atheoretical or classical Advantage: Does not hold transfer agent accountable for factors

organization theory that may be beyond control.

Disadvantage: Encourages cynicism and focuses on activity rather

than outcome

Market Impact Did the transferred technology Microeconomics of the firm Advantage: Focuses on a key feature of technology transfer.

have an impact on the firm’s sales Disadvantage: Ignores important public sector and nonprofit

or profitability? transfer; must accommodate market failure issues.

Economic Did technology transfer efforts lead Regional science and public Advantage: Appropriate to public sponsorship, focuses on results

Development to regional economic finance theory. to taxpayer.

development? Disadvantage: Evaluation almost always requires unrealistic

assumptions.

Political Did the technology agent or Political exchange theory, Advantage: Realistic.

recipient benefit politically from bureaucratic politics models Disadvantage: Does not yield to systematic evaluation.

participation in technology

transfer?

Opportunity Cost What was the impact of Political economy, cost–benefit Advantage: Takes into account foregone opportunities, especially

technology transfer on alternative analysis, public choice alternative uses for scientific and technical resources.

uses of the resources? Disadvantage: Difficult to measure, entails dealing with the

“counterfactual”

Scientific and Technical Did technology transfer activity Social capital theory (, Advantage: Treats technology transfer and technical activity as an

Human Capital lead to an increment in capacity to political science), human overhead investment.

perform and use research? capital theory () Disadvantage: Not easy to equate inputs and outputs.

Public Value Did technology transfer enhance Public interest theory, public Advantage: Excellent and easily sanctioned criteria for public

collective good and broad, value theory policy.

societally shared values? Disadvantage: Extremely difficult to measure systematically

38 B. Bozeman et al. / Research Policy 44 (2015) 34–49

institutes) seeking to determine factors related to the volume of suggests that the Out-the-Door models has some reach and via-

patenting, with no analysis of the impacts of the . Adams bility. Likewise, the obvious fact that technology transfer agents

et al. (2003) provide another study focusing on federal laboratories have clearly limited domains of control over the actions of trans-

and Cooperative Research and Development Agreements (CRADA’s, fer partners means that the criterion has some common sense

i.e. arrangements between a US public research institute and a com- appeal. Nevertheless, we must consider this: if one uses only Out-

pany to engage in collaborative R&D). They employ survey data for the-Door criteria one will likely never have direct knowledge that

two years (1996 and 1998). The sample for the survey is based on the technology transfer activities have achieved the goals of having

federal laboratory CRADA partners. They find that CRADAs stim- economic and social impacts beyond those accruing to the technol-

ulate both industrial patents and industrial R&D and do so to a ogy transfer partnership. Conceivably, despite the inferences one

greater extent than other technology transfer mechanisms. Thus, might wish to make, it is possible that in many instances simply

the Adams et al. (2003) study, focusing as it does on impacts inter- getting technology out the door achieves little beneficial impact

nal to the firm, is viewed as Out-the-Door with Transfer Partner and, absent more intensive analysis, may actually do harm. For

Impacts. example, in one case study (Kingsley and Farmer, 1997) of state

Most published technology transfer studies focus on university government transfer of a transportation technology, it was deter-

technology transfer and IP activity, perhaps because of the avail- mined that the technology had been successfully transferred to a

ability of data compiled by the Association of University Technology firm and for years the transfer was viewed as a major success. Only

Managers (AUTM). Thus, for example, Powers (2003) analyzes 108 later was it learned that the technology was in short order sold by

universities and finds that the number of licenses produced relates the acquiring company to a foreign firm who used it to develop

to the technology transfer offices’ year of origin and to higher lev- a strong competitive edge against U.S.-based firms, arguably driv-

els of R&D funding. Powers also examines revenues from licenses ing some out of . For many years (the technology is now

and finds that the sizes of technology transfer offices predict license being used in the U.S.) the transfer had a significant negative eco-

revenue (and, thus, the study falls in the Out-the-Door with Trans- nomic effect on U.S. firms. Was the technology transferred? Yes.

fer Agent Impacts category). Caldera and Debande (2010) find that Was it beneficial? Only if one provides an expanded geography

the size of the technology transfer office in 52 Spanish universi- of benefit.

ties is associated with greater R&D income, spinoffs, and licensing Despite its critical limitations, the Out-the-Door model is,

activity although not licensing revenue. While license activity and arguably, the most commonly used criterion and the basis for

revenue do not necessarily provide evidence of impacts outside most metrics employed for technology transfer. The Out-the-Door

the transferring institution (for example, companies could pay for model’s popularity seemingly goes hand-in-hand with the desire

a license to suppress activity) it is likely that license revenue is for objective measures or metrics to evaluate or track technology

usually an indication of external impacts. Whether the impacts transfer. To be sure, data derived from pure Out-the-Door, Out-

are in the Economic Development category is a question unan- the-Door with Transfer Partner Impacts, and Out-the-Door with

swered here. Moreover, it is even unclear whether spinoffs, which Transfer Agent Impacts measures could prove extremely useful.

do commonly fall into the Economic Development category, actu- They are certainly good indicators of levels of technology trans-

ally lead to broader economic outcomes for a region considering fer activity, but as stated they do not provide information about

the propensity of these spinoffs to fail or, in the pursuit of proxim- downstream impacts and outcomes. While most technology trans-

ity to financing and markets, move to another region (Breznitz and fer participants well understand that just getting technology or

Taylor, 2009). IP out the door certainly does not imply that there will be any

Despite obvious disadvantages to the Out-the-Door criterion, beneficial effect from the transfer, they are equally aware of the

the model has a certain compelling logic. Depending upon whom difficulty of measuring technology transfer by any other means.

one views as the transfer agent, care must be taken to give some Moreover, many technology transfer officers feel that their activ-

account of the agents’ domain of control. To put it another way, a ities, even when quite valuable may not have early, measurable

technology transfer agent such as an Office of Research and Tech- returns. As the U.S. General Accountability Office (GAO), the inves-

nology Applications (ORTA) officer (i.e. a technology transfer officer tigative and evaluative arm of the US Congress, noted more than a

for US public research institutes) typically has a domain of influ- decade ago:

ence but a limited one. For example, the ORTA office may have

(E)xperts in research measurement have tried for years to

some capability of strategic choice among technology options, may

develop indicators that would provide a measure of results of

be able to induce work on selected technologies, and may be able

R&D. However the very nature of the innovation process makes

to develop good administrative and business practices such that

measuring the performance of science-related projects difficult.

technology transfer can be facilitated. However there are many

For example, a wide range of factors determine if and when a

other factors over which the technology transfer agent may have

particular R&D project will result in commercial or other bene-

no control, particularly the ability of firms to effectively develop

fits. It can also take many years for a research project to achieve

and market technology or the ability of firms to manage products

results (GAO 1989).

once they have been brought to market.

Nevertheless, the demand for accountability and effectiveness

To be sure, some might argue that the technology transfer agent

measures is unlikely to be deterred by the challenge of develop-

is at least partly culpable if it transfers technologies to compa-

ing timely, valid measures. Nor should it be. Federal laboratories

nies who have inadequate capital, manufacturing ability, or market

and others in the technology transfer chain are not likely to receive

savvy to make a good technology into a good, profitable product.

a “pass” just because their results typically require more time to

However, since the transfer agent certainly does not control the

gestate and fully develop. Witness the 2011 memorandum from

transfer partner (or in many instances even have much influence

the US Executive Office of the President directed to the heads of

on the partner) and since many transfer agents have limited or

R&D-performing executive agencies and departments (US White

no background market forecasting (Piper and Naghshpour, 1996;

House Office of the Press Secretary, 2011), which called for improv-

Franza and Srivastava, 2009) it does not seem reasonable to hold

ing results of technology transfer and commercialization from the

the agent and its technology transfer professionals responsible for

agencies through tracking performance against metrics as well as

the actions or inactions of partnering firms.

through streamlining the technology transfer process and partner-

The expansion beyond the Pure Out-the-Door category to con-

ing with state and local institutions. However, one reaction to the

sider impacts on, respectively, transfer agents and transfer partners

B. Bozeman et al. / Research Policy 44 (2015) 34–49 39

need to develop metrics for near term results is that these types of Even if the Market Impact model is the gold standard for effec-

metrics are often developed to measure activity not impacts. tiveness it can in some instances prove to be fool’s gold. An

important problem with the Market Impact criterion is misattri-

3.2. Market impact/economic development criterion for bution of success and poor understanding failure. If a particular

technology transfer effectiveness instance of transfer is not commercially successful, is it because the

product or process transferred is of limited value? Perhaps. But the

The “Market Impact/Economic Development” criterion focuses failure may be owing to such factors as the recipient organization’s

on (1) the commercial success of the transferred technology includ- problems in development, manufacturing, marketing, or strategy.

ing (2) impacts on regional and or national economic growth. Thus, if a new drill bit project enables deeper drilling, opening up

Hereafter, the simpler term, Market Impact, criterion will be used North Sea oil exploration (Link, 1995), how much does one credit

to signify either. Generally, market impact pertains to commercial the project versus prior science? If a firm that has been work-

results obtained by a single firm or a few firms. However, much of ing for years on automobile battery technology and finally, with

the technology transfer activities undertaken by government agen- the help of a federal laboratory CRADA-based partnership, works

cies, as well as by universities, is rationalized by broader economic with a university consortium to produce a better battery and then

multipliers assumed to flow from technology transfer. brings it to market, how does one sort out the various contributions

To a large extent the Market Impact criterion is the ‘gold (Sperling, 2001; Sperling and Gordon, 2008)? How quickly would

standard’ for technology transfer effectiveness evaluation. For the technology have developed if not for the project? Most impor-

instance, to a large extent federal policy reflects quite comfort- tant, if a U.S.-developed technology provides great benefits abroad,

ably the idea that economic impact is de facto social impact and what does that do to the accounting? Analytical precision and close

that economic growth accruing from science and technology pol- accountings are nearly impossible.

icy investments are inherently good. Not all agree, but the Obama A number of studies employ the Market Impact model in

administration, like virtually every Presidential administration assessing technology transfer effectiveness. However, the studies

before it, is on record articulating that science and technology runs are not recent ones. Among the older studies, Bozeman and col-

the “engine for economic growth” in the US and economic growth leagues (Bozeman et al., 1995, 1999; Crow and Bozeman, 1998;

is the cardinal value for a great many federal programs. As noted in Bozeman, 1994, 1997) and Roessner and his colleagues (Feller

President Obama’s speech on November 23, 2009, announcing the and Roessner, 1995; Roessner and Bean, 1991) provide consis-

“Educate to Innovate” policy initiative: “Reaffirming and strength- tent evidence from different data sources that federal laboratory

ening America’s role as the world’s engine of scientific discovery partnerships yield a great deal of economic value in the transfer

and technological innovation is essential to meeting the challenges of knowledge. Some studies (e.g. Bozeman et al., 1995; Link and

3

of this century.” Moreover, while much of the language of the Scott, 2001; Meyers et al., 2003) go so far as to offer cost–benefit

aforementioned memorandum on technology transfer (US White estimates. Typical among these earlier studies is Bozeman et al.’s

House Office of the Press Secretary, 2011) is actually quite broad, (1995) study of 219 federal laboratory partnerships, most of them

so much so that it seems to encompass nearly all the effective- based on CRADAs. They find that the mean value for company

ness criteria presented here, the more specific terminology focuses managers’ estimates of net economic benefits to the firm is approx-

on economic impacts. Thus, the memo articulates the quite gen- imately $1.5 million per project, whereas the median estimate is

eral goal “to foster innovation by increasing the rate of technology zero. This implies that such partnerships yield a few “big winners”

transfer and the economic and societal impact from Federal R&D and quite a lot of “no impact” projects.

investments” (US White House Office of the Press Secretary, 2011, During the past decade or so, several technology transfer evalu-

p. 1), but when attention is turned to measures and metrics those ation studies have been produced using the Market Impact model

identified as examples are ones chiefly relating to or supporting and based on economic impact measures. Almost all of these stud-

economic and marketplace impacts: ies have focused on university technology transfer, and many of

these employ the AUTM database. Roessner and colleagues (2013)

These goals, metrics, and evaluation methods may vary by

use the AUTM annual surveys from 1996 to 2010 and economic

agency as appropriate to that agency’s mission and types of

input–output models to find that the impact of university licens-

research activities, and may include the number and quality

ing to the U.S. economy during that period is in excess of $162.1

of, among other things, invention disclosures, licenses issued

billion and that jobs created over the same period range from 7000

on existing patents, Cooperative Research and Development

to 23,000 per year. Using those same AUTM surveys, Cardozo et al.

Agreements (CRADAs), industry partnerships, new products,

(2011) examine aggregate university activity and find that growth

and successful self sustaining spinoff companies created for

in revenues seems to have crested as technology transfer processes

such products (US White House Office of the Press Secretary,

have become more costly and less efficient. In one of the few recent

2011, p. 1–2).

publications using Economic Impact criteria and focusing on federal

As mentioned in the discussion relating to the addition of

agencies, Rowe and Temple (2011) conduct a smaller-scale study

the Public Value criterion, we see that economic effectiveness

focused on 11 firms from the semiconductor industry partnering

criteria should perhaps not pre-empt all others. Nevertheless, it is

with NIST. Their interviews and cost–benefit analysis show that

clearly the case that most technology transfer policy is to a large

the NIST projects had benefits well in excess of the full cost of the

extent rationalized by its economic impacts. The use of science

projects.

and technology policy and, specifically, technology transfer to

spur economic development has sound basis in many public laws

3.3. Political reward criterion for technology transfer effectiveness

and policy documents and strong support from the general public

(Seely, 2003).

The Political Reward criterion receives relatively little attention

in the literature but is worth mentioning. Parties to technology

transfer think in terms of possible political rewards accruing from

3

The White House, Office of the Press Secretary, “President Obama Launches

compliance or from ‘good citizen’ activities. During various on-

‘Educate to Innovate’ Campaign for Excellence in Science, Technology, Engineering

site interviews (Crow and Bozeman, 1998), university and federal

& Math (Stem) Education,” November 23, 2009, downloaded January 24, 2013, from:

http://www.whitehouse.gov/the-press-office/president-obama-launches-educate- laboratory officials have on many occasions made direct or, more

innovate-campaign-excellence-science-technology-en. frequently, indirect reference to the political pay-offs expected

40 B. Bozeman et al. / Research Policy 44 (2015) 34–49

from technology transfer activities. Technology transfer activities or validation from customers that there is a viable business model,

are often seen as a way to curry favor or enhance political support but also “no go” decisions that there is no market for the technology,

rather than as a means providing significant economic and social but that were reached more quickly without requiring significant

benefit. In this sense it is a means not an end (Rogers et al., 2001; expenditure of technology transfer resources, thereby presumably

Guston, 2007). reducing opportunity costs.

As noted previously (Bozeman, 2000), there are at least three The literature on university technology transfer gives atten-

possible avenues to political reward. In the least likely of scenarios, tion to this criterion, especially in relation to possible impacts on

a transfer agent is rewarded because the technology it has trans- individual researchers’ research agendas (Bercovitz and Feldman,

ferred has considerable national or regional socio-economic impact 2008), teaching responsibilities (Mendoza, 2007) and, more gener-

and the agent’s role in developing and transferring the technology is ally, organizational culture (Lee and Gaertner, 1994; Slaughter and

recognized by policy superiors and, in turn, the transferring entity Rhoades, 2004). Few recent studies focus directly on opportunity

is rewarded with increased funding or other resources. This sce- costs and technology transfer. However, Saavedra and Bozeman’s

nario is not unprecedented but does not commonly occur. In the (2004) study of federal laboratories and Woerter’s studies of

first place, few technologies have such an impact. But even when university–industry activity do employ contingency-oriented mod-

there are huge impacts from technology transfer, funding processes els and show that certain “portfolios” of technical activity are more

usually do not respond to even documented ‘big successes.’ productive than others. That is, while some federal laboratories are,

Another way in which the Political Reward criterion may yield because of their technical focus, able to engage in technology trans-

resource results for the transfer agent is through the transfer recip- fer activities with win-win results (for both the technology transfer

ient. Under this scenario, the organization or industry benefiting and for their other technical missions), other labs suffer declines in

from the technology transfer, communicates to policymakers the effectiveness in some of their technical missions with an increase

value of its interaction with the technology transfer partner. The in technology transfer.

policymaker then, in turn, rewards the transfer agent for being a

“good industrial partner.” There is evidence of such political reward 3.5. Scientific and technical human capital criterion for

but, understandably, it is based on rumors and anecdotes. technology transfer effectiveness

Probably the most common and realistic rationale under the

Political Reward criterion is for the transfer agent to be rewarded A premise of the Scientific and Technical Human Capital model

for the appearance of active and aggressive pursuit of technology is that one of the most critical objectives in almost all aspects

transfer and commercial success. In this case, the Political Reward of science and technology policy is building human and institu-

criterion turns out to be much the same as Out-the-Door: activity tional capabilities, even aside from particular accomplishments

is its own reward. Much bureaucratic behavior seems to support reflected in discrete knowledge and technology outputs (Bozeman

this view. For example, often federal laboratories are as active in et al., 2001). The focus of Scientific and Technical Human Capi-

publicizing their technology transfer and economic development tal (hereafter STHC) is on long-term capacity building. Indeed, a

activities as in actually doing the transfer work. deep understanding the value of scientific and technical knowledge

requires a view of the role of scientific and technical human cap-

3.4. Opportunity cost criterion for technology transfer ital in the capacity for producing scientific work (Audretsch and

effectiveness Stephan, 1999; Corolleur et al., 2004; Canibano˜ et al., 2008) and an

understanding that all such work is produced in networks (Casper

When considering technology transfer activities it is well worth and Murray, 2005). The formal and informal networks of scien-

recognizing that technology transfer is one of many missions of an tists, engineers and knowledge users depend upon the conjoining

agency or organization, and often not the one viewed as the most of equipment, material resources, organizational and institutional

important. For instance, in hundreds of interviews with federal lab- arrangements for work, and the unique human capital embodied

oratory scientists Crow and Bozeman (1998) found a wide range in individuals (Dietz and Bozeman, 2005; Rigby and Edler, 2005;

of perspectives on technology transfer, ranging from enthusiasm Ponomariov and Boardman, 2010). At any level, from the indi-

and avid participation to outright hostility and cynicism. Even as vidual scientist to organizational actor, network, or entire fields,

technology transfer activity is enhanced and nurtured, it remains knowledge value is capacity—capacity to create new knowledge

important to understand that technology transfer takes its place, and technology (Bozeman et al., 2001).

and often a secondary place, to missions such as the advance of basic Capacity is revealed through the changing patterns of the sci-

research and scientific theory, providing equipment and infrastruc- entific and technical human capital footprints individuals leave

ture for the growth of scientific knowledge, training scientists and behind throughout their careers. Dietz and Bozeman (2005) and

engineers, and, in the case of government agencies, ensuring the Gaughan and Ponomariov (2008) define STHC as the sum total

nation can perform its defense, national security, public health and of personal skills, knowledge, and the social resources scientists

energy missions. and engineers bring to, and develop from, their work. Thus, STHC

While it is easy enough to understand the fact of opportunity includes not only the individual human capital endowments tra-

costs in technology transfer it is not so easy to draw practical lessons ditionally included in labor models (e.g. Becker, 1964; Schultz,

about technology transfer measures and metrics. The National Sci- 1963), but also the individual scientist’s tacit knowledge (Polanyi,

ence Foundation (NSF) established the Innovation Corps (I-Corps) 1969; Senker, 1995), craft knowledge, and know-how (Bidault and

program in 2011 to speed technology transfer of NSF research, Fischer, 1994). STHC further includes the social capital (Coleman,

particularly in light of limitations on the ability of professors to 1988) that scientists inevitably draw upon in framing research and

successfully start up and run technology-based firms originating technological questions, creating knowledge, and developing social

out of their publicly-funded research. I-Corps puts together teams and economic certifications for knowledge (Fountain, 1998; Landry

of would-be entrepreneurs (such as students of these professors), et al., 2002).

the professors who were principal investigators on NSF grants, As mentioned, much of scientific and technical human capital

and mentors experienced in technology transfer and takes them is embedded in social and professional networks or technologi-

through a six-week training process using lean startup and cus- cal communities (Liyanage, 1995; Murray, 2002). These networks

tomer discovery processes reflected in the popular business press integrate and shape scientific careers. They provide knowledge of

(Ries, 2011; Blank, 2013). Success not only concerns “go” decisions scientists’ and engineers’ work activities, serve as resources for job

B. Bozeman et al. / Research Policy 44 (2015) 34–49 41

opportunities and job mobility, and reveal possible applications for educational mission of universities. In reflecting on possible

scientific and technical work products. Increasing STHC generally impacts of universities’ technology development and transfer

enhances individuals’ capacities while simultaneously increasing roles, former Harvard University president Derek Bok (2003, p.

the capacity of networks of knowledge and technology producers. 106) warns: “Even the appearance of hiring professors for commer-

Some technology transfer professionals, especially those in cial reasons will lower the morale of the faculty and diminish the

government agencies (Bozeman and Rogers, 2001; Rogers and reputations of the university[.]” The limited number of studies pro-

Bozeman, 1997) take the view that technology transfer, even if it viding systematic empirical evidence (Stephan, 2001; Ponomariov,

does not have immediate effects from discrete projects, helps build 2009; Bozeman and Boardman, in press) on the impact of uni-

capacity within either a geographic area, a scientific and techni- versity technology commercialization and transfer activities on

cal field or an institution (Fritsch and Kauffeld-Monz, 2010; Florida university educational missions shows that impacts are diverse,

et al., 2010). For these reasons, among others, Autio and Laamanen sometimes undermining education but in other cases augmenting

(1995) and Sala et al. (2011) argue that evaluation of technology the mission. But the criticism remains worth noting: leaders must

transfer is most appropriately directed to impacts on networks of be vigilant that the primary public value of universities, education,

interconnected scientific and commercial actors. not be undermined by the secondary economic value of technology

While there are no technology transfer assessments based commercialization and transfer. This thwarting of public values can

exclusively on an STHC model, there are a few studies in which happen in federal laboratories as well. For example, if technology

STHC plays a significant role. One study of Italian research centers transfer activities undermine national security then there has been

(Coccia and Rolfo, 2002) focuses on the complimentary roles of a supplanting of public values (Mowery, 1988; Aronowitz, 1999;

research, education, and training and documents interdependent Jaffe and Lerner, 2001; Kassicieh et al., 2002; Evans and Valdivia,

impacts. Edler and colleagues (2011), find that 950 German aca- 2012). Likewise, if the private entrepreneurship enabled under the

demics’ visits outside of their home country did not ‘crowd out’ but Stevenson-Wydler Act (U.S. Congress, 1980, 1984a,b, 1986) were to

rather complemented knowledge and technology transfer activities diminish the core research capabilities of federal laboratories’ cor-

to firms in Germany. Focusing on university researchers affiliated porate research mission, here, too, would be a thwarting of public

with interdisciplinary centers, Lin and Bozeman (2006) employ an values (see Coursey and Bozeman, 1992; Butler and Birley, 1998).

STHC model to identify the impacts of industrial interaction on Overall, the “public values” criterion can be thought of as the

university researchers’ careers and their productivity. In another “keep-your-eye-on-the-prize” criterion in the sense that it focuses

study employing an STHC model, but not for technology transfer on provision of beneficial public outcomes as opposed to the

assessment, Bozeman and Corley (2004) examine the impacts of lesser value of organizational goal achievement. To this end, as

university researchers’ collaborations on their accumulated STHC. previously mentioned, the public values criterion is consistent

Perhaps the only full scale STHC research assessments are those with recent emphasis on responsible research and innovation. For

produced by Youtie and colleagues (2006) and by Gaughan and example, von Schomberg (2013) anchored responsible innovation

Ponomariov (2008), both focusing on knowledge impacts from within the ethical promotion of social justice, sustainable devel-

NIH research centers. Youtie and colleagues employ qualitative opment, and socially desirable quality of life. Rayner et al. (2013)

methodologies to trace the growth of collaborations and network put forth “The Oxford Principles,” in the UK which emphasized

activity resulting from research sponsored by the NIH’s National public good regulation, public participation, research disclosure,

Institute of Child Health and Human Development. Gaughan and independent assessment, and governance of widescale deployment

Ponomariov provide a quantitative, time-series analysis (hazard of geoengineering. Roco et al. (2011) emphasized four character-

models) of university faculty curricula vita to show the impacts istics of responsible innovation: that it be transformative across

of research center affiliation on the accumulation of STHC. disciplines and sectors; that it consider equitable access and envi-

ronmental, health, and safety concerns; that it include participation

3.6. Public value criterion for technology transfer effectiveness across governmental agencies and other stakeholders; and that it

take a long-term perspective with measures that anticipate and

The term “public value” has many meanings and implications adapt. Randles et al. (2012) found that in a Trans-Atlantic panel of

(Bozeman, 2002, 2007; Benington and Moore, 2010). Some use the European and US nanotechnology societal researchers, those from

term as equivalent to the collective good, others in connection Europe were more apt to enter the concept through and

with the public interest, and still others as a sort of residual cat- principles while those from the US emphasized points of practical

egory for commodities not encompassed in either private value or intervention but both groups regarded beneficial outcomes beyond

markets (Jørgensen and Bozeman, 2007). At the broadest level, we moving bench science along to be important aspects of responsible

can begin with, and then build upon, a public values definition innovation.

provided elsewhere (Bozeman, 2007, p. 37): Public value is a difficult and elusive criterion in terms of eval-

uation. Recently, there have been efforts to move from the realm

“A society’s “public values” are those providing normative con-

of broad values discourse to application (Bozeman and Sarewitz,

sensus about (1) the rights, benefits, and prerogatives to which

2005; Slade, 2011; Valdivia, 2011). Bozeman and Sarewitz (2011)

citizens should (and should not) be entitled; (2) the obligations

suggest that concerns about economic productivity have been dom-

of citizens to society, the state and one another; (3) and the

inant in science and technology policies and their assessment and

principles on which and policies should be based.”

that there is a need for greater infusion of public values in science

While this definition has some merit for present purposes, it and technology policy. These researchers ground their work in pub-

shows that public values may be the most fundamental criterion lic value failure theory (Bozeman, 2002, 2007), and the theory also

upon which to evaluate nearly any public policy. Its practical use offers some purchase for practically evaluating technology transfer

as a criterion for technology transfer is quite limited, however in terms of public values. For instance, Valdivia’s (2011) approach

would public value possibly be subverted in the case of technology shows promise. He employs the Public Value model in connection

transfer? A couple of examples will perhaps suffice. In the case of with university technology transfer. Specifically, he evaluates the

university–industry technology transfer, a cornerstone of so-called Bayh-Dole Act using public value failure theory and its attendant

“academic capitalism,” some critics (Kleinman, 2003; Slaughter analytical approach, Public Value Mapping (PVM). The overarching

and Rhoades, 2004; Henkel, 2005) have alleged that the increased idea is that public values can be identified by analyzing appropri-

commercialization of universities has undermined the core ate documents and other media. Once values are identified, it is

42 B. Bozeman et al. / Research Policy 44 (2015) 34–49

then possible to ascertain whether a particular policy has failed to specialists available to that purpose. For example, in the case of

achieve them by analyzing the corresponding social outcome indi- Transfer Recipient Impacts there may be desirable changes that

cators (notably, these are sometimes difficult to measure and/or do not immediately and directly translate into market impacts.

develop). Another application of PVM includes identifying the pro- For example, in working with a particular company a federal lab

cess that links a particular policy or program to a public value may have a strong impact on training firms’ personnel, benefits

outcome. Maricle (2011) labels these sets of linkages an organi- that will never show up directly and obviously in market indi-

zation’s “public value logic”. This idea is particularly pertinent to cators but that nonetheless have the potential to provide major

developing a public value failure approach to evaluating technol- advantages. Similarly, technology transfer recipients often ben-

ogy transfer activities. Although these activities are not always efit enormously from using state-of-the-art or even unique

expected to be direct mechanisms for achieving public value, they scientific equipment and instruments made available to them

are linked to public value pursuits through the public value logic by a transfer agent. Such benefits are out-the-door impacts, not

that the organization or entity employs. Public value failure the- (direct) market impacts and are well worth capturing. (For a

ory, therefore, suggests mapping public value logics and using case discussion of the indirect impacts of federal laboratories on

study methodology to assess whether technology transfer activities industry partners see Adams et al., 2003.)

successfully fulfill their specified role. (2) Identification of expected ranges of impact. A common problem

for most evaluation efforts, including attempts to evaluate tech-

4. Conclusions and recommendations nology transfer impacts, is the failure to understand the domain

of influence of the “intervention” (Midgley, 2006; Schalock and

In this concluding section a number of recommendations are Bonham, 2003). If at the beginning of a technology transfer

provided on the basis of implications of the literature reviewed effort there is at least some attention to providing a rationale

here. They focus on general issues in assessing technology transfer for the expected domain of influence of the transfer then there

effectiveness. is a guidepost to help one understand the diffusion of impacts.

Absent such guideposts, it is altogether natural to claim impacts

(1) Making the most of Out-the-Door. While the Out-the-Door model of great breadth when, in fact, the technology transfer activity is

of effectiveness is not ideal, it is realistic and useful (Geisler, one significant event in a multi-causal chain of events. Equally

1994; Lepori, 2006). For agencies able to develop large scale, important, having a pre-established hypothesis about domain

contract-out, resource-intensive technology transfer assess- of influence leads to subsequent cues for obtaining evidence of

ment regimes, Out-the-Door criteria can be improved upon. influence. An impact theory is a useful precursor to any attempt

But for agencies facing personnel scarcity, limited in-house to measure impact. In developing impact measures, the ana-

evaluation personnel, and no budget increment for external lyst does well to ask questions such as: (1) “What set of causal

evaluation contracting, it seems likely that the Out-the-Door assumptions need be true for impacts to occur?” (2) “What is

model will continue to be the primary basis of any mea- the likely chronology of impact, when should benefits begin to

surement activity (Geisler, 1994). Given these realities, the occur and why then?” (3) “What are the alternative causes that

recommendation is for Out-the-Door done right. Ways to do could result in this impact that seems to be caused by our tech-

this include the following: nology transfer efforts”? (i.e. alternative plausible hypotheses).

In recognition of the fact that some technology transfer out- Indeed, it may be worthwhile to routinely pose or even require

comes are going to occur in streams of benefits and costs answers to these and similar questions as part of any effort to

realized over time, there is no more vital Out-the-Door activ- measure out-the-door technology impacts or market impacts.

ity than providing good periodic benchmarks. If measures of One way to accomplish this is to increase the use of logic

activity are going to dominate metrics, then those measures models and mapping techniques. Systems of indicators are

need to be as precise as possible and need to be tracked over more valuable than lone, discrete indicators. Systems of indica-

time. A good number of the agencies’ responses recognize the tors brought together in logic models or mapping systems are

importance of quality, valid benchmark measures. For example, more valuable yet (Cooksy et al., 2001; Schalock and Bonham,

in the U.S. Department of Energy’s plans (U.S. Department of 2003). Logic models require attention to explicit assump-

Energy, 2012) for technology transfer metrics, one of the crite- tions, requiring the analyst not simply to list but to show

ria is “patenting effectiveness.” But rather than simply reporting the presumed causal connections among inputs (e.g. federal

the number of patents, they plan to report the ratio of patents laboratory technology), activities (e.g. marketing technologies),

in a given year to applications filed for a three year base outputs (e.g. licenses), and impacts (e.g. new products devel-

period, using a rolling three-year average as new metrics are oped by participating companies). Many textbooks on logic

reported. models include frameworks with specific templates that assure

Surprisingly few sets of Out-the-Door measures and metrics that temporally-relevant questions are asked and that causal

developed thus far give any consideration to the resources assumptions are explicated and inter-related (see Frechtling,

agencies and their facilities bring to technology transfer activ- 2007, p. 65–78).

ities. It is not useful, and may even be counterproductive, to (3) Further development of scientific and technical human capital indi-

show that the number of licenses has declined over a given cators. Research evaluators and program managers have known

time period when, in fact, that decline may be owing to a sharp for some time that it is often at least as valuable to enhance the

reduction of the technology transfer personnel available. For capacities of organizations or knowledge producing communi-

any valid inference about effectiveness, activity measures must ties as to provide beneficial direct outputs. If a small company

relate to resource measures. develops the capacity to use computer aided tools,

Perhaps it is time to move away from what are referred to that capacity may provide a stream of benefits stretching out

here as Pure Out-the-Door measures. While it is sometimes for many years. Some R&D managers assume that if knowledge

exceedingly difficult to document particular causes and effects, producers’ capacity is fully developed then good things hap-

it is possible and useful to at least develop measures of Out- pen with the level of production and the quality of outputs

the-Door Transfer Agent Impacts and Out-the-Door Transfer and, indeed, there is at least some evidence for this capac-

Recipient Impacts. These types of measures can likely be gath- ity focus (e.g. Ponomariov and Boardman, 2010). Furthermore,

ered and recorded even absent a large cadre of evaluation it is sometimes easier to develop valid measures of scientific

B. Bozeman et al. / Research Policy 44 (2015) 34–49 43

and technical human capital than valid measures of economic transferred. The Contingent Effectiveness Model can be helpful

impact in over-determined systems of interrelated economic in this regard. While all metrics are important – particularly

producers and consumers. Thus, for example, one could trace for example “out-the-door” metrics – one can expect differ-

the career trajectories of researchers who have interacted with ences in emphasis for the other metrics. Universities could

a federal laboratory, comparing those researchers to a group reasonably be expected to prioritize scientific and technical

similar in every other respect except that they have not inter- human capital measures and economic development measures.

acted with a federal laboratory. Using the laboratory interaction Government laboratories could be presumed to place greater

as an inflection point in time, it is possible to compare differ- weight on political considerations and public value effective-

ences in one set of researchers (those interacting with the labs) ness. Companies could be expected to be particularly concerned

with the other (who have not). With a sufficient sample size with market impact and opportunity costs. A pathway for future

for valid “treatment” and “comparison” groups, any difference research is to examine these anticipated differences and, if pos-

between the two sets’ career accomplishments could be owing itively supported by such research, should be incorporated into

to the resources and activities of their interactions with the fed- more customized evaluation designs.

eral laboratories. Previous studies have used curricula vitae as (6) Increase attention to and development of public value effective-

a convenient means of examining the impacts of such events ness criteria. As indicated by the adaptation to the Continent

on researchers’ careers (for examples of such applications see Effectiveness Model, the idea of systematic evaluation of tech-

Bozeman and Gaughan, 2007; Canibano˜ et al., 2008; Lepori and nology transfer activities in terms of Public Value is relatively

Probst, 2009). new. However, the emphasis that many agencies place on edu-

(4) Correlate process reforms and activity measures. If we take these cation and outreach impacts are to a certain extent related to

two categories (process and activity) of indicators together they Public Value criteria as are anecdotal reports of social impacts

comprise at least 90% of typically used performance metrics. of transferred technologies. Challenges to further development

The problem is that the two are not, under most plans, brought of public value indicators lie in their inherent measurement

together. While everyone recognizes that correlation is not cau- difficulties (e.g. Bozeman and Sarewitz, 2011; Gupta, 2002) as

sation, it is at least of heuristic value to track activity measures well as the relatively resource intensive methods necessary

against implemented changes in technology transfer processes to capture a picture of an organization’s public value pur-

and managerial approaches. suits. In spite of these difficulties, efforts to analyze technology

(5) Distinguish the relative importance of indicators associated with transfer through the lens of public value are bearing some

technology transfer by universities, government laboratories, and fruit (Costa-Font and Mossialos, 2006; Sorensen and Chambers,

industry. Studies of technology transfer programs should adapt 2008; Rubenstein, 2003), and theories such as public value

and customize their assessments to reflect differences in the failure theory point to potential pathways to systematically

orientation of the organizational home for the technology being develop and explore public value effectiveness measures.

Appendix A. Technology transfer literature organized by

categories of the revised contingent effectiveness model

Effectiveness In-text citation Full citation Relevant findings

criterion

Out-the-door Rogers et al. Rogers, Evertt, Carayannis, Elias G., Kurihara, Firms are critical of the amount of time and complexity

(1998) Kazuo, Allbritton, Marcel M., 1998. Cooperative necessary to form a CRADA.

Research and Development Agreements (CRADAs)

as technology transfer mechanisms. R&D

Management 28 (2), 79.

Out-the-Door Bercovitz et al. Bercovitz, Janet, Feldman, Maryann, Feller, Irwin, Differences in organizational structure and capacity

(2001) Burton, Richard, 2001. Organizational Structure as result in differences in technology transfer activities in

a Determinant of Academic Patent and Licensing terms of patenting, leveraging, and the likelihood that

Behavior: An Exploratory Study of Duke, Johns customer firms overlap across university units.

Hopkins, and Pennsylvania State Universities. The

Journal of Technology Transfer 26 (1–2), 21–35.

Out-the-Door Jaffe and Lerner Jaffe, Adam B., Lerner, Josh, 2001. Reinventing Federal technology transfer legislation and initiatives

(2001) public R&D: patent policy and the since the 1980s have had a significant effect on the

commercialization of national laboratory number of patents produced by DOE labs without a

technologies. RAND Journal of Economics (RAND commensurate decrease in patent quality.

Journal of Economics) 32 (1), 167–198.

Out-the-Door Thursby et al. Thursby, Jerry G., Jensen, Richard, Thursby, Marie For patents and sponsored research size of the

(2001) C., 2001. Objectives, Characteristics and Outcomes technology transfer office is positively associated with

of University Licensing: A Survey of Major U.S. higher levels. For licenses number of disclosures, size

Universities. The Journal of Technology Transfer 26 of the technology transfer office, and whether the

(1–2), 59–72. university has a medical school are statistically

significant. Also, the stage of technology development,

size of the tech transfer office, and quality of the

researchers is associated with greater royalty values.

Out-the-Door Thursby and Thursby, Jerry G., Kemp, Sukanya. (2002). Growth Licensing has increased for reasons other than overall

Kemp (2002) and productive efficiency of university intellectual increases in university resources.

property licensing. Research Policy 31 (1), 109–124.

44 B. Bozeman et al. / Research Policy 44 (2015) 34–49

Effectiveness In-text citation Full citation Relevant findings

criterion

Out-the-Door Adams et al. Adams, James D., Chiang, Eric P., Jensen, Jeffrey L., CRADAs stimulate industrial patents and industrial

(2003) 2003. The Influence of Federal Laboratory R&D on R&D, and do so to a much greater extent than other

Industrial Research. Review of Economics & tech transfer mechanisms.

Statistics 85 (4), 1003–1020.

Out-the-Door Friedman and Friedman, Joseph, Silberman, Jonathan, 2003. Incentives for researchers, university location within a

Silberman University Technology Transfer: Do Incentives, region with a concentration of high technology firms, a

(2003) Management, and Location Matter? The Journal of clear technology transfer mission, and previous

Technology Transfer 28 (1), 17–30. technology transfer experience are positively

associated with technology transfer performance.

Out-the-Door Powers (2003) Powers, Joshua B., 2003. Commercializing Universities with older technology transfer offices,

Academic Research: Resource Effects on higher quality researchers, and higher levels of R&D

Performance of University Technology Transfer. funding produce more patents.

The Journal of Higher Education 74 (1), 26–50. Those with older and larger technology transfer offices

produce more licenses.

Researcher quality and technology transfer office size

are positively associated with license revenue.

Out-the-Door Siegel et al. Siegel, Donald S., Waldman, David, Link, Albert, Invention disclosures are positively associated with

(2003) 2003. Assessing the impact of organizational both number of licenses and license revenue. The size

practices on the relative productivity of university of the technology transfer office staff results in more

technology transfer offices: an exploratory study. licenses but not more revenue. Spending on external

Research Policy 32 (1), 27–48. lawyers reduces the number of agreements but

increases license revenue.

Out-the-Door Chapple et al. Chapple, Wendy, Lockett, Andy, Siegel, Donald, University technology transfer offices in the U.K. are

(2005) Wright, Mike, 2005. Assessing the relative found to have low levels of efficiency and decreasing

performance of U.K. university technology transfer returns to scale.

offices: parametric and non-parametric evidence.

Research Policy 34 (3), 369–384.

Out-the-Door Link and Siegel Link, Albert N., Siegel, Donald S., 2005. Generating When licensing activities are the dependent variable,

(2005) science-based growth: an econometric analysis of organizational incentives (financial incentives) impact

the impact of organizational incentives on technology transfer performance.

university–industry technology transfer. The

European Journal of Finance 11 (3), 169–181.

Out-the-Door Anderson et al. Anderson, Timothy R., Daim, Tugrul U., Lavoie, There are both efficient and inefficient universities in

(2007) Francois F., 2007. Measuring the efficiency of terms of comparing research expenditures and

university technology transfer. Technovation 27 technology transfer outputs.

(5), 306–318. Universities with medical schools tend to be less

efficient than those without medical schools.

Out the Door Mowery and Mowery, David C., Ziedonis, Arvids A., 2007. Materials Transfer Agreements at universities do not

Ziedonis (2007) Academic patents and materials transfer appear to inhibit patenting and licensing activities.

agreements: substitutes or complements? The

Journal of Technology Transfer 32 (3), 157–172.

Out-the-Door Fukugawa Fukugawa, Nobuya, 2009. Determinants of Determinants of licensing activity vary based on the

(2009) licensing activities of local public technology phase of technology transfer.

centers in Japan. Technovation 29 (12), 885–892. Budget size and previous technology transfer

experience does not affect licensing. Employing high

quality scientists promotes licensing of granted

patents. Organizational efforts aimed at encouraging

scientists to understand the needs of small

increases royalty revenues.

Out-the-Door Swamidass and Swamidass, Paul M., Vulasa, Venubabu, 2009. Why Lack of resources in terms of staff and budget result in

Vulasa (2009) university rarely produce income? universities focusing on filing patent applications

Bottlenecks in university technology transfer. The rather than licensing technologies.

Journal of Technology Transfer 34 (4), 343–363.

Out-the-Door Park et al. Park, Jong-Bok, Ryu, Tae-Kyu, Gibson, David V., Membership in research consortia can increase the

(2010) 2010. Facilitating public-to-private technology technology transfer performance (in terms of invention

transfer through consortia: initial evidence from disclosures, patents, licenses executed, and royalties)

Korea. The Journal of Technology Transfer 35 (2), of participating public sector research institutions.

237–252.

Out-the-Door Heisey and Heisey, Paul W., Adelman, Sarah W., 2011. The study finds conflicting evidence of the short-term

Adelman Research expenditures, technology transfer effect of research expenditures on licensing revenues.

(2011) activity, and university licensing revenue. The Both early initiation of a technology transfer program

Journal of Technology Transfer 36 (1), 38–60. and technology transfer staff size positively affect

expected licensing revenues; however, they appear to

be substitutes.

Out-the-Door Bozeman and Bozeman, Barry, Crow, Michael, 1991. Red tape and Labs involved in tech transfer do not have higher levels

and Market Crow (1991a) technology transfer in US government laboratories. of red tape than other labs.

Impact The Journal of Technology Transfer 16 (2), 29–37. Out the door measures of tech transfer success are

associated with low levels of perceived red tape, and

measures of market impact are associated with low

levels of actual red tape in obtaining project funding

and low-cost equipment.

Out-the-Door Bozeman and Bozeman, Barry, Coker, Karen, 1992. Assessing the Multi-faceted, multi-mission labs with low

and Market Coker (1992) effectiveness of technology transfer from US bureaucratization, ties to industry, and a commercial

Impact government R&D laboratories: the impact of focus in project selection perform better on

market orientation. Technovation 12 (4), 239–255. out–the-door and market impact effectiveness measures.

B. Bozeman et al. / Research Policy 44 (2015) 34–49 45

Effectiveness In-text citation Full citation Relevant findings

criterion

Out-the-Door Bozeman Bozeman, Barry, 1994. Evaluating government There is wide variation in labs in regards to out the

and Market (1994) technology transfer: Early impacts of the door and market impact measures of effectiveness

Impact cooperative technology paradigm. Policy Studies with some evidence supporting a concentration of

Journal 22 (2), 322–337. success in a few labs.

Lab technology transfer strategy and lab mission are

correlated with effectiveness.

Different measures of success do not correlate well

with each other.

Out-the-Door Caldera and Caldera, Aida, Debande, Oliver, 2010. Performance The size of the technology transfer office in 52 Spanish

and Market Debande of Spanish universities in technology transfer: An universities is associated with greater R&D income,

Impact (2010) empirical analysis. Research Policy 39 (9), spinoffs, and licensing activity although not licensing

1160–1173. revenue.

Out-the-Door Siegel et al. Siegel, D.S., Veugelers, R., Wright, M., 2007. The researchers collect both quantitative and

and Market (2007) Technology transfer offices and commercialization qualitative data on the relative efficiency of university

Impact of university intellectual property: performance technology transfer offices. They find that differences

and policy implications. Oxford Review of in performance can be attributed to both

Economic Policy 23 (4), 640–660. environmental and institutional factors and may also

depend on organizational practices.

Out-the-Door, Rogers et al. Rogers, Everett, Takegami, Shiro, Yin, Jing, 2001. Articles in scientific journals are not an effective

Market (2001) Lessons learned about technology transfer. technology transfer mechanism. Spin-offs are an

Impact, and Technovation 21 (4), 253–261. effective technology transfer mechanism.

Economic Organizations that provide assistance with technology

Development transfer, coupled with favorable entrepreneurial leave

policies at federal labs, facilitate the growth of

spin-offs.

Out-the-Door Carlsson and Carlsson, Bo, Fridh, Ann-Charlotte, 2002. Organizational structure variables have an impact on

and Fridh (2002) Technology transfer in United States universities. technology transfer measures of licenses, patents, and

Economic Journal of Evolutionary Economics 12 (1/2). start-ups. However, based on their findings the authors

Development argue for technology transfer success to be considered

in broader context such as overall goals of the

organization.

Market Impact Cohen et al., Cohen, Wesley M., Nelson, Richard R., Walsh, John In general, public research plays an important role in

2002 P., 2002. Links and Impacts: The Influence of Public private sector manufacturing R&D. This impact flows

Research on Industrial R&D. Management Science through a variety of formal and informal channels and

48 (1), 1–23. tends to be greater for applied research rather than

basic research. There are some differences in impacts

across industries as well, but few, if any, systematic

differences between high tech industries and other

industries.

Market Impact Hertzfeld Hertzfeld, Henry R., 2002. Measuring the Economic For companies that developed spin-off products from

(2002) Returns from Successful NASA Life Sciences NASA investments the largest benefits accrued to large

Technology Transfers. The Journal of Technology companies. Many small companies reported profitable

Transfer 27 (4), 311–320. products and benefits as well, but lacked the resources

to expand to large scale production.

Market Impact Cardozo et al., Cardozo, Richard, Ardichvili, Alexandre, Strauss, Conceptualizing universities engaged in technology

2011 Anthony, 2011. Effectiveness of university transfer activities as an industry, results show that

technology transfer: an organizational population industry growth is slowing and technology transfer

ecology view of a maturing supplier industry. The processes are becoming less efficient.

Journal of Technology Transfer 36 (2), 173–202.

Market Impact Roessner et al. Roessner, David, Bond, Jennifer, Okubo, Sumiye, Summing over a 15 year period, the authors estimate

(2013) Planting, Mark, 2013. The economic impact of that the impact of university licensing on the U.S.

licensed commercialized inventions originating in economy is at least $162.1 billion. Estimates for jobs

university research. Research Policy 42 (1), 23–34. created per year over the period range from 7000 to

23,000. Models estimated with different substitution

rates still yield large effects on GDP.

Market Impact Hartmann and Hartmann, G. Bruce, Masten, John, 2000. Profiles of Small manufacturers tend to have faster growth rates

and Masten (2000) State Technological Transfer Structure and Its in states that focus technology transfer assistance on

Economic Impact on Small Manufacturers. The Journal of small firms.

Development Technology Transfer 25 (1), 83–88.

Market Impact Lindelöf and Lindelöf, Peter, Löfsten, Hans, 2004. Proximity as a New technology based firms located in university

and Löfsten (2004) Resource Base for Competitive Advantage: science parks exhibit a competitive advantage over

Economic University–Industry Links for Technology Transfer. firms not located in science parks in terms of product

Development The Journal of Technology Transfer 29 (3–4), development.

311–326.

Market Impact Coccia and Coccia, Mario, Rolfo, Secondo, 2002. Technology Lab rankings change depending on which measure of

and Scientific Rolfo (2002) transfer analysis in the Italian National Research technology transfer effectiveness is employed.

and Council. Technovation 22 (5), 291–299. Technological labs (applied science) perform better in

Technical terms of market-oriented tech transfer and

Human non-technological labs (economics and natural

Capital sciences) perform better in terms of

education-oriented tech transfer.

Market Impact Rowe and Rowe, Brent R., Temple, Dorota S., 2011. Economic impact estimates suggest that the transfer of

and Temple (2011) Superfilling technology: transferring knowledge to superfilling knowledge generated by NIST to industry

Opportunity industry from the National Institute of Standards were an efficient use of public resources.

Cost and Technology. The Journal of Technology

Transfer 36 (1), 1–13.

46 B. Bozeman et al. / Research Policy 44 (2015) 34–49

Effectiveness In-text citation Full citation Relevant findings

criterion

Economic Markusen and Markusen, Ann, Oden, Michael, 1996. National Barriers to business incubation and start-up at federal

Development Oden (1996) laboratories as business incubators and region labs are identified and suggestions for improvement are

builders. The Journal of Technology Transfer 21 offered.

(1–2), 93–108.

Economic Phillips (2002) Phillips, Rhonda G., 2002. Technology business Technology business incubators have widely varying

Development incubators: how effective as technology transfer rates of technology transfer, but overall levels are not as

mechanisms? Technology in Society 24 (3), high as expected.

299–316.

Economic Shane and Shane, Scott, Stuart, Toby, 2002. Organizational Founder’s social capital is key to the outcome for the

Development Stuart (2002) Endowments and the Performance of University new venture; firms with founders that have direct and

Start-ups. Management Science 48 (1), 154–170. indirect relationships with venture investors are more

likely to receive funding and less likely to fail.

Economic O’Shea et al. O’Shea, Rory P., Allen, Thomas J., Chevalier, Arnaud, Previous spinoff development, the presence of leading

Development (2005) Roche, Frank, 2005. Entrepreneurial orientation, researchers, the magnitude and nature of financial

technology transfer and spinoff performance of resources, and the amount of resources invested in

U.S. universities. Research Policy 34 (7), 994–1009. technology transfer office personnel at universities all

increase current spinoff activity.

Economic Golob (2006) Golob, Elyse, 2006. Capturing the Regional Universities that view their technology transfer

Development Economic Benefits of University Technology functions as revenue generators produce fewer start-ups

Transfer: A Case Study. The Journal of Technology than universities that have economic development as an

Transfer 31 (6), 685–695. objective. Also, entrepreneurs make location decisions

based on a variety of factors including existing

relationships with the licensing entity.

Economic Gulbranson Gulbranson, Christine A., Audretsch, David B., The authors discuss the utility of proof of concept

Development and Audretsch 2008. Proof of concept centers: accelerating the centers to facilitating transfer of university innovations.

(2008) commercialization of university innovation. The

Journal of Technology Transfer 33 (3), 249–258.

Economic Festel (2012) Festel, Gunter, 2012. Academic spin-offs, corporate Start-ups, spin-offs, and spin-outs are legitimate

Development spin-outs and company internal start-ups as mechanisms for technology transfer.

technology transfer approach. The Journal of

Technology Transfer, 1–17.

Economic Brown (1998) Brown, Kenneth M., 1998. Sandia’s Science Park: A Sandia’s science park presents a model of technology

Development new concept in technology transfer. Issues in transfer that requires different evaluation metrics than

and Scientific Science & Technology 15 (2). technology transfer under a CRADA. and Technical Human

Capital

Scientific and Edler et al. Edler, Jacob, Fier, Heide, Grimpe, Christoph, 2011. German academics’ visits outside of their home country

Technical (2011) International scientist mobility and the locus of did not ‘crowd out’ but rather complemented knowledge

Human knowledge and technology transfer. Research and technology transfer activities to firms in Germany.

Capital Policy 40 (6), 791–805.

Market Saavedra and Saavedra, Pablo, Bozeman, Barry, 2004. The Technology transfer effectiveness is increased when the

Impact/Opportunity Bozeman “Gradient Effect” in Federal Laboratory–Industry lab and firm play different but not far removed roles on

Cost (2004) Technology Transfer Partnerships. Policy Studies the basic-applied-development spectrum.

Journal 32 (2), 235–252.

Opportunity Woerter (2012) Woerter, Martin, 2012. Technology proximity Technology proximity (work in the same patent class)

Cost between firms and universities and technology fosters technology transfer intensity between firms and

transfer. The Journal of Technology Transfer 37 (6), universities. This is the case especially for smaller firms.

828–866. Also, if technology proximity is low, but expertise at a

university is high then technology transfer intensity is

increased.

Public Value Rubenstein Rubenstein, Kelly Day, 2003. Transferring Public USDA’s patent licensing is not revenue driven, and it

(2003) Research: The Patent Licensing Mechanism in does not appear to have altered the agency’s research

Agriculture. The Journal of Technology Transfer 28 priorities. Licenses vary in terms of four social benefits:

(2), 111–130. food safety, human nutrition, human health, and

environmental/natural resource protection. No evidence

is found of concentration of licenses in only a few firms.

Public Value Costa-Font and Costa-Font, Joan, Mossialos, Elias, 2006. The Public The knowledge and beliefs of individuals as well as

Mossialos as a Limit to Technology Transfer: The Influence of information channels affect attitudes towards new

(2006) Knowledge and Beliefs in Attitudes towards applications of biotechnology in the UK.

Biotechnology in the UK. The Journal of Technology

Transfer 31 (6), 629–645.

Public Value Sorensen and Sorensen, Jill Ann Tarzian, Chambers, Donald A., The authors suggest metrics that could be used to

Chambers 2008. Evaluating academic technology transfer evaluate technology transfer performance in terms of

(2008) performance by how well access to knowledge is increased access to knowledge.

facilitated—defining an access metric. The Journal

of Technology Transfer 33 (5), 534–547.

Public Value Bozeman and Bozeman, Barry, Sarewitz, Daniel, 2011. Public Suggested framework for including public values in

Sarewitz Value Mapping and Evaluation. science policy evaluation.

(2011) Minerva: A Review of Science, Learning & Policy 49

(1), 1–23.

Political Bozeman and Bozeman, Barry, Crow, Michael, 1991b. Technology Influence from political authority is a major determinant

Crow (1991b) transfer from U.S. government and university R&D of technology transfer activity, specifically whether the

laboratories. Technovation 11 (4), 231–246. technology is transferred to government or industry.

B. Bozeman et al. / Research Policy 44 (2015) 34–49 47

References Breznitz, D., Taylor, M., 2009. The communal roots of entrepreneurial-technological

growth? Social fragmentation and the economic stagnation of Atlanta’s IT clus-

ter. In: Industry Studies Association Annual Meeting, Chicago.

Adams, J.D., Chiang, E.P., Jensen, J.L., 2003. The influence of federal laboratory R&D

Canibano,˜ C., Otamendi, J., Andújar, I., 2008. Measuring and assessing researcher

on industrial research. Review of Economics and Statistics 85 (4), 1003–1020.

mobility from CV analysis: the case of the Ramón y Cajal Programme in Spain.

Adomavicius, G., Bockstedt, J., Gupta, A., Kauffman, R.J., 2008. Understanding

Research Evaluation 17 (1), 17–31.

evolution in technology ecosystems. of the ACM 51 (10),

117–122. Caldera, A., Debande, O., 2010. Performance of Spanish universities in technology

transfer: an empirical analysis. Research Policy 39 (9), 1160–1173.

Agrawal, A.K., 2003. University-to-industry : literature review

Cardozo, R., Ardichvili, A., Strauss, A., 2011. Effectiveness of university technol-

and unanswered questions. International Journal of Management Reviews 3 (4),

285–302. ogy transfer: an organizational population ecology view of a maturing supplier

industry. The Journal of Technology Transfer 36 (2), 173–202.

Albors-Garrigos, J., Hervas-Oliver, J.L., Hidalgo, A., 2009. Analysing high technology

Carlsson, B., Fridh, A.-C., 2002. Technology transfer in United State universities.

adoption and impact within public supported high tech programs: an empirical

Journal of Evolutionary Economics 12 (1/2).

case. The Journal of High Technology Management Research 20 (2), 153–168.

Casper, S., Murray, F., 2005. Careers and clusters: analyzing the career network

Albors, J., Hervas, J.L., Hidalgo, A., 2006. Analysing high technology diffusion and pub-

dynamic of biotechnology clusters. Journal of Engineering and Technology Man-

lic transference programs: the case of the European game program. The Journal

agement 22 (1), 51–74.

of Technology Transfer 31 (6), 647–661.

Chapple, W., Lockett, A., Siegel, D., Wright, M., 2005. Assessing the rel-

Anderson, Timothy R., Daim, Tugrul U., Lavoie, Francois F., 2007. Measuring the

ative performance of U.K. university technology transfer offices: para-

efficiency of university technology transfer. Technovation 27 (5), 306–318,

http://dx.doi.org/10.1016/j.technovation.2006.10.003. metric and non-parametric evidence. Research Policy 34 (3), 369–384,

http://dx.doi.org/10.1016/j.respol.2005.01.007.

Aronowitz, J.D., 1999. Controlling Militarily Significant Emerging Technologies.

Chesbrough, H., 2003. Open innovation: the new imperative for creating and prof-

Army War College, Carlisle, PA.

iting from technology. Harvard Business School Press, Boston.

Audretsch, D.B., Stephan, P., 1999. How and why does knowledge spill over in

Coccia, M., Rolfo, S., 2002. Technology transfer analysis in the Italian national

biotechnology? In: Audretsch, D.B., Thurik, R. (Eds.), Innovation, Industry Evo-

research council. Technovation 22 (5), 291–299.

lution and Employment. Cambridge University Press, Cambridge, pp. 216–229.

Cohen, W.M., Nelson, R.R., Walsh, J.P., 2002. Links and impacts: the influence of

Autio, E., Laamanen, T., 1995. Measurement and evaluation of technology transfer:

public research on industrial R&D. Management Science 48 (1), 1–23.

review of technology transfer mechanisms and indicators. International Journal

Coleman, J.S., 1988. Social capital in the creation of human capital. American Journal

of Technology Management 10 (7–8), 7–8.

of Sociology 94, S95–S120.

Becker, G.S., 1964. Human Capital Theory. Columbia, New York.

Cooksy, L.J., Gill, P., Kelly, P.A., 2001. The program logic model as an integrative

Bercovitz, J.E., Feldman, M.P., 2008. Academic entrepreneurs: organizational change

framework for a multimethod evaluation. Evaluation and program planning 24

at the individual level. Organization Science 19 (1), 69–89.

(2), 119–128.

Beierle, T.C., Konisky, D.M., 2000. Values, conflict, and trust in participatory environ-

Corolleur, C.D., Carrere, M., Mangematin, V., 2004. Turning scientific and technolog-

mental planning. Journal of Policy analysis and Management 19 (4), 587–602.

ical human capital into economic capital: the experience of biotech start-ups in

Benington, J., Moore, M.H. (Eds.), 2010. Public Value: Theory and Practice. Palgrave

France. Research Policy 33 (4), 631–642.

Macmillan, New York.

Costa-Font, J., Mossialos, E., 2006. The public as a limit to technology transfer:

Bidault, F., Fischer, W.A., 1994. Technology transactions: networks over markets.

the influence of knowledge and beliefs in attitudes towards biotechnol-

R&D Management 24 (4), 373–386.

ogy in the UK. The Journal of Technology Transfer 31 (6), 629–645,

Blank, S., 2013. Why the lean start-up changes everything. Harvard Business Review

http://dx.doi.org/10.1007/s10961-006-0019-3.

91 (5), 64–68.

Coursey, D., Bozeman, B., 1992. Technology transfer in US government and university

Bok, D., 2004. Universities in the Marketplace: The Commercialization of Higher

laboratories: advantages and disadvantages for participating laboratories. IEEE

Education. Princeton University Press, Princeton, N.J.

Transactions on Engineering Management 39 (4), 347–351.

Bozeman, B., 1994. Evaluating government technology transfer: early impacts of the

Crow, M., Bozeman, B., 1998. Limited by Design: R&D Laboratories in the U.S.

‘cooperative technology paradigm’. Policy Studies Journal 22 (2), 322–337.

National Innovation System. Columbia University Press, New York.

Bozeman, B., 1997. Commercialization of federal laboratory technology: results of a

Dietz, J.S., Bozeman, B., 2005. Academic careers, patents, and productivity: industry

study of industrial partners. In: Oakey, R.P. (Ed.), New Technology-Based Firms

experience as scientific and technical human capital. Research Policy 34 (3),

in the 1990, vol. 3. Paul Chapman Publishing Limited, London, pp. 127–139.

349–367.

Bozeman, B., 2000. Technology transfer and public policy: a review of research and

Edler, J., Fier, H., Grimpe, C., 2011. International scientist mobility and the locus of

theory. Research Policy 29 (4), 627–655.

knowledge and technology transfer. Research Policy 40 (6), 791–805.

Bozeman, B., 2002. Public-value failure: when efficient markets may not do. Public

Etzkowitz, H., Leydesdorff, L., 2000. The dynamics of innovation: from National Sys-

Administration Review 62 (2), 145–161.

tems and Mode 2 to a Triple Helix of university–industry–government relations.

Bozeman, B., 2007. Public Values and Public Interest: Counterbalancing Economic

Research Policy 29 (2), 109–123.

Individualism. Georgetown University Press, Washington, D.C.

Evans, S.A., Valdivia, W.D., 2012. Export controls and the tensions between academic

Bozeman, B., Boardman, C., 2014. Academic faculty working in university research

freedom and national security. Minerva, 1–22.

centers: neither capitalism’s slaves nor teaching fugitives. The Journal of Higher

Festel, G., 2012. Academic spin-offs, corporate spin-outs and company internal

Education (in press).

start-ups as technology transfer approach. The Journal of Technology Transfer,

Bozeman, B., Coker, K., 1992. Assessing the effectiveness of technology transfer from

1–17.

US government R&D laboratories: the impact of market orientation. Technova-

Feller, I., Roessner, D., 1995. Identifying and measuring the benefits of collabo-

tion 12 (4), 239–255.

rative research: lessons from a study of the Engineering Research Centers. In:

Bozeman, B., Corley, E., 2004. Scientists’ collaboration strategies: implications for

Proceedings of the Technology Transfer Society . . . Annual Meeting, Interna-

scientific and technical human capital. Research Policy 33 (4), 599–616.

tional Symposium and exhibit, vol. 1995.

Bozeman, B., Crow, M., 1991a. Technology Transfer from US Government and Uni-

Florida, R., Mellander, C., Stolarick, K.M., 2010. Talent, technology and tolerance in

versity R&D Laboratories. Technovation 11 (4), 231–246.

Canadian regional development. The Canadian Geographer/Le Géographe cana-

Bozeman, B., Crow, M., 1991b. Red tape and technology transfer in US government

dien 54 (3), 277–304.

laboratories. The Journal of Technology Transfer 16 (2), 29–37.

Fountain, J.E., 1998. Social capital: its relationship to innovation in science and tech-

Bozeman, B., Gaughan, M., 2007. Impacts of grants and contracts on academic

nology. Science and Public Policy 25 (2), 103–115.

researchers’ interactions with industry. Research policy 36 (5), 694–707.

Franza, R.M., Srivastava, R., 2009. Evaluating the Return on Investment for Depart-

Bozeman, B., Papadakis, M., Coker, K., 1995. Industry Perspectives on Commer-

ment of Defense to private sector technology transfer. International Journal of

cial Interactions with Federal Laboratories: Does the Cooperative Technology

Technology Transfer and Commercialisation 8 (2), 286–298.

Paradigm Really Work? Report to the National Science Foundation, Research on

Frechtling, J.A., 2007. Logic Modeling Methods in Program Evaluation. Jossey-Bass,

Science and Technology Program, January.

San Francisco.

Bozeman, B., Dietz, J.S., Gaughan, M., 2001. Scientific and technical human capital: an

Friedman, J., Silberman, J., 2003. University technology transfer: do incentives, man-

alternative model for research evaluation. International Journal of Technology

agement, and location matter? The Journal of Technology Transfer 28 (1), 17–30,

Management 22 (7), 716–740.

http://dx.doi.org/10.1023/A:1021674618658.

Bozeman, B., Rogers, J., 2001. Strategic management of government-sponsored R&D

Fritsch, M., Kauffeld-Monz, M., 2010. The impact of network structure on knowledge

portfolios. Environment and Planning C 19 (3), 413–442.

transfer: an application of social network analysis in the context of regional

Bozeman, B., Sarewitz, D., 2005. Public values and public failure in US science policy.

innovation networks. The Annals of Regional Science 44 (1), 21–38.

Science and Public Policy 32 (2), 119–136.

Fukugawa, N., 2009. Determinants of licensing activities of local public technology

Bozeman, B., Sarewitz, D., 2011. Public value mapping and science policy evaluation.

centers in Japan. Technovation 29 (12), 885–892.

Minerva 49 (1), 1–23.

Gaughan, M., Ponomariov, B., 2008. Faculty publication productivity, collaboration,

Bradley, S.R., Hayter, C.S., Link, A.N., 2013. Models and Methods of University Tech-

and grants velocity: using curricula vitae to compare center-affiliated and unaf-

nology Transfer. Now Publishers Incorporated, Boston.

filiated scientists. Research Evaluation 17 (2), 103–110.

Burgess, J., Stirling, A., Clark, J., Davies, G., Eames, M., Staley, K., Williamson, S.,

Geisler, E., 1994. Key output indicators in performance evaluation of research and

2007. Deliberative mapping: a novel analytic-deliberative methodology to sup-

development organizations. Technological Forecasting and Social Change 47 (2),

port contested science-policy decisions. Public Understanding of Science 16 (3),

299–322. 189–203.

Golob, E., 2006. Capturing the regional economic benefits of university technology

Butler, S., Birley, S., 1998. Scientists and their attitudes to industry links. International

transfer: a case study. The Journal of Technology Transfer 31 (6), 685–695.

Journal of Innovation Management 2 (01), 79–106.

48 B. Bozeman et al. / Research Policy 44 (2015) 34–49

Gulbranson, C.A., Audretsch, D.B., 2008. Proof of concept centers: accelerating the Mowery, D.C., Ziedonis, A.A., 2007. Academic patents and materials transfer agree-

commercialization of university innovation. The Journal of Technology Transfer ments: substitutes or complements? The Journal of Technology Transfer 32 (3),

33 (3), 249–258. 157–172.

Gupta, A., 2002. Public Value Mapping in a Developing Country Context: A Method- Murray, F., 2002. Innovation as co-evolution of scientific and technological

ology to Promote Socially Beneficial Public Biotechnology Research and Uptake networks: exploring tissue engineering. Research Policy 31 (8), 1389–1403.

in India. Prepared for the Rockefeller Foundation. Center for Science, Policy, and O’Shea, R.P., Allen, T.J., Chevalier, A., Roche, F., 2005. Entrepreneurial orientation,

Outcomes (CSPO), Columbia University. technology transfer and spinoff performance of U.S. universities. Research Policy

Guston, D.H., 2007. Between Politics and Science: Assuring the Integrity and Pro- 34 (7), 994–1009.

ductivity of Research. Cambridge University Press, Cambridge. Park, J.B., Ryu, T.K., Gibson, D.V., 2010. Facilitating public-to-private technology

Hartmann, G.B., Masten, J., 2000. Profiles of state technological transfer structure transfer through consortia: initial evidence from Korea. The Journal of Tech-

and its impact on small manufacturers. The Journal of Technology Transfer 25 nology Transfer 35 (2), 237–252.

(1), 83–88. Phillips, R.G., 2002. Technology business incubators: how effective as technology

Heisey, P.W., Adelman, S.W., 2011. Research expenditures, technology transfer activ- transfer mechanisms? Technology in Society 24 (3), 299–316.

ity, and university licensing revenue. The Journal of Technology Transfer 36 (1), Piper, W.S., Naghshpour, S., 1996. Government technology transfer: the effective use

38–60. of both push and pull marketing strategies. International Journal of Technology

Hendriks, J., 2012. Technology transfer in human vaccinology: a retrospective review Management 12 (1), 85–94.

on public sector contributions in a privatizing science field. Vaccine 30 (44), Polanyi, M., 1969. Knowing and Being. Edited with an Introduction by Marjorie

6230–6240. Grene. University of Chicago Press, Chicago.

Henkel, M., 2005. Academic identity and in a changing policy environ- Ponomariov, B., 2009. Student centrality in university–industry interactions. Indus-

ment. Higher Education 49 (1), 155–176. try and Higher Education 23 (1), 50–62.

Hertzfeld, H.R., 2002. Measuring the economic returns from successful NASA life sci- Ponomariov, B.L., Boardman, P.C., 2010. Influencing scientists’ collaboration and

ences technology transfers. The Journal of Technology Transfer 27 (4), 311–320. productivity patterns through new institutions: university research cen-

Jaffe, A.B., Lerner, J., 2001. Reinventing public R&D: patent policy and the com- ters and scientific and technical human capital. Research Policy 39 (5),

mercialization of national laboratory technologies. Rand Journal of Economics, 613–624.

167–198. Powers, J.B., 2003. Commercializing academic research: resource effects on perfor-

John, P., 1998. Analysing Public Policy. Pinter Publishing Limited, London. mance of university technology transfer. The Journal of Higher Education 74 (1),

Jørgensen, T.B., Bozeman, B., 2007. Public values an inventory. Administration & 26–50.

Society 39 (3), 354–381. Protogerou, A., Caloghirou, Y., Siokas, E., 2012. Twenty-five years of science-industry

Kassicieh, S.K., Kirchhoff, B.A., Walsh, S.T., McWhorter, P.J., 2002. The role of small collaboration: the emergence and evolution of policy-driven research networks

firms in the transfer of disruptive technologies. Technovation 22 (11), 667–674. across Europe. The Journal of Technology Transfer, 1–23.

Kerr, C., 2001. The Uses of the University. Harvard University Press, Cambridge, MA. Ramakrishnan, S., 2004. An industrial ecology framework to assist transferring

Kingsley, G., Farmer, M.C., 1997. Using technology absorption as an evaluation crite- environmental technologies. International Journal of Technology Transfer and

rion: case studies from a state research and development program. Policy Studies Commercialisation 3 (2), 147–165.

Journal 25 (3), 436–450. Randles, S., Youtie, J., Guston, D., Harthorn, B., Newfield, C., Shapira, P., Wickson, F.,

Kitagawa, F., Lightowler, C., 2013. Knowledge exchange: a comparison of policies, Rip, A., von Schomberg, R., Pidgeon, N., 2012. A trans-atlantic conversation on

strategies, and funding incentives in English and Scottish higher education. responsible innovation and responsible governance. In: Van Lente, H., Coenen,

Research Evaluation 22 (1), 1–14. C., Fleischer, T., Konrad, K., Krabbenborg, L., Milburn, C., Thoreau, F., Zülsdorf, T.

Kleinman, D.L., 2003. Impure Cultures: University Biology and the World of Com- (Eds.), Little by Little: Expansions of Nanoscience and Emerging Technologies.

merce. University of Wisconsin Press, Madison, WI. Akademische Verlagsgesellschaft, Heidelberg, pp. 169–180.

Landry, R., Amara, N., Lamari, M., 2002. Does social capital determine innova- Rayner, S., Heyward, C., Kruger, T., Pidgeon, N., Redgwell, C., Savulescu, J., 2013. The

tion? To what extent? Technological Forecasting and Social Change 69 (7), Oxford principles. Climatic Change 121 (3), 499–512.

681–701. Ries, E., 2011. The Lean Startup: How Today’s Entrepreneurs Use Continuous Inno-

Lee, Y., Gaertner, R., 1994. Technology transfer to industry: a large scale experiment vation to Create Radically Successful Businesses. Random House Digital, Inc.,

with technology development and commercialization. Policy Studies Journal 22. New York.

Lepori, B., 2006. Methodologies for the analysis of research funding and expen- Rigby, J., Edler, J., 2005. Peering inside research networks: some observations on

diture: from input to positioning indicators. Research Evaluation 15 (2), the effect of the intensity of collaboration on the variability of research quality.

133–143. Research Policy 34 (6), 784–794.

Lepori, B., Probst, C., 2009. Using curricula vitae for mapping scientific fields: a small- Roco, M.C., Harthorn, B., Guston, D., Shapira, P., 2011. Innovative and responsible

scale experience for Swiss sciences. Research Evaluation 18 (2), governance of nanotechnology for societal development. Journal of Nanoparticle

125–134. Research 13 (9), 3557–3590.

Lin, M.W., Bozeman, B., 2006. Researchers’ industry experience and productivity in Roessner, J.D., Bean, A., 1991. How industry interacts with federal laboratories.

university–industry research centers: a scientific and technical human capital Research Technology Management 34 (July/August (4)), 22.

explanation. The Journal of Technology Transfer 31 (2), 269–290. Roessner, J.D., Bond, J., Okubo, S., Planting, M., 2013. The economic impact of licensed

Lindelöf, P., Löfsten, H., 2004. Proximity as a resource base for competitive commercialized inventions originating in university research. Research Policy

advantage: university–industry links for technology transfer. The Journal 42 (1), 23–34.

of Technology Transfer 29 (3–4), 311–326, http://dx.doi.org/10.1023/B:JOTT. Rogers, E., Carayannis, E.G., Kurihara, K., Allbritton, M.M., 1998. Cooperative Research

0000034125.29979.ae. and Development Agreements (CRADAs) as technology transfer mechanisms.

Link, A.N., 1995. Evaluation of the economic impacts associated with the NIST power R&D Management 28 (2), 79.

and energy calibration services (No. PB-95-188850/XAB; NISTIR-5565). National Rogers, E., Takegami, S., Yin, J., 2001. Lessons learned about technology transfer.

Inst. of Standards and Technology (EEEL), Gaithersburg, MD (United States). Technovation 21 (4), 253–261.

Electricity Div. Rogers, J.D., Bozeman, B., 1997. Basic research and the success of federal lab-industry

Link, A.N., Scott, J.T., 2001. Public/private partnerships: stimulating competition partnerships. The Journal of Technology Transfer 22 (3), 37–47.

in a dynamic market. International Journal of Industrial Organization 19 (5), Rose, R., 1993. Lesson-Drawing in Public Policy: A Guide to Learning Across Time

763–794. and Space. Chatham House Publishers, Chatham, NJ.

Link, A.N., Siegel, D.S., 2005. Generating science-based growth: an econometric Rowe, B.R., Temple, D.S., 2011. Superfilling technology: transferring knowledge to

analysis of the impact of organizational incentives on university–industry tech- industry from the National Institute of Standards and Technology. The Journal

nology transfer. The European Journal of Finance 11 (3), 169–181. of Technology Transfer 36 (1), 1–13.

Liyanage, S., 1995. Breeding innovation clusters through collaborative research Rubenstein, K.D., 2003. Transferring public research: the patent licensing mecha-

networks. Technovation 15 (9), 553–567. nism in agriculture. The Journal of Technology Transfer 28 (2), 111–130.

Maricle, G., 2011. Prediction as an impediment to preparedness: Lessons from the US Saavedra, P., Bozeman, B., 2004. The gradient effect in federal laboratory–industry

hurricane and earthquake research enterprises. Minerva: A Review of Science, technology transfer partnerships. Policy Studies Journal 32 (2), 235–252.

Learning & Policy 49 (1), 87–111. Sala, A., Landoni, P., Verganti, R., 2011. R&D networks: an evaluation framework.

Markusen, A., Oden, M., 1996. National laboratories as business incubators and International Journal of Technology Management 53 (1), 19–43.

region builders. The Journal of Technology Transfer 21 (1–2), 93–108. Schalock, R.L., Bonham, G.S., 2003. Measuring outcomes and managing for results.

Mendoza, P., 2007. Academic capitalism and doctoral student socialization: a case Evaluation and Program Planning 26 (3), 229–235.

study. The Journal of Higher Education 78 (1), 71–96. Schultz, T.W., 1963. The Economic Value of Education. Columbia University Press,

Meyers, S., McMahon, J.E., McNeil, M., Liu, X., 2003. Impacts of US federal energy New York.

efficiency standards for residential appliances. Energy 28 (8), 755–767. Seely, B.E., 2003. Historical patterns in the scholarship of technology transfer. Com-

Midgley, G., 2006. Systems Thinking for Evaluation. Systems Concepts in Evaluation: parative Technology Transfer and Society 1 (1), 7–48.

An Expert Anthology. Edge Press, Point Reyes, CA, pp. 11–34. Senker, J., 1995. Networks and tacit knowledge in innovation. Economies et Societes

Mohammed, Y., Sax, U., Dickmann, F., Lippert, J., Solodenko, J., Voigt, G., Rienhoff, O., 2 (9), 99–118.

2010. On transferring the grid technology to the biomedical community. Studies Shane, S., Stuart, T., 2002. Organizational endowments and the performance of uni-

in Health Technology Information 159, 28–39. versity start-ups. Management Science 48 (1), 154–170.

Mowery, D.C., 1988. The changing structure of the US national innovation system: Siegel, D.S., Veugelers, R., Wright, M., 2007. Technology transfer offices and com-

implications for international conflict and cooperation in R&D policy. Research mercialization of university intellectual property: performance and policy

Policy 27 (6), 639–654. implications. Oxford Review of Economic Policy 23 (4), 640–660.

B. Bozeman et al. / Research Policy 44 (2015) 34–49 49

Siegel, D.S., Waldman, D., Link, A., 2003. Assessing the impact of organizational prac- U.S. Congress, 1984a. Uniform Patent Procedures Act of 1983. Public Law 98-620

tices on the relative productivity of university technology transfer offices: an enacted November 8.

exploratory study. Research Policy 32 (1), 27–48. U.S. Congress, 1984b. Stevenson-Wydler Technology Innovation Act of 1980. United

Slade, C.P., 2011. Public value mapping of equity in emerging nanomedicine. Minerva States Code, Title 15, Section 3701-3714: 96-517. USGPO, Washington, DC.

49 (1), 71–86. U.S. Congress, 1986. Federal Technology Transfer Act of 1986. 99th Congress, 2nd

Slaughter, S., Rhoades, G., 2004. Academic Capitalism and the New Economy: Mar- Session. USGPO, Washington, DC.

kets, State, and Higher Education. Johns Hopkins University Press, Baltimore, U.S. Department of Agriculture Mission Statement, 2013. United States Depart-

MD. ment of Agriculture. http://www.usda.gov/wps/portal/usda/usdahome?navid=

Sorensen, J.A.T., Chambers, D.A., 2008. Evaluating academic technology transfer per- MISSION STATEMENT

formance by how well access to knowledge is facilitated—defining an access U.S. Department of Energy, 2012. U.S. DOE Plan for Transfer and Commer-

metric. The Journal of Technology Transfer 33 (5), 534–547. cialization of Technology. http://www.nist.gov/tpo/publications/upload/DOE-

Sperling, D., 2001. Public–private technology R&D partnerships: lessons from US Tech-Transfer-Plan-3.pdf

partnership for a new generation of . Transport Policy 8 (4), 247–256. U.S. General Accounting Office, 1989. Technology Transfer: Implementation Status

Sperling, D., Gordon, D., 2008. Advanced passenger transport technologies. Annual of the Federal Technology Transfer Act of 1986. USGPO, Washington, DC.

Review of Environment and Resources 33, 63–84. U.S. White House, Office of the Press Secretary, 2011. Presidential Memoran-

Stephan, P.E., 2001. Educational implications of university–industry technology dum: Accelerating Technology Transfer and Commercialization of Federal

transfer. The Journal of Technology Transfer 26 (3), 199–205. Research in Support of High Growth Businesses Downloaded November 10, 2012

Swamidass, P.M., Vulasa, V., 2009. Why university inventions rarely produce from: http://www.whitehouse.gov/the-press-office/2011/10/28/presidential-

income? Bottlenecks in university technology transfer. The Journal of Technol- memorandum-accelerating-technology-transfer-and-commerciali

ogy Transfer 34 (4), 343–363. Valdivia, W.D., 2011. The stakes in Bayh-Dole: Public values beyond the pace of

Thursby, J.G., Jensen, R., Thursby, M.C., 2001. Objectives, characteristics and out- innovation. Minerva 49 (1), 25–46.

comes of university licensing: a survey of major U.S. universities. The Journal of von Schomberg, R., 2013. A vision of responsible research and innovation. In: Owen,

Technology Transfer 26 (1–2), 59–72. R., Bessant, J., Heintz, M. (Eds.), Responsible Innovation: Managing the Respon-

Thursby Jerry G., Kemp Sukanya, 2002. Growth and productive efficiency of univer- sible Emergence of Science and Innovation in Society. John Wiley & Sons, pp.

sity intellectual property licensing. Research Policy 31 (1), 109–124. 51–74.

Tran, T.A., Kocaoglu, D.F., 2009. Literature review on technology transfer from Woerter, M., 2012. Technology proximity between firms and universities and

government laboratories to industry. In: Portland International Conference technology transfer. The Journal of Technology Transfer 37 (6), 828–866,

on Management of Engineering & Technology, 2009. PICMET 2009, IEEE, pp. http://dx.doi.org/10.1007/s10961-011-9207-x.

2771–2782. Youtie, J., Libaers, D., Bozeman, B., 2006. Institutionalization of university research

U.S. Congress, 1980. Stevenson-Wydler Technology Innovation Act of 1980, P.L. 96- centers: the case of the National Cooperative Program in Infertility Research.

517, United States Code, Title 15, Section 3701-3714. Technovation 26 (9), 1055–1063.