![Business and Management Research Themes and Impact [Version 1; Peer Review: 1 Approved, 1 Not Approved]](http://data.docslib.org/img/47ec379b874ebcc6278a844c438d10a3-1.webp)
Emerald Open Research Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
RESEARCH ARTICLE
Business and management research themes and impact [version 1; peer review: 1 approved, 1 not approved]
Evidence from REF 2014
Paul A. Phillips 1, Stephen Page2, Joshua Sebu 3
1Kent Business School,, University of Kent, UK, Canterbury, Kent, CT2 7AJ, United Kingdom 2Hertfordshire Business School, University of Hertfordshire, UK, Hatfield, United Kingdom 3School of Economics, University of Cape Coast, Ghana, University of Cape Coast,, Ghana, Ghana
v1 First published: 24 Nov 2020, 2:67 Open Peer Review https://doi.org/10.35241/emeraldopenres.13987.1 Latest published: 24 Nov 2020, 2:67 https://doi.org/10.35241/emeraldopenres.13987.1 Reviewer Status
Invited Reviewers Abstract This paper examines the theoretical issues and research themes of 1 2 business and management impact. Our empirical setting is the UK Research Excellence Framework 2014 (REF 2014) and the focus is on version 1 the nature of research impact. Stakeholders, including Governments, 24 Nov 2020 report report now expect academic outputs to translate to real world benefits beyond the narrow bibliometric type metrics. 1. Julie Davies , Manchester Metropolitan Despite decades of academic literature devoted to business and management research impact, current theories cannot explain the University, Manchester, United Kingdom apparent disconnect between academic, economic and societal practice. Adopting a UK Business and Management perspective to 2. Ludo Waltman , Leiden University, frame our investigation, we consider the highly contested rhetorical Leiden, Netherlands question – what are the current themes and impacts of Business and Management research? Any reports and responses or comments on the We propose a definition for research impact and consider its article can be found at the end of the article. measurement. Then, using the 410 Impact Case Studies submitted to REF 2014 - Unit of Assessment 19, business and management, we examine how high impact unfolds. The implications for business and management research impact from the perspectives of economic, knowledge and responsibility impacts are considered.
Keywords Business and management research impact, REF 2014, REF 2021, Impact Set Theoretic Approach, Research performance
This article is included in the Responsible Management gateway.
Page 1 of 20 Emerald Open Research Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
This article is included in the Quality Education for All gateway.
Corresponding author: Paul A. Phillips ([email protected]) Author roles: Phillips PA: Conceptualization, Data Curation, Methodology, Project Administration, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing; Page S: Conceptualization, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing; Sebu J: Data Curation, Investigation, Methodology, Software, Writing – Original Draft Preparation Competing interests: No competing interests were disclosed. Grant information: The authors wish to thank the Kent Business School, University of Kent and Hertfordshire Business School, University of Hertfordshire for funding towards the research reported in this study. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Copyright: © 2020 Phillips PA et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. How to cite this article: Phillips PA, Page S and Sebu J. Business and management research themes and impact [version 1; peer review: 1 approved, 1 not approved] Emerald Open Research 2020, 2:67 https://doi.org/10.35241/emeraldopenres.13987.1 First published: 24 Nov 2020, 2:67 https://doi.org/10.35241/emeraldopenres.13987.1
Page 2 of 20 Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
Background of how high impact unfolds. We adopt a pluralistic approach UK business schools (BSs) contribute £3.35bn to the UK econ- (Aguinis et al., 2019) and assess BS impact by examining omy and provide financial cross subsidies to other university Research Excellence Framework 2014 (REF 2014) impact schools (Bradshaw, 2017). Taking note of its business model, case studies (ICS). Following on from these discussions we BSs were initially praised as a success story of higher educa- formulate our research questions as follows: tion (Alajoutsijärvi et al., 2015). BSs remain influential and possess the ability to attract huge numbers of premium fee-paying 1) How should research impact be defined, and how is it students on their undergraduate, postgraduate and executive measured? MBA programs. 2) How does high quality business and management ICS research impact unfold within the academy? Yet a series of staff inspired polemical critiques of the nature of BS research have raised questions regarding whether such organ- 3) What should the role of BSs be in society, and what type izations have lost their way, due to criticisms by authors of a of impact should they have? perceived lack of rigor and relevance in their staff research (Irwin, 2019), along with an inward focus on high quality The paper now considers the definition and measurement of scientific research that has little application to the real world research impact, including theoretical aspects of the concept (Kapstein & Yip, 2011), a feature polemicized by Caplan’s of impact, REF 2014 and ICSs to contextualise the paper. The (2018) debate on the value of Universities. Van de ven (1989) methodology employed to examine the ICSs and key findings reminds us that the central mission of management scholarship is then presented. We then outline the implications, conclusions is contributing to management practice. Expectations exist that and consequences for policy development. outputs from the academy should translate into tangible real- world benefits including societal impacts, environmental and Definition and measurement of research impact economic impacts (Pettigrew & Starkey, 2016; Watermeyer, 2014). Definition of research impact. Existing definitions of research impact are wide ranging making the construction of a working However, the last decade has witnessed a Tayloristic shift definition of impact complex. Definitions have varied depend- with geopolitical forces reshaping business schools. Calls for ing on the focus of the researcher, which Penfield et al. (2014) global action initiatives such as The United Nations Sustain- divided into academic impact or external socio-economic impact. able Development Goals (SDGs) place increasing pressure for The UK Economic and Social Research Council1, has indi- BSs to demonstrate impact to their wider economic and soci- cated that academic research has impact if it is able to demon- etal audiences (Weybrecht, 2020). According to the Association strate some contribution to society and the economy, or more to Advance Collegiate Schools of Business (AACSB) more simply put it generates impact beyond academic stakehold- than 800 BSs are signatory to the Principles for Responsible ers. The ESRC has traditionally framed this, like many other Management Education (PRME) with increasing numbers research councils in the UK do, in terms of developing pathways looking beyond SDG 4 (providing a quality education). to impact where ‘high quality Pathways to Impact will include explicit awareness of principles and practices of knowledge Such observations illustrate the internal challenge that BSs’ face exchange – including the application of principles and practices with two principal audiences for their research: academics and of co-production – as opposed to dissemination’ (http://www.esrc. practitioners (Kapstein & Yip, 2011). The blame varies from ac.uk/research/impact-toolkit/developing-pathways-to-impact/). academics being caricaturized as being obsessed with theory However, in March 2020, UK research and innovation (includ- and long-time horizons (Rynes & Bartunek, 2017; Trank, 2014); ing the research councils) advocated dropping the pathways physics envy (Thomas & Wilson, 2011); the academy possess- to impact, with impact now included as a core element of the ing a myopic view on what constitutes managerial scientific entire grant application process, illustrating the enhanced impor- practice (Seal, 2012); academics seeing impact as a dilution of tance of impact as it is now embedded in the research funding rigor and threat to academic autonomy (Johnson & Orr, 2019). process. In the case of the ESRC, research outcomes that affect Counter-arguments view practitioners as academic phobic, society, culture, the environment and economy are deemed to short-term natured and susceptible to the latest management have had impact beyond academia. Academic impact, in con- fads advocated by management consultants (Phillips et al., trast, has often been measured in relation to citations of arti- 2019). The different contexts of academics and practitioners have cles. The entire area of academic impact is one which is highly been acknowledged, so the gap is hardly surprising. These challenging (e.g. Amara & Landry, 2012; Garfield, 1979; range from reward systems (e.g. Phillips et al., 2018); belief Pendlebury, 2009), with its qualitative and quantitative measures systems (e.g. Kieser & Leiner, 2009); communications flows of impact. (e.g. Hughes et al., 2017); lost in translation from theory to practice (e.g. Shapiro et al., 2007). Lejeune et al. (2019) prefer a more granular approach towards impact, developing a three-fold categorisation comprising Research problem This paper seeks to unravel what we know about business and management research impact and propose some more granular 1The Economic and Research Council (ESRC) is one of the UKs major public definitions. Then, with this impact agenda we seek to delve funders of social science research, including studies related to Business and into this apparent disconnect to disentangle the practical issues Management and the disciplines that are often located within Business Schools.
Page 3 of 20 Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
economic impact, knowledge impact and responsibility outside academia, a worrying feature given the external reach impact as illustrated in Table 1. We proffer some definitions to embodied in many BSs missions. Aguinis et al. (2012) advo- capture the importance of some element of a transfer of value cated that academic impact is a multidimensional construct in all three categorisations. We believe this is crucial as it and not easily captured and incorporated through conventional moves the dial away from activities, and focuses on real impact academic measures. among actors. There are two underpinning concepts which REF 2014 intro- Measurement of research impact. Analysis of academic ‘impact’ duced to guide the assessment of research impact in terms of has traditionally adopted quantitative measures, focusing on the significance of the research to the wider community and frequency of research citations and the journal impact factor how far it had reached these end users captured in two docu- (IF) (Hall & Page, 2015; Holmberg et al., 2015). A common ments submitted by each UOA2. The impact score accounted complaint is the disconnect between the knowledge produced for 20% of the total score in RE2014 and it is rising to 25% in by academics and that which is consumed by practitioners REF 2021. REF 2014 defines research impact as “the effect that (Aguinis et al., 2019; Jackson et al., 2014), a gap that arguably research has had on the economy, policy, culture, the environ- is widening. The focus on traditional scholarly impact is depicted ment or quality of life outside the academic sphere”. Research in terms of an internal exchange within the field of academia performance being measured on a scale from 4* (world-leading) (Aguinis et al., 2019). This internal focus normally consid- to unclassified. With 4* Quality world leading; 3* internationally ers journal evaluation techniques such as journal impact factors, excellent; 2* internationally; 1* nationally. including the H-Index, SCImago Journal Rank (SJR), Source Normalized Impact per Paper (SNIP) and Eigen factors (Mingers Theoretical aspects of the concept of business school & Yang, 2017). But, these methods of measuring impact only impact help to ascertain the benefit of research within academia Theoretical challenges. As the growing number of papers on and so are only ever going to be a partial analysis of impact this subject posit, a number of major theoretical challenges (Bornmann & Marx, 2013). The main challenge remains expand- for the academy are prominent. For example, Teagarden et al. ing the external reach of BS research beyond academia to (2018: 304) argue that it is important to rise to the theoriza- permeate practitioners, given the scope of many BSs mis- tion challenge in business research as ‘contextualizing…[helps] sions to prepare students for the world of work and engage with to achieve research rigor and practical relevance is a challenge businesses to help create economic value. faced by all sub-disciplines.’. This means looking at research through a different lens and adopting multiple levels of analysis to New frameworks have been developed as outlined by Penfield incorporate theory and impact. This tension between rigor et al. (2014), which advocated a mixed method approach incor- and relevance as embodied in the debate on research impact porating the case study method as a way of collating all avail- able information, data and evidence to coherently summarise impact. Aguinis et al. (2012) suggested an alternative approach 2According to HEFC (http://www.ref.ac.uk/2014/media/ref/content/pub/REF to impact, adopting the number of pages indexed by Google %20Brief%20Guide%202014.pdf), Impact case studies comprised four-page to assess the impact of academic research outside academia as documents which described impacts that had occurred between January 2008 opposed to the use of citation data. Aguinis et al. (2012) adopted and July 2013. The submitting university must have produced high quality five steps and decision points to ensure the validity of their meas- research since 1993 that contributed to the impacts. Each submission included ure. Using a sample of 384 highly cited management scholars one case study, plus an additional case study for every 10 staff. An impact template was also submitted. This document explained how the submitted unit over the past 30 years, they demonstrated that scholarly works had enabled impact from its research during the period from 2008 to 2013, only affected stakeholders within academia and not stakeholders and its future strategy for impact.
Table 1. Definition of research impact.
Nature of impact Definition Examples in Literature
Economic impact A deliberate managed process across university boundaries that Azmat et al., 2018; creates value through connections, and delivers financial benefits Cooke & Galt, 2010; to external communities Kelly & McNicoll, 2011; Valero & Van Reenen, 2019;
Knowledge impact A deliberate managed process across university boundaries that Amara et al., 2016; Bager, 2018; creates value through connections, and results in knowledge Tho, 2017; Tho & Trang, 2015. exchange that enables external communities to move forward
Responsibility impact A deliberate managed process across university boundaries that Murcia et al., 2018; creates value through connections, and delivers societal oriented Snelson-Powell et al., 2020 benefits to external communities
Page 4 of 20 Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
has become a substantial problem (Head & Alford, 2015) (i.e. wisdom, good judgement and common sense in management one which is where there is incomplete theoretical knowledge, practice, as a basis for extending the theoretical debate on rigor and so is impossible to address because of conflicting objec- and relevance. Such a seed of change is almost calling for a tives) as the debates over the role of BSs continue to empha- theoretical shift away from the conventional disciplines and size (see Hardy & Everett, 2013 for a more detailed discussion). their theoretical bases, as a means to address business-related This problem led Hamet & Michel (2018) to explore the theo- problems away from their narrow disciplinary focus. Instead, retical explanations of the problem further, arguing that it is a research becomes more impactful by embracing a more critical consequence of multiple factors such as the mission of BSs (i.e. social science range of specialisms and harnessing these with the education-research nexus), the favouring of positivist epis- external stakeholders, which begins to move us away from theo- temologies, while impactful research is recognized as lacking retically anchored multidisciplinary approaches to research and rigor. This is inherent in the research-practice debates which education and towards new modes of knowledge production led Kieser et al. (2015) to suggest the causes of the research- around critical social science. practice debate was associated with various streams of thought including: There is a long history in research, and consequently a vast • A transmission problem, with poor dissemination to research stream debating if BSs should impact society, and what the wider communities of practice who popularise the their impact can be. Recent literature has considered such issues. research One school of thought suggests the modern BS is an overloaded institution responsible to multiple stakeholders and for multi- • Institutional causes, where narrow disciplines do not ple purposes (Irwin, 2019). A core driver in the reward mecha- collaborate with business reflecting the weaknesses of nisms for BSs is the playing and winning the “indicator game”. the academy in practice Universities vie for accreditations and rankings and this distorts behaviour and outputs. Nevertheless, despite BSs vision and • An overconcentration on theoretical research and mission statements conceptualizing themselves as “for and about high-level journals, to the detriment of action research business” the reality is much different. As Bennis & O’Toole methodologies, especially qualitative methodologies. (2005) retort in their seminal article on this “dirty little secret”, academics are preoccupied with their self-serving attitude with Beer’s (2001) arguments would seem to support these explana- scant regards to stakeholder needs. BSs should think and act tions, on the reasons why so little business and management more pro-actively about wider societal goals, as opportuni- research have any impact (i.e. they are unimplementable). This ties will be lost to set the agenda globally beyond the “indicator is attributed to the limited success of new management episte- game”. In response to this situation, Parker (2018) contends mologies and an auto-referential system of knowledge production that if Universities have responsibilities to society, BSs have to that reinforces the positivist paradigm. Yet these explana- comply or be shut down. tions appear to not be supported in other disciplines, since in Science the knowledge acquisition process by non-academic REF 2014 and Impact case studies. The empirical setting for stakeholders has overcome the schism between researchers and this study in terms of REF2014 and ICS has been described in practitioners. Phillips et al. (2020) in relation to the field of Tourism, Sport, Leisure and Hospitality and their outcomes in REF 2014. Busi- This has led Narasimhan (2018) to argue that the reward struc- ness and Management was one of the largest Units of Assess- ture of BSs has placed a premium on high level scholarship ment (UOA19) in contrast to the area examined by Phillips et al. to the detriment of the real world experience of academics. (2020) which was smaller and more niche in focus. In this Yet as we argue in this paper, research impact as espoused by paper we delve into the themes and impacts of BSs. the REF exercise actually promotes rigor (i.e. the intellectual quality of the research which is highly subjective) and relevance In accordance with UK Research and Innovation, REF 2014 (i.e. the production of useful knowledge that can address real impact for this study is broadly defined “as an effect on, change world issues). This has led Van de Ven & Heath (2007) to pro- or benefit to the economy, society, culture, public policy or mote the concept of the engaged scholar. In a similar vein, services, health, the environment or quality of life, beyond Birnik & Billsberry (2008) suggest the need for a shift away academia”. Its measurement is potentially based on a peer review from pure self-interest of the academic to address a crisis of rel- system, with scores being determined by a combination of evance in BSs, by adopting a more pluralistic approach where factors, including quality of publications, quantity and value of self-interest and altruism is fostered. In many countries the state research income, team profile and characteristics, spin offs- cre has promoted ‘Grand Challenges’ to stimulate altruistic think- ated and evidence of esteem among influential stakeholders ing and to enhance the application of research to well-being (Chowdhury et al., 2016). and the public good. In the £250m REF 2014 exercise 6,975 ICSs were submitted All of these activities need to be undertaken by the researcher for all UOAs and in UOA19, 410 case studies were submitted to develop a multifaceted and outward facing process. In some by BSs and 37.7% of them were judged to be outstanding (4*) cases, innovations in teaching (see Antonacopoulou, 2010) with 42.5% judged as very considerable (3*) and this was only suggest the solution to making BSs more impactful is phro- 3% below the average score for impact for all of the REF 2014 nesis. This concept, from Ancient Greek, extols the virtues of ICSs. Although this could be used as evidence and reflection of
Page 5 of 20 Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
some excellent impact generated by BS’, key questions that the Secondary analysis of REF 2014 ICS’ REF 2014 does not answer are: how have BS’ affected their The first stage in the research process was to undertake a descrip- main constituents? And have they produced distinct and dispa- tive analysis of all 410 ICS’ submitted to UOA19, gathered by rate pathways to impact? While there are a variety of impact downloading them from the REF 2014 website. Initial grade outputs, the main route to impact for BS’ given their mission point average (GPA) scores exist for the impact and the output should logically be businesses, or a contribution to government scores for all 101 submitting institutions. This was then used policies that directly improve business performance and so to initially find the relationship between the impact score as 3 economic activities . A significant question is what do BS’ the dependent variable and the output score as the independ- see as their primary and secondary mandates, and how is the ent variable by way of a scatter plot as an initial starting point creation of impactful research informing and influencing the which was developed in Kellard & Śliwa’s (2016) study. management of these organizations? In other words, what does a 4* UOA19, high-quality ICS look like? A scatter plot of the relationship between the impact GPA and output GPA scores is shown in Figure 1, a feature explored We must also recognise that in REF 2014 ICS’ were often in terms of staff interest in each by Saltera et al. (2017). The chosen centrally by participating institutions (mainly universi- plot shows a positive relationship between the two GPA scores ties). So, this makes it challenging to draw general conclusions which can also be inferred from the slope of the regression line on BS impact for the following reasons: (0.86). The slope implies that an uplift in the output GPA score a) There was a very limited data set of cases from each by one unit increases the impact GPA score by approximately institution. 0.86. However, the R-squared shows that the output GPA score only explains about 37% of the variation in impact GPA b) The entries were not necessarily those advocated by BS’. score. This implies that there are other factors together with the c) Many ICS’ mapped to REF 2014 criteria and frequently output GPA score that explain the score of ICS. The scatter went beyond economic impact. plot clearly shows an outlier in the data. This is the University of York St. John which had the minimum impact GPA as d) The impact of BS was excluded. observed earlier. To prevent further analysis being influenced by this outlier, University of York St. John was taken out of the e) ICS’ did not necessarily include the magnitude of impacts. data. Therefore, the University of Chester then becomes the institution with the least impact GPA score of 1.60. Following the publication of the results of REF 2014, several studies have attempted to identify what makes a good ICS and The second stage was to identify three clusters to group institu- how to improve one’s scores for the next REF (e.g. Hughes tions into top, middle and bottom groups based on their impact et al., 2017; Saltera et al., 2017). But there has been no study GPA scores. 10 institutions were then selected from each that has provided an in-depth analysis of ICS for UOA19 group, extracting their “summary of impact” from the ICS. The despite the scale and significance of this UOA, second only to sample of 30 ICS was selected so as to confirm the results fields allied to Medicine. of the Kellard & Śliwa’s (2016) study to ensure a consistent methodological approach4. The first top 10 institutions with the highest impact GPA were chosen and these are presented Methods The study is rooted in a two-stage sequential mixed meth- in Table 2. The middle and bottom ten institutions with the ods approach, adopting a pragmatic research paradigm. The least impact GPA score were also obtained and presented in reason for adopting a pragmatic paradigm is the decision to Table 3 and Table 4. combine the exploratory analysis of a large body of secondary data that is predominantly based on qualitative data with a more The middle ten was obtained by first obtaining the median detailed quantitative investigation of outcomes. The first stage which was the 49th term occupied by University of Central adopts a qualitative analysis of the themes based on secondary Lancashire. We moved four steps up and five steps down to get data generated by the REF 2014 process associated with ICSs. all the middle ten institutions. In all cases, where institutions The second stage was to use a set theoretic approach to ascer- had a similar impact GPA score, their respective output GPA tain which configurations arise from the ICS’ as a body of score was used to rank them. For example, in selecting the qualitative textual data. The purpose was to find an analytical bottom 10 institutions, University of West of Scotland, University technique capable of combining different attributes, which of Westminster, University of Wolverhampton and University create successful or unsuccessful outcomes. This would of Sunderland had similar impact GPA scores of 2.3. Count- identify some of the paths that would explain high and low ing from the bottom, only two institutions from this list were levels of external impact. needed to complete the bottom ten. Arranging them in descending
4The sample was designed to replicate this study because we cannot access the score for each of the 410 case studies. Instead it adopts the same methodol- 3 What is notable is that few successful ICS’ can be observed in REF 2014 ogy as Kellard and Śliwa because they looked at the institutional level rather from UOA19 that went down this route. ICS level.
Page 6 of 20 Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
Figure 1. Scatter plot of Impact and Output GPA for UoA 19.
Table 2. Top 10 institutions. Table 4. Bottom 10 institutions.
Institution Impact Output Institution Impact Output
University of Cambridge 3.84 3.19 University of Westminster 2.30 2.34 University of Aberdeen 3.80 2.77 University of Sunderland 2.30 1.13 University of Strathclyde 3.75 2.81 Keele University 2.23 2.46 University of St Andrews 3.73 2.88 University of Surrey 2.18 2.82 University of Brighton 3.73 2.57 University of Bedfordshire 2.10 2.49 University of Bristol 3.70 2.87 University of Northampton 2.00 1.81 University of Ulster 3.67 2.82 London School of Economics and 3.64 3.29 Teesside University 1.90 2.48 Political Science University of Worcester 1.90 1.64 Bournemouth University 3.63 2.37 University of Hertfordshire 1.80 2.16 University of Sheffield 3.60 2.91 University of Chester 1.60 1.57
Table 3. Middle 10 institutions. order of their output GPA score, University of Wolverhampton Institution Impact Output (2.0) and University of Sunderland (1.13) were the least. University of Kent 3.12 2.80 However, the UOA19 impact case studies for University of Wolverhampton could not be located among the case studies University of Exeter 3.10 2.76 submitted on the REF website. Hence, University of Westmin- University of Birmingham 3.10 2.66 ster, which has the next highest output GPA of 2.34, was used Kingston University 3.10 2.65 in place of Wolverhampton. University of Central Lancashire 3.10 2.19 Middlesex University 3.10 2.51 Set theoretic approach The structure of a higher education institution (HEI), like any Robert Gordon University 3.10 2.42 major organization, is made up of a complex set of attributes University of Sussex 3.08 2.74 that work together in order to achieve an outcome. Organiza- tions possess different skills, resources, structures, and envi- London Business School 3.06 3.28 ronments that all come together to influence what could be Royal Holloway, University of London 3.06 2.95 achieved depending on how they are used together. Hence, in any
Page 7 of 20 Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
analysis of performance or outcome, one should examine an combination of organizational factors combine to produce a organization in a systematic and holistic way. No one attribute particular outcome, like performance. The factors or attributes or characteristic of an organization is able to achieve an are converted into truth values, true or false, using a value outcome without the influence of other attributes, since it of 1 and 0 respectively. involves the interconnection among various attributes or char- acteristics in that organization. An appropriate piece of sup- The configurations in this study are made up of attributes that porting theory is configuration theory. According to Fiss (2007) interconnect for HEIs to produce impactful research, where the a configurational approach predicts that organizations can be impact scores obtained for an impact research evaluation is the understood to consist of clusters of interconnected structures, outcome or dependent variable. The first stage of the research profiles or patterns and practices. method is to identify factors or attributes that could interrelate to produce impactful research. This must be based on existing knowledge of the possible factors influencing the outcome. Unlike traditional multivariate techniques that mostly assume For this reason, we examined the existing research impact a linear relationship between a set of independent variables literature to identify what variables could influence outcomes and a dependent variable, configuration theory assumes a (see Table 5). complex interconnection between variables, attributes or char- acteristics. These interconnections can be in the form of a direct Data for these variables was obtained from the study by relationship, a reverse causality or have an inverse relation- Kellard & Śliwa (2016). These were a combination of their ship between factors. The absence or presence of some factors Table 6, Table 8 and Table 10, which were the ICS metrics of could enable an increase or decrease in the outcome. Hence, the top ten, middle ten and bottom ten groups. Table 2, Table 3 different organizations may yield similar results using different & Table 4 present the members of each group and their respec- strategies or even facing different conditions and factors. These tive scores in terms of impact and output GPA scores. The data configurations are a combination of different attributes, which was calibrated into fuzzy set scores for use in the fuzzy set quali- could yield different or similar outcomes. tative analysis (fsQCA) software (Ragin, 2006) that undertakes ST analysis. Fuzzy sets are an extension of the crisp sets, which To empirically test how configurations affect an outcome, allows membership scores ranging from 0 to 1 (Ragin, 2006). the set theoretic (ST) approach is used. This study employs Three-value fuzzy sets were used in this study. Three thresh- Fiss’ (2007) application of a ST approach in business and man- olds were determined to help with the calibration and the agement research, which uses the aggregated relationships and ratio-scale and interval scale variables for fuzzy member- configurations which can be traced back to the development of ship scores. The thresholds included 0.95 for full membership set theory in mathematics. Set theory postulates that different (high impact score), 0.5 for cross-over point and 0.05 for full cases could influence a certain outcome. The ST approach adopts non-membership (low impact score). Table 6 displays the Boolean algebra to determine which configurations, or the calibration thresholds chosen for the conditions in the data.
Table 5. Factors influencing impactful research.
Variable name Description Number of case studies Number of impact case studies submitted by each institution. Average number of outputs Average number of underpinning research output listed to support each constitutional impact case study. Percentage of journals Percentage of the underpinning research outputs represented by journal articles Average ABS score Average Association of Business School 2012 (ABS2010) Quality Guide ranking of each journal in which underpinning research was published Average listed grant (£k) Average listed grant amounts for each institutional impact case study Average number of key researchers/ Average number of key researchers for each institutional impact case study Number of researchers Average longest time in post (years) Average length of service for longest serving key researcher for each institutional impact case study (i.e. the length of time a key researcher has been working in the institution prior to REF2014) Percentage of women key Percentage of women key researchers for each institutional impact case study researchers Average percentage public of public Percentage of an institution’s case studies stemming from primary interaction with public interaction and non-profit organisations Average percentage national reach Percentage of national reach which represents the proportion of an institution’s case studies where the impact was generated within one country only Sources: Kellard & Śliwa (2016) and Chowdhury et al. (2016)
Page 8 of 20 Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
Table 6. Calibration thresholds for conditions.
Calibration Threshold
Factors non-membership cross-over Full membership (0.05) (0.5) (0.95)
1. Number of cases 2 3 8
2. Average number of outputs 4.5 5.475 6
3. Percentage of journals 47 70.795 100
4. Average ABS score 1.67 2.67 3.45
5. Average grant (£k) 0 125.43 954.85
6. Average number of key researchers 1 1.9 4.67
7. Average longest time in post (years) 5 17.25 31.67
8. Percentage of women key researchers 0 31.37 100
9. Average percentage public 0 77.5 100
10. Average percentage national reach 0 50 100
11. Impact 1.8 3.09 3.8
Following the calculation of calibration thresholds, the next step a high impact factor. For the fourth solution a product of a high was to create a truth table. The truth table stipulates the outcome number of ICS’ submitted and a high average number of key for each configuration and determines which configurations to researchers produced a high impact factor. The last solution is include in the analysis. The next step involved specifying the a product of a high number of ICS’ submitted and longevity causal configurations and outcomes to be minimized. The truth of time in post as shaping the high impact score. table was then generated which included 1024 configurations. These were all the possible configurations for our data. The The solution output included measures of coverage and consist- output of the truth table recorded 1s and 0s, which indicate dif- ency for each of the configurations and the solution as a whole. ferent corners of the vector space defined by the fuzzy set The consistency measures according to Ragin (2006) are the causal conditions. Configurations linking a frequency thresh- measure of the degree to which each configuration term and the old of cases of at least one was set as the rule for classifying solution as a whole are subsets of the outcome. For the individual which configurations were relevant. A consistency threshold solutions, a solution is said to be consistent if membership in of 0.9 was used to identify outcomes (impact) that will be given the solution is less or equal to membership in the outcome. The a 1 and 0 for those configurations with consistency score of less fourth solution has the highest consistency score of 0.95 and than 0.9 and this is shown in Table 7 below. the first solution has the least consistency of 0.78. Thesolu- tion consistency is 0.85 and this measures the degree to which Results membership in the solution is a subset of membership in the Boolean logic was applied to determine commonalities among outcome. configurations that lead to the outcome and to generate logi- cal statements. The Quin-McCluskey algorithm is applied to The raw coverage measures the number of or percentage of simplify ST statements using the fsQCA software package outcomes of cases in the outcome explained by each term (Roig-Tierno et al., 2016). The parsimonious solutions generated of the solution. Solution coverage measures the proportion of from the analysis are reported in Table 8. Five solutions were memberships in the outcome that are explained by the complete obtained from the analysis revealing several paths to achieving solution (Ragin, 2006). According to Elliott (2013) the consist- a high impact score. Solution one shows that the average grant ency and coverage scores should be in ranges that validate the listed in thousands of pounds was the most important factor solutions. Hence, with both the solution consistency and solution for achieving a high impact factor. Secondly, achieving a high coverage score being high and relatively close, the results could impact factor is a product of high number of ICS’ cases sub- be said to be valid. mitted and a low percentage of national reach. This may seem contradictory as it was expected that a high average percentage Main findings of national reach would produce a high impact score. The third Theoretical implications. As such, our results contribute to solution revealed that a product of having spent several years the “black box” by outlining five solutions in terms of paths in post and a lesser percentage of national reach also produce of achieving a high impact score. Evidence of a significant
Page 9 of 20 Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
Table 7. Truth table for the outcome high impact score.
Number Raw PRI SYM 1 2 3 4 5 6 7 8 9 10 of cases Impact consist. consist. consist
1 0 0 1 1 0 0 0 1 0 1 1 0.993711 0.96875 0.96875
1 1 1 1 0 1 1 0 1 0 1 1 0.992565 0.973684 0.973684
1 0 0 0 1 0 1 0 1 1 1 1 0.992395 0.962963 0.962963
0 0 1 0 1 1 1 0 0 0 1 1 0.991416 0.960784 0.960784
1 1 1 1 1 1 0 1 0 1 1 1 0.991379 0.958333 0.958333
1 0 1 1 1 0 1 0 0 0 1 1 0.978903 0.166665 0.166665
1 1 0 0 1 1 1 0 1 1 1 1 0.977848 0.925532 0.977528
0 1 1 0 1 0 1 1 1 1 1 1 0.972549 0.916667 0.916667
1 0 0 1 1 0 0 0 1 1 1 1 0.972308 0.871428 0.938462
1 1 1 0 1 0 0 1 0 0 1 1 0.964444 0.619048 0.684211
1 0 0 0 0 1 1 1 0 0 1 1 0.951542 0.153846 0.153846
1 0 1 1 1 0 0 0 1 0 1 1 0.9375 0.238095 0.294118
0 0 1 1 0 0 1 1 0 1 1 0 0.734884 0.033898 0.033898
1 0 0 1 0 0 0 0 0 1 1 0 0.728745 0.014706 0.014706
0 1 1 0 0 1 1 0 1 1 2 0 0.698171 0.282608 0.282608
0 0 0 0 0 0 1 1 0 1 1 0 0.667897 0.021739 0.021739
0 0 1 1 0 0 0 1 1 0 1 0 0.662447 0.012346 0.012346 Note Numbers 1 to 10 in the first row correspond to the numbering of factors in Table above.
Table 8. Parsimonious solution for high impact score.
Solutions Raw coverage Consistency
Average listed grant k 0.743539 0.781882
No. cases*~average percentage national reach 0.455268 0.823741
Average longest time in post (years)*~average 0.434725 0.786571 percentage national reach
No. cases*average no. of key researchers 0.46786 0.948925
No. cases*average longest time in post (years) 0.432074 0.832695
Solution coverage: 0.851557
Solution consistency: 0.723536
level of grants was the most influential factor, which in itself of impact, together with a low staff turnover of key staff were can be a due to a myriad of characteristics including past pro- helpful in terms of ICS success. As can be seen, many of these ductivity, collaboration with stakeholders, BS environment, and attributes refer to internal impact. Relationships across bounda- research strategy. It was noted that being part of a large research ries reside at the heart of successful ICS and these can include group with several years in post contributed to a high ICS score. business, professions, policy, and civil society. If relation- The results illustrate that a track record of winning research ships are an important driver of success, then further analy- grants, size of research team, number of ICS produced, reach sis is required to operationalise research impact with interested
Page 10 of 20 Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
stakeholders. We will now reconsider our definitions and cat- internally oriented academic community. BSs need to acknowl- egorisations of economic impact, knowledge impact and edge that influential studies such as Giddens (1998) The Third responsibility impact. Way, have shifted the expectation to also focus on citizenship and the development of a more active civil society (Edwards, 2013) Economic impact. Interestingly, issues that can be explained which is an emergent research strand for BSs (see Page et al., as external impact are not significantly made in the business 2017) around business engagement and grand societal challenges. domain. The significant point from this ICS study, in terms of research impact themes, is that BSs possess a limited focus The modern plurality focus of BSs together with more demand- on business despite this being a core element of their mission. ing stakeholders necessitate consideration of the third way and There appears to be a failure of practicing what they preach. civil society agenda. The UK Goverment with its number of From the theoretical debates on the rigor-relevance debate, it is outputs from the Department for Business, Energy and Indus- clear that this is not a philosophical debate around epistemolo- trial Strategy (see Building our Industrial Strategy: green gies. This is pertinent to the debate, illustrating a lack of what paper and the Made Smarter Review) illustrates the challenge Van de Ven & Heath (2007) term as engaged scholarship and opportunities for BSs. Phillips et al. (2020) articulate this in which incorporates both rigor and relevance. So, an apt ques- greater detail. Nevertheless, the apparent gulf between theory and tion to ask is what do they focus on? From an external per- practice suggest the need to develop new mechanisms for pro- spective, the ICS provide a novel mechanism to examine the ducing the outcomes to close the gap. But the capability if not policy implications for BS research and engagement with the capacity is prevalent in BS, and there is need for BSs to show stakeholders. how their research can influence policy and practice. Research is now gathering pace with the Economic and Social Research More importantly, there was the expectation that as a business Council (ESRC) and UK Research and Innovation (UKRI) and management UOA, most of these themes would be eco- announcing rapid response calls for coronavirus disease 2019 nomic impact related with an input-output focus. However, the (COVID-19), which is the current enduring problem for HEIs. evidence clearly shows that this was not the case. The impli- cation of recognising the effect of this lack of business focus Conclusions should not be lost in terms of contribution to the UK long-term This paper suggests that BSs research strategies that have pri- macroeconomic performance. marily focused on high quality scholarship and outputs are not enhancing the engagement agenda for economic impact. Knowledge impact. BSs can engage with the National Indus- The challenge for universities that wish to see their BSs engage trial Strategy, which outlines some of the major pillars (and its more fully with stakeholders, unlike many other faculties, is the successor launched in July 2020 - The Research and Devel- large student population and revenue generating role they have opment Roadmap) with one of its key objectives to ‘become for their organizations. It means prioritising day-to-day opera- world-class at securing the economic and social benefits from tional and strategic foci on delivering the annual surplus, pre- research’ (i.e. impact). The Industrial Strategy and its succes- dominantly from teaching income. Therefore, repositioning BSs sor include key points. BSs have research activity that span to make their research more highly engaged with stakehold- many of these themes such as supporting businesses to start and ers will need a major rethink, especially about the mission and grow, developing skills, encouraging trade and investment and implications for the theoretical approaches and focus on the improving procurement. So BSs understand the nature of academy. these changes. Research England’s Knowledge Exchange Framework could be a driver for change, as it seeks to make A more liberal approach to what is deemed rigorous and rel- available performance information about Universities knowl- evant needs recognising and given a greater weighting in reward edge exchange activities. The metrification of this activity will structures. There is no disputing BSs commitment to undertak- reinforce the importance to both universities and BSs. ing high quality scholarship, but that alone will not serve it or its diverse stakeholders and the research climate it is operating BSs aiming to deliver value creation strategies via knowledge in. The focus of output excellence over the last three decades on impact need to consider the issue of value contextualisation research assessment exercises cannot easily be reversed even (conceptualisation, construction, delivery and realisation). Then with the growing significance of ICS’. Developing ICS’ is a consider if value is symbolic or tangible. This will provide new skill for many academics to adopt, understand and imple- some stimulus for their knowledge impact. For example, is the ment with external partners that places their research at the heart beneficiary of the proposed value offer willing to pay a premium of the triple helix if executed skilfully (Ekzkowitz, 2008). We price? Branding is crucial, as the more symbolic and abstract have demonstrated that a ST approach to the analysis of ICS may the knowledge the higher will be the perceived knowledge help in understanding what pathways we need to adopt to reach and fee. that central point in the triple helix so as to create more impact- ful research not only for ICS’ but more generally that meet the Responsibility impact. BSs are inherently multidisciplinary, twin objectives of rigor and relevance. These findings illus- producing complexity in its constituent parts, and impact is a trate that there is strength in clustering research around areas of very serendipitous activity, based on networks and collabora- excellence based on longevity of service, research income and tive behaviour of individuals and groups. Stakeholders now teams which can be applied to address research problems over expect BS staff to be influential externally as well as within their a sustained period of time. Putting the ‘business’ back into BS
Page 11 of 20 Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
research appears to be the greater challenge from the analysis of upon the theoretical development in Science and how they have ICS’ if these are representative of the wider BS academy become successful (see Greenwood et al., 2002). It is the com- and impactful research. munication and relationship building to articulate relevance synergies with business that may need a rethink in the 21st In terms of knowledge and responsibility impacts, BSs have century. This goes to the very essence of BSs purpose, par- a national contribution to make towards policy and R&D, as ticularly their intended outward-facing mission so that business well as society (Morsing & Rovira, 2011) but the ICS evi- engagement is reinvigorated to give it meaning, where it is evi- dence does not immediately align BSs with the needs of denced through impactful research and applied problem-solving national policy where the underlying driver of the industrial capabilities locally, regionally, nationally and internationally. strategy is improving productivity and industrial growth. These issues are not peculiar to BSs and to an extent affect the percep- tion of Universities in the social sciences, where their contri- Data availability bution is not always directly associated with cause and effect Source data This link leads to information on the output and impact research relationships that are more visible in science, where scores of each submitting institution: https://results.ref.ac.uk/ there is more sustained engagement with business. (S(fpqtu01uy4jjci5ubrrqdp5x))/Results/ByUoa/19 Nevertheless, this study does provide BSs and Universities with a critical debate around the extent to which they produce This link leads to the 410 case studies that were submitted to impactful research, how they evidence that and more impor- the Business and Management Unit of Assessment: https:// tantly, communicate that to external stakeholders, drawing impact.ref.ac.uk/casestudies/Results.aspx?UoA=19
References
Aguinis H, Ramani RS, Alabduljader N, et al.: A pluralist conceptualization of Chartered Association of Business School. London. 2017. scholarly impact in management education: Students as stakeholders. Reference Source Academy of Management Learning & Education. 2019; 18(1): 11–42. Caplan B: The Case Against Education. Princeton University Press. 2018. Publisher Full Text Reference Source Aguinis H, Suárez-González I, Lannelongue G, et al.: Scholarly Impact Chowdhury G, Koya K, Philipson P: Measuring the impact of research: Revisited. Acad Manag Perspect. 2012; 26(2): 105–132. Lessons from the UK’s Research Excellence Framework 2014. PLoS One. Publisher Full Text 2016; 11(6): e0156978. Alajoutsijärvi K, Juusola K, Siltaoja M: The legitimacy paradox of business PubMed Abstract | Publisher Full Text | Free Full Text schools: losing by gaining? Academy of Management Learning & Education. Cooke A, Galt V: The impact of business schools in the UK. The Association of 2015; 14(2): 277–291. Business Schools. 2010. Publisher Full Text Edwards M: The Oxford Handbook of Civil Society. Oxford University Press, Amara N, Landry R: Counting citations in the field of business and Oxford. 2013. management: why use Google Scholar rather than the Web of Science. Reference Source Scientometrics. 2012; 93: 553–581. Publisher Full Text Ekzkowitz H: The Triple Helix: University-Industry-Government Innovation in Action. Routledge, New York. 2008. Amara N, Halilem N, Traoré N: Adding value to companies’ value chain: Role Reference Source of business schools scholars. J Bus Res. 2016; 69(5): 1661–1668. Publisher Full Text Elliott T: Fuzzy set qualitative comparative analysis. Research Notes: Statistics Group, UCI. 2013. Antonacopoulou EP: Making the Business School More ‘Critical’: Reflexive Reference Source Critique Based on Phronesis as a Foundation for Impact. British Journal of Management. 2010; 21(s1): s6–s25. Fiss P: A set-theoretic approach to organizational configurations. Acad Publisher Full Text Manage Rev. 2007; 32(4): 1180–1198. Publisher Full Text Azmat G, Murphy R, Valero A, et al.: Universities and industrial strategy in the UK (No. 539). Centre for Economic Performance, LSE. 2018. Garfield E:Citation indexing: Its Theory and Application in Science, Reference Source Technology, and Humanities. John Wiley, New York. 1979. Reference Source Bager T: Knowledge exchange and management research: barriers and potentials. European Business Review. 2018; 30(2): 169–182. Giddens A: The Third Way: The Renewal of Social Democracy. Polity, Publisher Full Text Cambridge.1998. Reference Source Beer M: Why management research findings are unimplementable: An action science perspective. Reflections: The Sol Journal. 2001; 2(3): 58–65. Greenwood R, Suddaby R, Hinings C: Theorizing Change: The Role of Reference Source Professional Associations in the Transformation of Institutionalized Fields. Acad Manage J. 2002; 45(1): 58–80. Bennis W, O’Toole J: How business schools have lost their way. Harv Bus Rev. Publisher Full Text 2005; 83(5): 96–104. PubMed Abstract Hall CM, Page S: Following the Impact Factor: Utilitarianism or academic compliance? Tour Manag. 2015; 51: 309–312. Birnik A, Billsberry J: Reorienting the business school agenda: The case Publisher Full Text for relevance, rigor, and righteousness. J Bus Ethics. 2008; 82(4): 985–999. Publisher Full Text Hamet J, Michel S: Rigor, relevance, and the knowledge “market”. European Bornmann L, Marx W: How good is research really? Measuring the citation Business Review. 2018; 30(2): 183–201. impact of publications with percentiles increases correct assessments and Publisher Full Text fair comparisons. EMBO Rep. 2013; 14(3): 226–30. Hardy G, Everett D: Shaping the Future of Business Education: Relevance. PubMed Abstract | Publisher Full Text | Free Full Text 2013. Bradshaw D: Rethinking business education: Fit for the future. The Publisher Full Text
Page 12 of 20 Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
Head BW, Alford J: Wicked problems: Implications for public policy and Edu. 2016; 15(4): 649–664. management. Administration & society. 2015; 47(6): 711–739. Publisher Full Text Publisher Full Text Phillips PA, Moutinho L, Godinho P: Developing and testing a method to Holmberg K, Didegah F, Bowman T: The different meanings and levels of measure academic societal impact. High Educ Q. 2018; 72(2): 121–140. impact of altmetrics. Accepted for oral presentation at the 11th International Publisher Full Text Conference on Webometrics, Informetrics and Scientometrics & 16th COLLNET Phillips PA, Subramanian KV, Newman V: Management Consultancy Through Meeting, 26-28 November, New Delhi, India. 2015. an Academic and Practitioner Perspective. Notionpress.com, India. 2019. Hughes T, Webber D, O’Reagan N: Achieving wider impact in business and Reference Source management: Analysisng the case studies from REF2014. Studies in Higher Phillips PA, Page SJ, Sebu J: Achieving research impact in tourism: Modelling Education. 2017; 44(3): 1–15. and evaluating outcomes from the UKs Research Excellence Framework. Publisher Full Text Tour Manag. 2020; 78: 104072. Irwin A: Re-making ‘quality’within the social sciences: The debate over Publisher Full Text rigour and relevance in the modern business school. Sociol Rev. 2019; 67(1): Ragin CC: Set relations in social research: Evaluating their consistency and 194–209. coverage. Polit Anal. 2006; 14(3): 291–310. Publisher Full Text Publisher Full Text Jackson SE, Schuler RS, Jiang K: An aspirational framework for strategic Roig-Tierno N, Huarng KH, Ribeiro-Soriano D: Qualitative comparative human resource management. Acad Manag Ann. 2014; 8(1): 1–56. analysis: Crisp and fuzzy sets in business and management. J Bus Res. 2016; Publisher Full Text 69(4): 1261–1264. Johnson S, Orr K: What is business school research for? Academic and Publisher Full Text stakeholder perspectives, politics and relationality. Stud High Educ. 2019; Rynes SL, Bartunek JM: Evidence-based management: Foundations, 45(5): 1–22. development, controversies and future. Annu Rev Organ Psych. 2017; 4: Publisher Full Text 235–261. Kapstein M, Yip G: The future of Business School research. In: Morsing, M. Publisher Full Text and Roviura, A. (Eds.): Business Schools and Contribution to Society. Sage: London, Saltera A, Saladrab R, Walker J: Exploring preferences for impact versus 2011; 114–124. publications among UK business and management academics. Res Policy. 2017; 46(10): 1769–1782. Kellard NM, Śliwa M: Business and Management Impact Assessment in Publisher Full Text Research Excellence Framework 2014: Analysis and Reflection. Brit J Manage 2016; 27(4): 693–711. Seal W: Some proposals for impactful management control research. Publisher Full Text Qualitative Research in Accounting Management. 2012; 9(3): 228–244. Publisher Full Text Kelly U, McNicoll I: Through a glass, darkly: Measuring the social value of universities. National Co-ordinating Centre for Public Engagement, Bristol. 2011. Shapiro DL, Kirkman BL, Courtney HG: Perceived causes and solutions of the Reference Source translation problem in management research. Acad Manage J 2007; 50(2): 249–266. Kieser A, Nicolai A, Seidl D: The practical relevance of management research: Publisher Full Text Turning the debate on relevance into a rigorous scientific research programme. Acad Manag Ann. 2015; 9(1): 143–233. Snelson-Powell AC, Grosvold J, Millington AI: Organizational hypocrisy in Publisher Full Text business schools with sustainability commitments: The drivers of talk- action inconsistency. J Bus Res. 2020; 114: 408–420. Kieser A, Leiner L: Why the Rigour–Relevance Gap in Management Research Publisher Full Text Is Unbridgeable. J Manage Stud. 2009; 46(3): 516–533. Teagarden MB, Von Glinow MA, Mellahi K: Contextualizing international Publisher Full Text business research: Enhancing rigor and relevance. J World Bus. 2018; 53(3): Lejeune C, Starkey K, Kalika M, et al.: The Impact of Business Schools: 303–306. Increasing the Range of Strategic Choices. Management International/ Publisher Full Text International Management/Gestión Internacional. 2019; 23(2): 88–98. Tho ND, Trang NTM: Can knowledge be transferred from business schools Publisher Full Text to business organizations through in-service training students? SEM and Mingers J, Yang, L: Evaluating journal quality: A review of journal citation fsQCA findings. J Bus Res. 2015; 68(6): 1332–1340. indicators and ranking in business and management. Eur J Oper Res. 2017; Publisher Full Text 257(1): 323–337. Tho ND: Knowledge transfer from business schools to business Publisher Full Text organizations: the roles absorptive capacity, learning motivation, acquired Morsing M, Rovira A: Business Schools and Their Contribution to Society. knowledge and job autonomy. J Knowl Manag. 2017; 21(5): 1240–1253. Sage, London. 2011. Publisher Full Text Publisher Full Text Thomas H, Wilson AD: ‘Physics Envy’, Cognitive Legitimacy or Practical Murcia MJ, Rocha HO, Birkinshaw J: Business schools at the crossroads? A trip Relevance: Dilemmas in the Evolution of Management Research in the UK. back from Sparta to Athens. J Bus Ethics. 2018; 150(2): 579–591. Brit J Manage. 2011; 22(3): 443–456. Publisher Full Text Publisher Full Text Narasimhan R: The fallacy of impact without relevance – reclaiming Trank CQ: “Reading” Evidence-Based Management: The Possibilities of relevance and rigor. European Business Review. 2018; 30(2): 157–168. Interpretation. Acad Manag Learn Edu. 2014; 13(3): 381–395. Publisher Full Text Publisher Full Text Page SJ, Hartwell H, Johns N, et al.: Case study: Wellness, tourism and small Valero A, Van Reenen J: The economic impact of universities: Evidence from business development in a UK coastal resort: Public engagement in across the globe. Econ Educ Rev. 2019; 68: 53–67. practice. Tour Manag. 2017; 60: 466–477. Publisher Full Text Publisher Full Text Van de Ven AH: Nothing is quite so practical as a good theory. Acad Manage Parker M: Shut down the business school. University of Chicago Press Rev. 1989; 14(4): 486–489. Economics Books Chicago. 2018. Publisher Full Text Reference Source Van de Ven AH, Heath V: Engaged Scholarship: A Guide for Organizational Pendlebury D: The use and misuse of journal metrics and other citation and Social Research. OUP Oxford. 2007. indicators. Arch Immunol Ther Ex. 2009; 57(1): 1–11. Reference Source Publisher Full Text Watermeyer R: Impact in the REF: Issues and obstacles. Stud High Educ. 2014; Penfield T, Baker MJ, Scoble R,et al.: Assessment, evaluations, and definitions 41(2): 199–214. of research impact: A review. Res Evaluat. 2014; 23(1): 21–32. Publisher Full Text Publisher Full Text Weybrecht G: The Launch of a Blueprint for SDG Integration in Business Pettigrew A, Starkey K: From the Guest Editors: The Legitimacy and Impact Schools. 2020. of Business Schools—Key Issues and a Research Agenda. Acad Manag Learn Reference Source
Page 13 of 20 Emerald Open Research Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
Open Peer Review
Current Peer Review Status:
Version 1
Reviewer Report 29 April 2021 https://doi.org/10.21956/emeraldopenres.15072.r27453
© 2021 Waltman L. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Ludo Waltman Centre for Science and Technology Studies, Leiden University, Leiden, Netherlands
First of all, I need to acknowledge that I have no specific expertise in business research. The paper extensively discusses the role of business schools, in particular in the UK. While this seems to be an important discussion, I don’t have the expertise to evaluate this element of the paper. I will focus my review primarily on the methodology adopted by the authors.
I appreciate the authors’ idea of using a methodology based on configuration theory (as opposed to a more commonly used methodology such as regression analysis). The authors take a set theoretical approach. Unfortunately, however, the paper does not provide enough methodological information to understand this approach in sufficient detail. Since the set theoretical approach taken by the authors is not very common, readers cannot be assumed to be familiar with this approach. It is therefore important to provide relatively detailed methodological information. Let me give a few examples of the lack of clear methodological information: ○ “Three-value fuzzy sets were used in this study. Three thresholds were determined to help with the calibration and the ratio-scale and interval scale variables for fuzzy membership scores. The thresholds included 0.95 for full membership (high impact score), 0.5 for cross- over point and 0.05 for full non-membership (low impact score).”: This is not clear to me. What do the values 0.95, 0.5, and 0.05 represent? Do they represent percentiles? Also, what happens when a value is below the lowest threshold (0.05)? Shouldn’t this also be interpreted as non-membership?
○ “The next step involved specifying the causal configurations and outcomes to be minimized.”: What do you mean by minimizing the outcomes? This is not clear to me. The rest of this paragraph is also hard to understand. The methodological information needs to be expanded. Providing an example may help to clarify the methodological details.
○ “The Quin-McCluskey algorithm is applied to simplify ST statements using the fsQCA software package (Roig-Tierno et al., 2016).”: It would be helpful if some discussion could be provided explaining what exactly the Quin-McCluskey algorithm is doing. Providing a full technical explanation may not be possible, but as a reader I would like to get at least a
Page 14 of 20 Emerald Open Research Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
proper intuitive understanding of what the algorithm is doing.
○ “The consistency measures according to Ragin (2006) are the measure of the degree to which each configuration term and the solution as a whole are subsets of the outcome. For the individual solutions, a solution is said to be consistent if membership in the solution is less or equal to membership in the outcome.”: I don’t understand this. This requires a more detailed explanation. In the first paragraph of the Results section, the authors should not use the term ‘impact factor’. The use of this term is confusing, because the term could easily be understood incorrectly as the journal impact factor.
Finally, the authors need to carefully reread their paper to fix some typos and minor inconsistencies in the text. Some sentences could also be phrased in a more elegant way (e.g., “a large body of secondary data that is predominantly based on qualitative data” and “theoretical debates on the rigor-relevance debate”).
Is the work clearly and accurately presented and does it cite the current literature? Partly
Is the study design appropriate and is the work technically sound? Partly
Are sufficient details of methods and analysis provided to allow replication by others? Partly
If applicable, is the statistical analysis and its interpretation appropriate? Partly
Are all the source data underlying the results available to ensure full reproducibility? Yes
Are the conclusions drawn adequately supported by the results? Partly
Is the argument information presented in such a way that it can be understood by a non- academic audience? Partly
Does the piece present solutions to actual real world challenges? Partly
Is real-world evidence provided to support any conclusions made? Partly
Could any solutions being offered be effectively implemented in practice? Partly
Page 15 of 20 Emerald Open Research Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: I have expertise in scientometrics and research evaluation, but I do not have expertise in business research.
I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.
Reviewer Report 03 December 2020 https://doi.org/10.21956/emeraldopenres.15072.r27256
© 2020 Davies J. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Julie Davies Manchester Metropolitan University, Manchester, United Kingdom
Feedback: Thank you for inviting me to review this article. Overall, it provides a useful overview of debates in the literature about tensions between management scholars and management practitioners. It draws on configuration theory to frame the analysis from a systemic perspective, specifically set theoretic configurational methods. The sample of REF 2014 impact cases in the UK is clearly explained. The findings are particularly interesting in terms of the context of impact case teams with a focus on economic, knowledge, and responsibility. This paper provides clear definitions and explanations about measuring research impact. It also offers emphasizes the need to demonstrate greater impact with businesses with insights that are generalizable beyond the UK.
The paper would benefit from reflections on its limitations. I would also like to see the authors’ thoughts on potential specific interventions. What future agenda on research impact related to communications and relation building do you propose in the light of consultations about REF 2027 and in the context of the pandemic?
Abstract: There is scope to sell the contribution of this paper as the only in-depth analysis of cases for business and management studies, the second largest unit of assessment. It would be helpful to mention the set theoretic (ST) methodology, configuration theory, and specific Tourism, Sport, Leisure and Hospitality case. Can you provide some indication of the answers to your question here: ‘what are the current themes and impacts of business and management research?’ Refer to your key findings about key factors that support impact case study success which are important takeaways: primarily winning significant research grants, as well as a large research team, reach of impact, low turnover of key staff. The insight that there is a need to put ‘business’ back into BS research is an interesting finding. This is also an important point to include in the abstract: ‘relationships across boundaries reside at the heart of successful ICs.’
Page 16 of 20 Emerald Open Research Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
Introduction: It would be useful to include an introduction that indicates the problem statement, key questions, theoretical framing, methodological approach, and structure of the paper.
Background: You provide a clear overview about debates relating to management academics and management practitioners as audiences. ○ Can you include a citation about your point that business schools are influential? It might be helpful to start with the broader picture first and then focus on the UK later in terms of justifying the choice of context and sample. ○ Include a citation for Tayloristic.
○ You might cite Bennis and O’Toole in reference to business schools losing their way.
○ SDG4 – note that this is about providing a quality education.
○ Consider the requirement in AACSB standards to demonstrate impact and EFMD’s BSIS in the business school sector. Research problem: The research questions are clearly stated. It would be useful to see some insights into why the research problem matters, e.g. for individuals’ careers, the legitimacy of business schools in society, and some indication perhaps about consultations regarding the REF after 2021 in terms of a greater focus on research teams (Rob Blackburn is a good point of contact) to alleviate current problems.
Definition and measurement of research impact: You might refer to the Australian context as this is where a national policy on evaluating research impact began. The UK approach appears to be based on PESTLE.
Theoretical challenges: You may find that these articles relevant here: ○ Johnson S. and Orr K. (2020) What is business school research for? Academic and stakeholder perspectives, politics and relationality1. ○ Ployhart, R.E. and Bartunek, J.M. (2019). Editors’ comments: There is nothing so theoretical as good practice—a call for phenomenal theory[ref2]. REF 2014 and impact case studies: It is worth noting the number of impact cases selected and that not all UK academics are engaged in the REF impact case studies agenda.
Methods: These are clearly explained drawing on an approach in a previous study.
Results: This should be in an earlier section: "Boolean logic was applied to determine commonalities among configurations that lead to the outcome and to generate logical statements. The Quin- McCluskey algorithm is applied to simplify ST statements using the fsQCA software package (Roig- Tierno et al., 2016)."
Main findings:
Page 17 of 20 Emerald Open Research Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
It would be useful to present these diagrammatically as a configuration. What are the limitations of the methodology and your analysis?
Conclusions: What practical solutions might be offered in terms of implications for practice looking forward to post REF 2021? How might publishers, business journalists and other stakeholders support communications and relationship building for impact? Further literature you might consider in relation to drivers for performativity in UK HEIs, the need for systemic change in UK business schools to support research impact, and multidisciplinary approaches through integrating the business school better within the university: ○ Watermeyer, R. (2019) Competitive accountability in academic life: The struggle for social impact and public legitimacy3. ○ Davies, J., Yarrow, E. and Syed, J., 2020. The curious under‐representation of women impact case leaders: Can we disengender inequality regimes?4. ○ Currie, G., Davies, J. and Ferlie, E., 2016. A call for university-based business schools to “lower their walls:” Collaborating with other academic departments in pursuit of social value 5. Presentation points: There are quite a few grammatical and punctuation points to correct. For example: ○ Keywords: Impact Set Theoretic Approach, Research performance – be consistent with using lower case and there should be a comma after impact. ○ Add ‘impact to’ before "their wider economic".
○ "its business model" - for plural BSs needs to be amended.
○ "Series" should be followed by 'has not have'
○ ", so the gap" – no comma.
○ Insert comma before "we formulate".
○ Omit comma: "But, these...".
○ "Growing number" should be followed by singular verb.
○ Number is repeated.
○ No comma before "or the reasons"
○ "To not be supported" – change to not to be
○ "Favouring" – use US spelling consistently
○ "Do you mean hard, physical science when you talk about Science/Such a seed…" - this sentence is very long. ○ "Towards" – use toward in US spelling, same for "regards".
○ Insert comma before "we delve".
○ BS’ and ICS’ needs to be corrected throughout.
○ Use either bullet points or a), b) etc., but not both.
○ Delete "the" before "Kellard".
○ Use ‘an’ not "a" before ST.
○ "Categorisation" should be 'categorization', same for "conceptualisation" – US spelling.
○ "To the UK long-term" – use UK’s.
○ "Goverment" – misspelt.
Page 18 of 20 Emerald Open Research Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
References 1. Johnson S, Orr K: What is business school research for? Academic and stakeholder perspectives, politics and relationality. Studies in Higher Education. 2020; 45 (3): 557-578 Publisher Full Text 2. Ployhart R, Bartunek J: Editors’ Comments: There Is Nothing So Theoretical As Good Practice—A Call for Phenomenal Theory. Academy of Management Review. 2019; 44 (3): 493-497 Publisher Full Text 3. R Watermeyer: Competitive accountability in academic life: The struggle for social impact and public legitimacy. 2019. 4. Davies J, Yarrow E, Syed J: The curious under‐representation of women impact case leaders: Can we disengender inequality regimes?. Gender, Work & Organization. 2020; 27 (2): 129-148 Publisher Full Text 5. Currie G, Davies J, Ferlie E: A Call for University-Based Business Schools to “Lower Their Walls:” Collaborating With Other Academic Departments in Pursuit of Social Value. Academy of Management Learning & Education. 2016; 15 (4): 742-755 Publisher Full Text
Is the work clearly and accurately presented and does it cite the current literature? Yes
Is the study design appropriate and is the work technically sound? Yes
Are sufficient details of methods and analysis provided to allow replication by others? Yes
If applicable, is the statistical analysis and its interpretation appropriate? Not applicable
Are all the source data underlying the results available to ensure full reproducibility? Yes
Are the conclusions drawn adequately supported by the results? Yes
Is the argument information presented in such a way that it can be understood by a non- academic audience? Yes
Does the piece present solutions to actual real world challenges? Yes
Is real-world evidence provided to support any conclusions made? Yes
Could any solutions being offered be effectively implemented in practice?
Page 19 of 20 Emerald Open Research Emerald Open Research 2020, 2:67 Last updated: 10 SEP 2021
Yes
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: The business and management education field, impact cases, UK context
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
Page 20 of 20