International Journal of 37 (2017) 1380–1390

Contents lists available at ScienceDirect

International Journal of

journal homepage: www.elsevier.com/locate/ijinfomgt

Revisiting the information audit: A systematic literature review and synthesis

Robert B. Frost, Chun Wei Choo ∗

Faculty of Information, University of Toronto, 140 St. George. St., Toronto, ON M5S 3G6, Canada article info a b s t r a c t

Article history: The purpose of this paper is to revitalize the theory and practice of the Information Audit (IA) by con- Received 25 May 2016 necting it with recent developments in information management theories and methods While the IA is Received in revised form 8 September 2016 a powerful information management practice, the methods and applications of IA have not been wed- Accepted 11 October 2016 ded to recent developments in the study of information management capability and information quality management. This study addresses that gap. The paper also introduces and applies a systematic method- Keywords: ology for conducting literature reviews that combines concept mapping, review scoping, and a structured Information audit search and analysis process. The resulting search in Scopus and Proquest and subsequent analysis of the Information management Information quality recent literature (2011–2016) on IA and quality, evaluation, measurement, and maturity in the con- Quality management text of information management yielded the following findings and recommendations. IA research and Literature review practice could do well to: pursue contingency frameworks rather than seek universal standardization; investigate the relationship between IA and the dimensions of information quality and information man- agement quality; undertake case studies that apply more foundational IA in full; develop theories of IA maturity and IA maturity modelling methods; recognize that measurement and evaluation of information management quality and information quality are necessary elements of the IA and should be explicitly incorporated into IA methodology. © 2016 Elsevier Ltd. All rights reserved.

1. Introduction cover every step of the baseline. Henczel’s methodology involves seven stages (pp. 18–19): The information audit (IA) has been defined by Buchanan and Gibb (2007) as “a holistic approach to identifying and evaluating an organization’s information resources and information flow, in order 1) Planning the audit by setting objectives, identifying stakehold- to facilitate effective and efficient information systems” (p. 171). ers, scoping the project and allocating resources, selecting a The IA provides “an invaluable structure of knowledge” in formu- methodology, and developing communications and business lating an organizational information strategy (Orna, 2004; p. 105), strategies. and as Buchanan and Gibb (2008) note, the IA’s influence on infor- 2) Collecting data in an information resources database, design- mation management, technology, systems, and content are well ing and distributing questionnaires, holding focus groups, and established in much of the foundational literature on IA (Buchanan conducting personal interviews. & Gibb, 1998; Burk & Horton, 1998; Henczel, 2001; Orna, 1999). 3) Analyzing the collected data and research. In its fullest form, the IA encompasses all the methods and tools 4) Evaluating gaps and duplications in information, mapping and needed to catalogue, model, evaluate, quality-control, and analyze interpreting information flows, formulating recommendations, an organization’s information assets and information management. and developing a change management plan. In their comparative analysis of the common IA methodologies 5) Communicating recommendations to stakeholders through established in the late 1990s and early 2000s, Buchanan and Gibb written reports, presentations and seminars, webpages, and per- (2008) propose a seven step methodological baseline for the IA and sonal feedback. find that the methodologies of Orna (1999) and Henczel (2001) 6) Implementing recommendations through implementation programs, formal change plans, post-implementation strategies, and information policies. 7) Ongoing information service management to measure and ∗ Corresponding author. E-mail addresses: [email protected] (R.B. Frost), [email protected] assess the changes through a regular information audit and ser- (C.W. Choo). vice evaluation cycle. http://dx.doi.org/10.1016/j.ijinfomgt.2016.10.001 0268-4012/© 2016 Elsevier Ltd. All rights reserved. R.B. Frost, C.W. Choo / International Journal of Information Management 37 (2017) 1380–1390 1381

Other established methodologies cover all seven stages of the the conceptual mapping process led to the development of an ana- baseline in different sequences or with added stages (e.g. Orna, lytical framework, which would ensure a consistent approach to 1999), do not provide guidance for the planning the audit (e.g. analysis in later stages of the literature review. Buchanan & Gibb, 1998), or do not provide guidance for planning Two perspectives on quality were considered: information qual- or change management following a report of the audit’s findings ity and information management quality. Information quality was (e.g. Burk & Horton, 1998). Griffiths (2012) notes that the tradi- understood with respect to Floridi’s (2013) characterization of tional, established IA methods are generally concerned with Hard information quality as the categories, dimensions, purpose-depth, IA, involving “notions of compliance, regulation and accuracy” (p. and purpose-scope that shape a unit of information. Informa- 43), rather than improving the usability of information assets, tion management quality was understood with respect to Mithas, increasing the efficiency of information use, or finding opportuni- Ramasubbu, and Sambamurthy’s (2011) characterization of high- ties for business innovation by changing information management quality information management capability as “the ability to practices (what Griffiths calls Soft IA). Henczel’s later emphasis on provide data and information to users with the appropriate levels user-centric interviews and focus groups, information flow map- of accuracy, timeliness, reliability, security, confidentiality, con- ping, as well as integration with change management and service nectivity, and access and the ability to tailor these in response to management showed that the IA could have applications beyond changing business needs and directions” (p. 238). Measurement the “hard” realm of accountability and compliance. and evaluation were understood interchangeably as appraisals of While the IA is a powerful information management practice, a quality criterion with reference to a specific performance indi- the methods of both hard and soft IA described in the literature have cator (e.g. the appraisal of a form’s accessibility with reference not been wedded to recent developments in the study of informa- to the quantity or severity of access barriers it contains). Matu- tion management capability (as described by Mithas, Ramasubbu, rity was understood with respect to Marchand, Kettinger, and & Sambamurthy, 2011) and information quality management (as Rollins’ (2001) view of information orientation maturity as a mix- conceptualized in the framework of Baskarada & Koronios, 2014). ture of highly developed information capabilities. Benchmarking Furthermore, measurement and evaluation techniques such as was understood as an alternative approach to maturity modelling information asset registration and maturity modelling remain in which an organization’s internal measurements are compared to largely absent from most IA methodologies, their synthesis with IA external measurements, rather than compared to internal targets. representing a direction for further research (Griffiths, 2012). With With the key concepts for analysis defined, their links to an IA the advent of widespread digital transformation and the rise of big were established in relation to Buchanan and Gibb’s (2007) descrip- data, it is now more important than ever for organizations to have tion of the IA as “a holistic approach to identifying and evaluating methods and tools for auditing and evaluating their information an organization’s information resources and information flow, in assets. order to facilitate effective and efficient organizational information With the aim of better connecting the IA literature with recent systems” (p. 171). In deconstructing Buchanan and Gibb’s descrip- information management theories and methods, this paper poses tion of the IA process, four implications for the linkages between an three research questions for investigation: IA and the other concepts under investigation in this paper become RQ1: What recent research (from 2011 to 2016) has been done evident: on IA? Implication 1: The IA requires an evaluation of information RQ2: What recent research (from 2011 to 2016) has been done resources and flows. on quality, evaluation, measurement, and maturity in the context Implication 2: The evaluation of information resources entails of information management? a measurement of information quality. RQ3: In the future, how might IA researchers and practitioners Implication 3: The evaluation of information flows entails a synthesize the recent research on IA with the recent research on measurement of information management quality. information management quality, evaluation, measurement, and Implication 4: The measures of information quality and maturity? information management quality are determinants of maturity, This paper will address the above three research questions contributing to effective and efficient information systems. through a systematic literature review, present and discuss the The four linkages derived from Buchanan and Gibb’s (2007) def- results of the review, outline the implications of the review for inition were formalized in a concept map, shown in Fig. 1. The IA researchers and practitioners, and describe the limitations of concept map illustrates the causal relationships between each of the review. To better aggregate the large volume of articles under the concepts under analysis, and also provides a basic rationale review, the discussion of results will provide a high-level overview model of IA goals and IA-maturity links for use in future maturity of recent trends in IA methods and theories rather than a com- modelling. plete analysis of every reviewed article. The paper offers unique With working understandings of the concepts under investiga- contributions in both its literature review methodology and its find- tion established and formalized in the concept map as a result of the ings on recent trends in IA and information management quality, preliminary analysis, it was possible to proceed with the creation evaluation, measurement, and maturity. literature review using a well-defined and consistent analytical framework.

2. Concept mapping

In preparation for the systematic literature review, a conceptual 3. Methodology map of the linkages between IA, information quality, information management quality, evaluation, measurement, and maturity was The literature review was systematic and performed in accor- created through a preliminary analysis of pivotal works from the dance with the methodology described by vom Brocke et al. (2009). IA and information management literature. Mindful of the fact that The methodology is rigorous in its approach, prescribing five phases researchers and practitioners in different domains often utilize dif- to the systematic literature review: scoping, conceptualization, ferent terminologies for similar concepts, the need to establish literature search, analysis/synthesis, and stating the expected con- working understandings of quality, evaluation, measurement, and tributions of the review to the broader research agenda. This maturity was recognized. Applying the working understandings to literature review largely followed the prescriptions of vom Brocke 1382 R.B. Frost, C.W. Choo / International Journal of Information Management 37 (2017) 1380–1390

Fig. 1. Concept map of the linkages between IA and the other concepts under investigation in this paper, with reference to Buchanan and Gibb’s (2007) definition of the IA.

et al., with some liberties being taken to adapt the methodology to Decision 4: The perspective of the review would ultimately be the specific needs of this study. argumentative rather than a neutral representation of the litera- To scope the review, vom Brocke et al. recommend using the ture, espousing a position on the best paths forward for future IA taxonomy of literature review characteristics proposed by Cooper research based on the review’s results. (1988). Table 1 utilizes Cooper’s scoping table to outline the over- Decision 5: The audience for the review would be specialized IA all scope of this review, producing the following methodological scholars and practitioners. decisions: Decision 6: The extent of the review’s coverage of the litera- Decision 1: To maintain a focus on methodological and con- ture would be representative, seeking a large enough sample of ceptual trends, the review would focus on methods and theories the literature to be representative of current trends rather than an of IA, information quality, information management quality, mea- exhaustive search of all recent literature. surement, evaluation, maturity, and benchmarking. A thorough After scoping the review, vom Brocke et al. recommend concep- discussion of the individuals, industries, and/or sectors involved tualizing the topic(s) to be studied—this phase of the methodology in the application of methods and theories would be beyond the development was already performed in the preliminary analysis, scope of the review, as would detailed analysis and criticism of the with the creation of the concept map. Accordingly, the concept map research outcomes of an empirical study. provided a conceptualization of the topics, which could be used to Decision 2: The goal of the review would be to integrate those inform the analysis and synthesis phase. methods and theories into insights on recent developments in IA In developing the literature search process, an adaptation of the and points of connection between IA and other recent areas of search process proposed by vom Brocke et al. was used, specifically, information management research. the adaptation used by Kowalczyk, Buxmann, and Besier (2013) in Decision 3: The results and discussion of the review would be their systematic review of the business intelligence and analytics organized conceptually, with reference to the conceptual mapping literature. The literature search process began with the creation performed during the review’s preliminary analysis. of inclusion criteria to determine the conditions under which an

Table 1 The scope of this review using the scoping table recommended by Cooper (1988), with this review’s characteristics highlighted in grey. R.B. Frost, C.W. Choo / International Journal of Information Management 37 (2017) 1380–1390 1383

Fig. 2. The information flow of the literature search process used in this review, with the entire process being performed once to address RQ1 and a second time to address RQ2. article would be included in or excluded from analysis. The five inclusion criteria in the Scopus and ProQuest user interfaces: only inclusion criteria decided upon were: articles published between 2011 and 2016 in scholarly journals, Criterion 1: The article was published in a peer-reviewed schol- trade journals, and conference proceedings were included in the arly journal, a trade journal, or the proceedings of an academic or search results. trade conference. With the query structures optimized and defined, two separate Criterion 2: The queried data source provided full text of the instantiations of the literature search process were carried out in article or a link to the full text of the article. response to RQ1 and RQ2, respectively. Both search processes were Criterion 3: The article was written in English. carried out in careful conformance with a predefined process model Criterion 4: The article was published between 2011 and 2016. (depicted in Fig. 2). Both search processes consisted of five steps: Criterion 5: The article examines one or more of the concepts Step 1: After querying the Scopus and ProQuest search engines, included in the concept map from an information management per- article information was extracted from Scopus and ProQuest and spective (papers on the topic of maturity or quality measurement results counts were recorded in an extraction log. from an operations management perspective with no relationship Step 2: The extracted articles had their titles and abstracts ana- to information management, for example, were excluded). lyzed to assess their degree of fit with the concept map and their Next, query structures were iteratively developed through trial potential value in answering the research question at hand. Arti- and error. Query structures were experimentally entered into the cles that were not well-suited to the conceptualization of the topic search engines of Scopus and ProQuest until a set of query struc- or had little value in answering the research question at hand tures, which were optimal for addressing the research questions were excluded from further consideration, and the count of arti- and inclusion criteria, had been identified. Two structured queries cles remaining after the title and abstract analysis was recorded in were created to extract articles from Scopus and ProQuest, which the extraction log. could provide potential answers to RQ1, and another two struc- Step 3: The remaining articles had their full text analyzed in tured queries were created to extract articles from Scopus and full to assess their degree of conceptual fit and potential value in ProQuest, which could provide potential answers to RQ2. All four greater detail. Again, articles that were not well-suited conceptu- query structures are documented in Appendix A of this paper. RQ3 ally or had little value in answering the research question at hand was to be addressed through a discussion of the literature review’s were excluded from further consideration. The count of articles results rather than through a query of its own. remaining after the full text analysis was recorded in the extraction To ensure the results more accurately reflected the inclusion log. criteria, the queries were accompanied by manual selections of

Table 2 Concept matrix template used to record and count conceptual information about the articles. The actual concept matrix resulting from the review can be found in Appendix B.

Concept Matrix Template

Year Author(s) Research Info. Audit IM Quality Info. Quality Measure- Maturity/ Focus ment/Evaluation Bench- marking

(Year) (Authors) 1384 R.B. Frost, C.W. Choo / International Journal of Information Management 37 (2017) 1380–1390

Table 3 Extraction log of the results from each step of both the literature search addressing RQ1 and the literature search addressing RQ2.

Extraction Log

Research Question Database Name Structured Query Results Title/Abstract Results Full Text Results

RQ1 Scopus 98 18 9 RQ1 ProQuest 194 6 4 RQ2 Scopus 289 5 4 RQ2 ProQuest 416 11 6 997 40 22

Table 4 tate analysis of trends in the literature over time, and the results of Total counts of the amount of articles adopting particular research and conceptual the research focus and conceptual reviewing for each article were foci from the pool of 22 articles remaining after the full text analysis. tallied. Concept Matrix Totals Table 4 depicts the total counts for each research and concep- Research Focus Article Count tual focus, which were observed in each of the 22 reviewed articles. Note that it is possible for a single article to have multiple research Outcomes 4 Methods 15 or conceptual foci, and thus the research and conceptual foci counts Theories 14 do not add up to 22. The research foci most often represented in the Applications 9 22 reviewed articles were methods and theories, with 15 articles adopting a methods focus and 14 articles adopting a theories focus. Concept Matrix Totals The conceptual foci most often represented in the 22 reviewed arti- Conceptual Focus Article Count cles were measurement/evaluation and information audit, with 13

Information Audit 12 articles analyzing and discussing measurement/evaluation in detail Information Management Quality 3 and 12 articles analyzing and discussing information audit in detail. Information Quality 10 A more detailed discussion of the trends revealed by the concept Measurement/Evaluation 13 matrix and the implications of those trends for researchers and Maturity/Benchmarking 5 practitioners will be presented in later sections of this paper. A full version of the concept matrix with research and conceptual foci Step 4: All of the articles remaining after the full text analysis had counts for each of the 22 articles can be found in Appendix B of this their authorship information and year of publication recorded in paper. an adaptation of the concept matrix template (depicted in Table 2) Considering the initial queries for RQ1 and RQ2 resulted in a total provided by Webster and Watson (2002). The articles were then of 997 articles being extracted from Scopus and ProQuest, it may reviewed in full a second time to discern their focus or foci with seem surprising that so few articles remained after full text analy- reference to the categories the scoping table: outcomes and results sis was performed. The significant gap between initial query results of research studies, methods of conducting research and analysis, and full text results is in large part attributable to the fact that many theories that conceptually model and explain the nature of a target initial query results did not approach the concepts under inves- of analysis, or applications of research methods and theories to spe- tigation from an information management perspective, and as a cific case studies or contexts. The second full text review was also result, use terms such as “information audit”, “information quality”, used to discern the extent to which they analyzed the concepts out- and “measurement” in senses, which are unrelated to the prevalent lined in the concept map. The research focus/foci and concept map conceptualizations of those terms within the information manage- relationships of each article were recorded in the concept matrix. ment literature. For example, in one excluded article, which had Step 5: The counts of each research focus and concept map rela- its title and abstract analyzed in response to RQ1, Yanjun (2013) tionship were tallied and used for analysis and synthesis in Section discusses highly technical methods of auditing the functionality of 5 of this review. computer systems, and in doing so, uses the phrase “massive infor- The literature searches in response to RQ1 and RQ2 were both mation audit” (p. 409) to describe the need for computer system carried out over the span of one week in March 2016. The rigor, auditors to develop new computer auditing methods in response reproducibility, and research question orientation of the method- to the big data paradigm. In another example of an excluded article, ology was expected to answer all three research questions posed which had its title and abstract analyzed in response to RQ2, Orozco, by this study, offering three sets of unique contributions to the IA Tarhini, Masa’deh, and Tarhini (2015) briefly discuss concepts of research agenda. measurement, evaluation, and maturity, but only in the context of improving IT governance architectures through increasing infor- 4. Results mation system and business alignment. The article does not analyze those concepts of measurement, evaluation, and maturity models Between both search processes, 997 initial results were in much detail, nor does it link its domain-specific perspective to a returned, 40 remained after title and abstract analysis, and 22 broader information management perspective or the relationships remained after full text analysis. In accordance with the method- expressed in this paper’s concept map. ology, a breakdown of the step-by-step results was recorded in an Despite a smaller sample of results remaining after the full text extraction log, which can be found in Table 3. analysis than was initially anticipated, the results are represen- All 22 of the articles remaining after the full text analysis had tative enough of recent trends in the IA literature and literature their authorship information and year of publication entered into related to the concept map established in the preliminary analy- a concept matrix. Their full text was then reviewed a second time sis to be able to answer RQ1 and RQ2. Recent developments and to determine the research and conceptual foci of each article. After trends in IA literature (addressing RQ1) and relevant information the research and conceptual foci of each article had been recorded management literature (addressing RQ2) will be discussed in the in the concept matrix, the results of the review were sorted in following section with reference to the literature search processes, descending order based on year of publication in order to facili- and ultimately, the findings from the two bodies of literature will R.B. Frost, C.W. Choo / International Journal of Information Management 37 (2017) 1380–1390 1385 be synthesized into implications for researchers and practitioners, 5.4. Thirdly, there have been few notable attempts to extend or which address RQ3. modify established IA methodologies for specific industries or specialized domain areas

5. Discussion The research foci of the remaining four IA applications are divided between methods and applications, with the authors Given the number of papers found in the literature search, it proposing their own extensions or modifications of established IA is not feasible to elaborate on the content of each item. Instead, methodologies, then applying their modified methods to a case we provide a big-picture view, which highlights the main contours study. Ariffin et al. (2014) propose modifications of IA methods of the IA and information management terrain covered by recent to better suit the electricity supply industry, determining that published research. In analyzing the literature, we sought out the IA implementation is currently a challenge in the industry due main areas of research emphasis, as well as the areas that require to the limited linkage of existing IA frameworks with systems research but have hitherto received little attention. development processes, the lack of scoping or resource sampling guidelines in most IA methods, the lack of standardized IA method- ologies, and the lack of change management guidelines to facilitate 5.1. Recent literature on IA the implementation of IA-driven recommendations. Mircea et al. (2012) devise a variation of the IA referred to as a “decision pro- To address the first research question (RQ1), our analysis of the cess audit” with five steps: planning the audit, identification of literature on IA leads us to make the following four observations key decisions, description of decision processes, evaluation of deci- that we believe characterize the state of recent research on the sion processes, and drafting the audit report. They then apply the subject. decision process audit method to a case study of a public pro- curement contract for photocopy repair and maintenance services 5.2. Firstly, much more attention has been given to theories and in the Bucharest University of Economic Studies. Ocholla (2011) methods of IA than IA research outcomes and applications leverages IA methods such as process mapping and resource inven- tory to propose research audit methods for mapping and auditing The recent theory-building observed in the IA literature always scholarly research, then conducts a research audit of the research goes hand-in-hand with more prescriptive guidelines and meth- projects and research output of the University of Zululand between ods of IA practice. For example, Fraser-Arnott (2014) develops a 1994 and 2008. Vo-Tran (2011) combines Henczel’s (2001) IA theoretical framework of technical, interpersonal, analytical, and methodology with an action research methodology, modifying the knowledge competencies within librarianship, which—when fused “implementing recommendations” stage of Henczel’s methodology with knowledge management techniques—form domain-specific to be better applicable to architectural design. Vo-Tran then applies methods and tools of knowledge auditing. A second example the hybrid methodology to a case study of the architectural design of IA theories and methods being fused include Buchanan and process of a new building at an Australian university. McMenemy’s (2012) discussion of the nature of elicitation, decom- position, and representation in process modelling methods used 5.5. Fourthly, there is a near absence of research that focuses on to model digital library information flow. In a third example, IA implementation and outcomes Griffiths (2012) analyzes of the fundamental, common character- istics between IA methodologies, the division between “hard” IA Only one article extracted from the literature search has a focus focused on compliance and “soft” IA focused on information map- on IA research outcomes: Carvalho and Esteban-Navarro’s (2016) ping, and the manner in which that division and the characteristics study of intelligence system auditing, in which the results of a of hard and soft IA are expressed in practical techniques of IA such literature review, participant observation, and case study are tri- as compliance management, information systems audit, and infor- angulated, enabling the authors to propose empirically-grounded mation asset registration. methods of auditing the informational and operational aspects of business intelligence systems. In their literature review, Carvalho and Esteban-Navarro separate audit methods into information 5.3. Secondly, there has been limited research that focuses on IA resources audit, information audit, knowledge audit, communica- applications and case studies tion audit, and intelligence audit, recognizing the unique value of each set of audit methods as part of a more holistic audit of intelli- Among the 22 articles remaining after the full text analysis, only gence systems. Their incorporation of “information ecology theory, five have a focus on IA applications (Ariffin, Latif„ Faudzi, Shariff, & information asset audits, user studies, knowledge management and Nadzir, 2014; Jones, Mutch, & Valero-Silva, 2013; Mircea, Ghilic- intelligence studies” (p. 58) into traditional IA methods, inspired Micu, & Stoica, 2012; Ocholla, 2011; Vo-Tran, 2011). Of those five by the outcome of their research, implies that traditional IA meth- articles, only Jones et al. (2013) have a pure focus on application, ods must be responsive and amenable to the growing importance applying the well-established IA methodology of Buchanan and of knowledge management and business intelligence systems if IA Gibb (1998) to a case study of information flow mapping at the is to remain a valuable information management practice in the Nottingham City Homes public housing service. Their case study current business environment. follows three steps: identifying the organizational context and pro- The lack of any outcome-focused empirical studies of IA imple- cess structure by reviewing business documentation; interviewing mentation beyond Carvalho and Esteban-Navarro’s indicates a staff members to determine the processes/activities involved in significant research gap for IA. Although many articles focus on their role, the information flows they mediate, and the information methods and theories, which extend the capability of the tradi- resources they utilize; analyzing and reporting on recommenda- tional IA methods, there is little research being done to analyze, tions, then seeking the implementation of those recommendations. criticize, and develop new methods and theories out of the planning Jones, Mutch, and Valero-Silva conclude that the results of their and implementation of specific IA projects. In studying applications application “affirm the value of the [IA] approach” (p. 297), but of IA methods and theories, it is worth noting that compared to the suggest that challenges exist in convincing key stakeholders to rec- early IA methodologies, the objectives of audits are increasingly ognize the value of and engage with the information audit process. taking a hybrid approach, auditing for hard IA reasons of compli- 1386 R.B. Frost, C.W. Choo / International Journal of Information Management 37 (2017) 1380–1390 ance and accountability, as well as soft IA reasons of usability and mation quality, were reviewed in total. Of those 10 articles, seven innovation. However, there does not yet exist a contingency model conceptually overlap with measurement/evaluation (including the outlining which hard IA or soft IA methods an organization should three examples of information systems management articles noted adopt based on the audit context. above). This close relationship between information quality and measurement/evaluation is somewhat unsurprising, given the fact 5.6. Recent literature on quality, measurement/evaluation, & that information quality is aptly defined by one of those 10 articles maturity/benchmarking as “a measure [emphasis added] of where the information consis- tently meet [sic] all user expectations” (Gunawan & Suhardi, 2014, In response to the second research question (RQ2), and p. 1). The lack of overlap between information quality and IA con- based on our analysis of the recent literature, we again make ceptual foci, though, is more surprising: out of all of the articles a number of observations that we believe characterize the state reviewed, only Griffiths (2012) discusses both information qual- of recent research on quality, measurement/evaluation, matu- ity and IA. In fact, in discussing the need to combine business rity/benchmarking in the information management context. intelligence and compliance functions to better manage informa- First, we note that the greatest research attention has been given tion quality, Griffiths voices “concern that IA practitioners may be to measurement/evaluation. Of the concepts from the concept map placing emphasis on the quality of the subject organization’s data which were investigated in the literature search process addressing processing when the key element should be an assessment of the RQ2, the concept with the greatest attention paid to it was mea- quality of the data itself” (p. 45). While Griffiths does not provide surement/evaluation, with a total of 13 articles including it as a a framework for assessing information quality as part of an orga- conceptual focus (see Appendix B for the full mapping of every arti- nization’s IA, his concern about data and information quality being cle aligned with the measurement/evaluation focus). Of those 13 overlooked within IA practice signals the need for researchers and articles, however, only four discuss measurement/evaluation from practitioners to begin bridging this conceptual gap. an information management perspective in addition to discussing Third, there has been limited research attention on matu- the concept of IA (Buchanan & McMenemy, 2012; Jones et al., 2013; rity/benchmarking in the information management context. In Mircea et al., 2012; Shamel, 2014). Shamel (2014) situates the IA contrast to measurement/evaluation and information quality, within a larger knowledge assessment process, first formulated the concept of maturity/benchmarking in the context of infor- by Henczel (2000), in which the information audit is preceded mation management received relatively little attention in the by needs analysis and followed by a knowledge audit. Shamel reviewed literature: only five articles analyze and discuss matu- then suggests methods of measuring the gap between informa- rity/benchmarking (Chuah, 2014; Gunawan & Suhardi, 2014; Zou, tion and knowledge management goals and the actual performance Flanagan, Jewell, & Tang, 2013; Chuah & Wong, 2012; Pranicevic,´ revealed by the audit, as well as a list of common strengths and Alfirevic,´ & Stemberger,ˇ 2011). Following an initial study (Chuah weaknesses to be vigilant for during evaluation of the audit results. & Wong, 2012), which determined that further exploration of the Jones et al. (2013) describe the methods that they used to have staff dimensions of business intelligence quality and maturity were members of an organization, which is undergoing an IA, evaluate needed, Chuah (2014) constructs a maturity model for enterprise their organization’s information resource usage with reference to business intelligence, linking the development of organizational a set of best practices. Buchanan and McMenemy (2012) discuss and information-managerial capabilities to the maturity level of measures of success and common pitfalls in modelling information the enterprise business intelligence system. Gunawan and Suhardi flows as part of an IA, such as failing to elicit vital information from (2014) create a theoretical and methodological framework for total staff, failing to decompose information flow processes appropri- information quality management, combining quality management ately, and underrepresenting less tangible activities or resources. principles from the total quality management and information Mircea et al. (2012) discuss the five steps of evaluating decision quality management schools into an evaluable, five-level maturity process quality as part of a decision process audit: identifying the model. Zou et al. (2013) create an information maturity model tai- dimensions of the managerial situation, developing alternatives, lored to the decision-making needs of the construction industry, analyzing and evaluating alternatives, choosing and implement- proposing that a unit of information’s overall maturity is composed ing the best alternative, and evaluating the overall quality of the of four measures: its format, informational value, informational decision. quality, and the timeliness of its delivery to end users. Based on Other articles in the segment of measurement/evaluation lit- their empirical study of the relationship between information sys- erature, which do not share a conceptual focus on IA, tend to tem maturity and hotel performance measurement, Pranicevic´ et al. focus on the management of information systems. For example, (2011) introduce a maturity model for information systems in the Visser, van Biljon, and Herselman (2013) analyze frameworks and hospitality industry. Despite the breadth of domain-specific theo- survey tools for evaluating the performance of management infor- ries and methods of maturity modelling displayed throughout the mation systems in Further Education and Training colleges. Petter, reviewed literature, no article was found with conceptual over- DeLone, and McLean (2012) review the historical evolution of infor- lap between IA and maturity/benchmarking, leaving a gap in the mation systems success measures followed by the suggestion of research to be filled by a maturity model of IA practice. research directions for new success measures. Mohammed and Finally, the concept with the least attention paid to it was informa- Yusof (2012) offer criteria for effective information quality man- tion management quality. Only three articles analyze and discuss the agement in health information systems. All of these articles present concept (Fraser-Arnott, 2014; Sheriff, Bouchlaghem, El-Hamalawi, novel approaches to the measurement and evaluation of informa- & Yeomans, 2012 ;Griffiths, 2012). Sheriff et al. (2012) explore the tion systems, but are not connected to the literature on IA. However, drivers of and barriers to effective information management in all three of the above examples share an additional conceptual UK-based architecture and engineering organizations, determin- focus on information quality, indicating the importance of not only ing from the outcome of their study that the ability to improve end managing information systems, but managing the information they products, the ability to improve processes, the effective transfer contain as well. of learning, legal and regulatory compliance, and risk mitigation Second, we note a surprising lack of overlap between information are all drivers of information management quality. However, Sher- quality and IA as conceptual foci in the literature. On the whole, iff et al.’s unique perspective on information management quality information quality also received a great deal of attention in the does not share a conceptual focus on IFraser-Arnott (2014) and reviewed literature: 10 articles, which analyzed or discussed infor- Griffiths (2012) do exhibit conceptual foci on both IA and informa- R.B. Frost, C.W. Choo / International Journal of Information Management 37 (2017) 1380–1390 1387 tion management quality, but their engagement with the drivers, a case study—as exemplified in Ariffin et al. (2014), Mircea et al. barriers, and dimensions of information management quality is not (2012), Ocholla (2011), and Vo-Tran (2011)—but there is only one as direct and thorough as Sheriff et al.’s. As a result, there is a gap instance of a foundational IA methodology being applied to a case in the research to be filled by an outcome-focused empirical study study in full, as exemplified in Jones et al. (2013). Moreover, there of the causal relationships between IA methods and information are few articles focused on the research outcomes of empirical management quality. studies of IA implementation compared to articles focused on IA methods, theories, and applications. Future research focusing on 6. Implications for researchers the outcomes of applying foundational and recent IA methods and theories could make a significant contribution to the IA literature Our characterization of the recent literature in IA and informa- by reporting on the strengths and weaknesses of IA methodologies tion management allows us to identify the areas of emphasis in the in different use contexts, resulting in more empirical studies of IA research, as well as the gaps and opportunities for further research. outcomes. Based on the discussion in the last section, we propose four direc- tions for future IA research that would address RQ3, that is, how 6.4. Direction 4: develop theories of IA maturity and IA maturity might IA researchers better synthesize the recent research on IA modelling methods and information management? In the reviewed literature, there is no article which exhibits 6.1. Direction 1: pursue contingency frameworks rather than conceptual overlap between IA and maturity. Griffiths (2012) has standardization suggested that the development of a maturity model represents a future direction in his research, and indeed, this review suggests Ariffin et al. (2014) and Griffiths (2012) have recently lamented that theories of IA maturity and IA maturity modelling methods that the lack of standardization in IA methodologies is a central rea- are future research directions, which still remain unrealized. son for IA failing to be put into widespread practice. Buchanan and Gibb’s (2008) “methodological baseline” for IA practice contends 6.5. Contribution to research methodology that IA methodology selection should be based on organizational requirements and auditor experience, but stops short of building In addition to its implications for the content of future IA a contingency framework of context-specific methods and tools, research, this literature review claims a unique methodological which could supplement the baseline. This review’s finding that contribution, which stands to benefit all information management most recent IA research focuses on methods, signals that the prolif- researchers. vom Brocke et al. (2009) provided the overall structure eration of different methodological variations and permutations of of this review’s methodology, Kowalczyk et al. (2013) provided a the IA has become too spread out between different use cases and refinement of the literature search process, Webster and Watson industry-specific needs to be effectively standardized. In light of (2002) provided the concept matrix template, and Cooper (1988) that finding, it appears that the pursuit of cross-industry or industry provided the scoping table template. However, to the knowledge standardization is a dead end. of the author, the creation of a concept map and its direct integra- Future research should instead focus on designing contingency tion into research questions, an extraction log, concept matrix, and frameworks for IA practice, with different sets of methods and results-driven conceptual analysis and synthesis has not previously tools being prescribed for different industries, business scenarios, been performed in the same manner as this review. Additionally, and information cultures. For example, aligning particular sets of this review’s use of a detailed data flow model to illustrate the spe- methods and tools with the relationship-based, risk-taking, rule- cific actions involved in the literature search process and make the following, and results-oriented information cultures typologized process easily replicable from the diagram alone is unique in the by Choo (2013) could prove more widely valuable and applicable information management literature. This methodology is ideal and than a one-size-fits-all approach to IA methodology. easily replicable for potentially any literature review, which aims to sample only a representative portion of the literature and syn- 6.2. Direction 2: explore the relationship between IA and quality thesize the findings, but strive for a high degree of rigor in doing dimensions in more detail so. Because this methodology is so easily replicable, it is an ideal Despite the close conceptual link between IA, information man- candidate for the methodology of future IA reviews. An update to agement quality, and information quality established in this paper, this literature review could be conducted years after the publication the relationship between those concepts is scarcely explored in the of this review using the exact same methodology, and an expansion reviewed literature. Only Fraser-Arnott (2014) and Griffiths (2012) of this literature review to analyze more data sources and concepts discuss the nexus of IA and information management quality, and could easily be done by making minor adjustments to the concept only Griffiths discusses the nexus of IA and information quality. map, extraction log, and concept matrix. Future research, which seeks a deeper understanding of the causal relationships between IA and quality, will not only provide novel 7. Implications for practitioners contributions to the research agenda, but would also validate the power of the IA if a causal link between IA practice and heightened In a series of post-hoc discussions, the authors of the paper quality could be empirically demonstrated. reflected further on the findings of our literature survey, and arrived at three recommendations for IA practitioners. 6.3. Direction 3: apply more foundational IA methodologies in full Recommendation 1: Measurement and evaluation of informa- to case studies tion management quality and information quality are necessary parts of the IA. Define information management quality and infor- There is a shortage of case studies applying foundational IA mation quality dimensions in advance of the audit, then measure methodologies such as Burk and Horton (1998), Buchanan and Gibb and evaluate those quality dimensions as part of the audit. Mithas (1998), Orna (1999), and Henczel (2001). In the reviewed literature, et al. (2011) implicitly offer insights into the quality dimensions of researchers will often propose their own modifications of foun- information management through their examination of informa- dational IA methodologies and apply the modified components to tion management capability factors; Floridi (2013) offers a readily 1388 R.B. Frost, C.W. Choo / International Journal of Information Management 37 (2017) 1380–1390 useable typology of information quality dimensions. Measurement 9. Conclusion and evaluation of the quality dimensions can then be incorporated into the phase of an IA methodology, which involves evaluation This literature review posed and answered three research ques- and qualitative judgment of the audit’s findings, such as Henczel’s tions, which aimed to better discern recent developments in the (2001) “data evaluation” phase or Buchanan and Gibb’s (2008) IA literature, recent developments in quality, evaluation, measure- “account” phase. ment, and maturity in the information management literature, Recommendation 2: In the reviewed literature, modelling and how those two sets of developments could be synthe- and diagramming techniques are rarely recommended or situ- sized with one another. A conceptualization of the topics was ated within a larger IA process, but the use of these techniques mapped out, and a systematic literature review methodology was is essential to the holistic mapping of information resources and described. The review was conducted and the results of the review flow. Observing Buchanan and Gibb’s (2007) separation of resource, were presented and discussed, with special attention paid to process, and strategic perspectives when conducting the IA, there the research foci and conceptual foci of the reviewed literature. is a need for using different modelling and diagramming tech- Implications of the review for researchers and practitioners were niques when auditing within each of the three perspectives. then outlined, and the limitations of the review were acknowl- Use entity-relationship diagrams and/or classification schemes to edged. model information resources when working in the resource per- This literature review indicates that interest in IA research has spective; use data flow, UML, and/or BPMN diagrams to model remained highly specialized yet steady since 2011. Embedding information flows when working in the process perspective; use recent developments in information management, information Allee’s (2000) value network diagram and/or the strategic view dia- quality, measurement and evaluation, and maturity and bench- gram template found in Buchanan and Gibb (2007, p. 169) to model marking more deeply into IA theories, methods, and applications alignment between information resources, flows, and strategies could propel interest in IA research up to higher levels. The when working the strategic perspective. practicality of IA methodologies could be improved by having Recommendation 3: Document and publish more applications more IA case studies on record, as well as through the design of IA methods so that researchers and practitioners have a greater of IA contingency frameworks, IA maturity models, and informa- knowledge base to draw from in studying the performance of dif- tion/information management quality measurement techniques. ferent IA method and in innovating new IA methods and tools. The IA is clearly an extremely valuable information management practice, but there is still much work to be done by both IA 8. Limitations researchers and practitioners to make the practice more widely known and used. It is unfortunate that at a time when information In advance of this review being conducted, the coverage of management has become a necessity in nearly every industry and this literature review was predefined as representative rather than sector, IA research and practice have lagged behind other facets exhaustive, as outlined in Section 3. Accordingly, only two major of information management. There now exists a great opportu- academic databases (Scopus and ProQuest) were queried as part of nity for an intellectual renewal of IA research and practice, and the literature review. The body of literature on IA is quite small and it is hoped that this literature review will serve as a guidepost for highly specialized, so an exhaustive review encompassing more anyone seeking to push the IA agenda forward. data sources will likely not produce any significant insights into RQ1 beyond what has been covered in this review. Nonetheless, Funding this review’s coverage of only two data sources does constitute a limitation of the study. This research did not receive any specific grant from funding A more significant limitation arises from the query structures, agencies in the public, commercial, or not-for-profit sectors. which were used to address RQ2 (found in Appendix A). Those query structures were designed to limit the query results to articles, Appendix A. Query Structures which discussed information management, information quality, and one other conceptual keyword, with the assumption that those The query structure used to extract articles from Scopus in articles would be of greatest relevance to the scope of this lit- response to RQ1 was structured as follows: erature review. In the experimental development of the query information audit f g structures; query structures seeking a match for only one con- The query structure used to extract articles from ProQuest in ceptual keyword were tested. Unfortunately; each of those query response to RQ1 was structured as follows: structures resulted in tens of thousands of results being returned “information audit” NOT auditory from both Scopus and ProQuest. If the number of data sources The query structure used to extract articles from Scopus in were expanded beyond two; the total number of query results per- response to RQ2 was structured as follows: taining to any one conceptual focus could easily extend into the information management AND information quality AND f g f g hundreds of thousands. Although an exhaustive review of the lit- (“evaluat*” OR “measure*”) AND (“maturity” OR “benchmark*”) erature on information management quality; information quality; The query structure used to extract articles from ProQuest in and measurement/evaluation/maturity/benchmarking in the con- response to RQ2 was structured as follows: text of information management would surely reveal enormous (“information management” AND “information quality”) AND potential for cross-pollination with IA theories and methods; such (“evaluat*” or “measure*”) AND (“maturity” OR “benchmark*”) a review would likely not be possible without a data mining and analysis tool custom-built for the research project. R.B. Frost, C.W. Choo / International Journal of Information Management 37 (2017) 1380–1390 1389

Appendix B. Full Concept Matrix

Concept Matrix

Reference Research Focus Info. Audit IM Quality Info. Quality Measure- Maturity/Bench- ment/Evaluation marking

Carvalho and Esteban-Navarro (2016) Outcomes, Methods x Ayyash (2015) Theories x Fraser-Arnott (2014) Theories, Methods x x Ariffin et al. (2014) Methods, Application x Shamel (2014) Theories, Methods x x Chuah (2014) Theories x x Gunawan and Suhardi (2014) Theories, Methods x x x Bastos, Moreira, Bruno, Filho, and Filho (2014) Theories, Methods, Application x x Jones, Mutch, and Valero-Silva (2013) Application x x Zhi-hong, Da-wei, and Hong-yun (2013) Methods x Visser et al. (2013) Theories, Methods, Application x x Zou et al. (2013) Theories, Application x x x Mohammed and Yusof (2013) Methods, Application x x Sheriff et al. (2012) Outcomes, Theories x Reifsnider (2012) Methods x Buchanan and McMenemy (2012) Theories, Methods x x Griffiths (2012) Theories, Methods x x x Chuah and Wong (2012) Outcomes x x x Petter et al. (2012) Theories x x Mircea et al. (2012) Methods, Application x x Ocholla (2011) Theories, Methods, Application x Vo-Tran (2011) Methods, Application x Pranicevic´ et al. (2011) Outcomes, Theories x x 12 3 10 13 5

References

Allee, V. (2000). Reconfiguring the value network. Journal of Business Strategy, Henczel, S. (2000). The information audit as a first step towards effective 21(4), 36–39. knowledge management: An opportunity for the special librarian. INSPEL, Ariffin, I., Latif, A. A., Faudzi, M. A., Shariff, S. S., & Nadzir, M. M. (2014). Information 34(3), 210–226. audit in electricity utilities: Roles, methodologies, issues and challenges. 2014 Henczel, S. (2001). The information audit: A practical guide. Munich: K.G. Saur. international conference on computer and information sciences (ICCOINS), 1–6. Jones, A., Mutch, A., & Valero-Silva, N. (2013). Exploring information flows at Ayyash, M. M. (2015). Identifying information quality dimensions that affect nottingham city homes. International Journal of Information Management, 33, customers satisfaction of e-banking services. Journal of Theoretical and Applied 291–299. Information Technology, 82(1), 122–130. Kowalczyk, M., Buxmann, P., & Besier, J. (2013). Investigating business intelligence Baskarada, S., & Koronios, A. (2014). A critical success factor framework for and analytics from a decision process perspective: A structured literature information quality management. Information Systems Management, 31, review. ECIS 2013 completed research [Paper 126]. 276–295. Marchand, D. A., Kettinger, W. J., & Rollins, J. D. (2001). Information orientation: The Bastos, C. A. M., Moreira, M. R., Bruno, A. C. M., Filho, S. M., & Filho, J. R. F. (2014). link to business performance. New York: Oxford University Press. Information flow modeling: A tool to support the integrated management of Mircea, M., Ghilic-Micu, B., & Stoica, M. (2012). Information audit for decision information and knowledge. KMIS 2014–International conference on knowledge processes. Economic Computation and Economic Cybernetics Studies and management and information sharing, 76–86. Research, 3, 41–58. Buchanan, S., & Gibb, F. (1998). The information audit: An integrated strategic Mithas, S., Ramasubbu, N., & Sambamurthy, V. (2011). How information approach. The International Journal of Information Management, 18(1), 29–47. management capability influences firm performance. MIS Quarterly, 35(1), Buchanan, S., & Gibb, F. (2007). The information audit: Role and scope. 237–256. International Journal of Information Management, 27, 159–172. Mohammed, S. A., & Yusof, M. M. (2013). Towards an evaluation framework for Buchanan, S., & Gibb, F. (2008). The information audit: Methodology selection. information quality management (IQM) practices for health information International Journal of Information Management, 28, 3–11. systems – Evaluation criteria for effective IQM practices. Journal of Evaluation Buchanan, S., & McMenemy, D. (2012). Digital service analysis and design: The role in Clinical Practice, 19, 379–387. of process modelling. International Journal of Information Management, 32, Ocholla, D. N. (2011). An overview of issues, challenges, and opportunities of 251–256. scholarly publishing in information studies in Africa African Journal of Library. Burk, C. F., & Horton, F. W. (1998). InfoMap: A complete guide to discovering Archives and Information Science, 21(1), 1–16. corporate information resources. Englewood Cliffs, NJ: Prentice-Hall. Orna, E. (1999). Practical information policies (2nd ed.). Aldershot: Gower. Carvalho, A. V., & Esteban-Navarro, M. (2016). Intelligence audit: Planning and Orna, E. (2004). Information strategy in practice. Aldershot: Gower. assessment of organizational intelligence systems. Journal of Librarianship and Orozco, J., Tarhini, A., Masa’deh, R. M. T., & Tarhini, T. (2015). A framework of Information Science, 48(1), 47–59. IS/business alignment management practices to improve the design of IT Choo, C. W. (2013). Information culture and organizational effectiveness. governance architectures. International Journal of Business and Management, International Journal of Information Management, 33, 775–779. 10(4), 1–12. Chuah, M., & Wong, K. (2012). A framework for accessing an enterprise business Petter, S., DeLone, W., & McLean, E. R. (2012). The past, present: And future of IS intelligence maturity model (EBI2M): Delphi study approach. African Journal of success. Journal of the Association for Information Systems, 13, 341–362. Business Management, 6(23), 6880–6889. Pranicevic,´ D. G., Alfirevic,´ N., & Stemberger,ˇ M. I. (2011). Information system Chuah, M. (2014). The impact of capability level on maturity of business maturity and the hospitality enterprise performance. Economic and Business intelligence initiatives. IJCSI International Journal of Computer Science Issues, Review, 13(4), 227–249. 11(6–1), 144–149. Reifsnider, C., 2012. Instituting a business intelligence process. Online, May–June Cooper, H. M. (1988). Organizing knowledge syntheses: A taxonomy of literature 2012, 28–32. reviews. Knowledge in Society, 1, 104–126. Shamel, C. (2014). Planning for knowledge management: Conducting a knowledge Floridi, L. (2013). Information quality. Philosophy and Technology, 26(1), 1–6. assessment. In U. de Stricker (Ed.), Knowledge management practice in Fraser-Arnott, M. (2014). Moving from librarian to knowledge manager. organizations 59–97. Hershey, PA: IGI Global. Partnership: the Canadian Journal of Library and Information Practice and Sheriff, A., Bouchlaghem, D., El-Hamalawi, A., & Yeomans, S. (2012). Information Research, 9(2), 1–10. management in UK-based architecture and engineering organizations: Drivers, Griffiths, P. (2012). Information audit: Towards common standards and constraining factors, and barriers. Journal of Management in Engineering, 28(2), methodology. Business Information Review, 29(1), 39–51. 170–180. Gunawan, I. G. N. A. R., & Suhardi. (2014). Designing process maturity measure of Visser, M., van Biljon, J., & Herselman, M. (2013). Evaluation of management total information quality management (TIQM). 2014 2nd international information systems: A study at a further education and training college. SA conference on business and information management (ICBIM), 47–52. Journal of Information Management, 15(1) [Art. #531]. 1390 R.B. Frost, C.W. Choo / International Journal of Information Management 37 (2017) 1380–1390 vom Brocke, J., Simons, A., Niehaves, B., Niehaves, B., Reimer, K., Plattfaut, R., et al. Robert B. Frost is an information management researcher with a Master of Informa- (2009). Reconstructing the giant: On the importance of rigour in documenting tion degree from the University of Toronto. His research interests include methods the literature search process. ECIS 2009 proceedings [Paper 161]. of auditing information assets, quality dimensions of information management, and Vo-Tran, H. (2011). Adding action to the information audit. The Electronic Journal methods of leveraging information assets to improve service quality. His Masters Information Systems Evaluation, 14(2), 167–282. degree culminated in a capstone project, which involved the design and pilot- Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: testing of a holistic audit framework for the cataloguing, modelling, and quality Writing a literature review. MIS Quarterly, 26(2), xiii–xxiii. management of information assets. Yanjun, T. (2013). Study on computer audit based on OLAP technology. Journal of Digital Information Management, 11(6), 409–417. Chun Wei Choo is Professor of the Faculty of Information at the University of Zhi-hong, Z., Da-wei, X., & Hong-yun, L. (2013). Research on intellectual property Toronto. His recent books include The Inquiring Organization (2016, Oxford Uni- information audit based on independent innovation. Advanced Materials versity Press); The Knowing Organization (2nd ed., 2006, Oxford University Press); Research, 860–863, 3100–3103. The Strategic Management of Intellectual Capital and Organizational Knowledge Zou, R. R., Flanagan, R., Jewell, C., & Tang, L. C. M. (2013). An exploratory study of (2002, Oxford University Press); and Information Management for the Intelligent information maturity in construction and developing a decision-making Organization (3rd ed., 2002, Information Today). model. In S. D. Smith, & D. D. Ahiaga-Dagbui (Eds.). Proceedings of the 29th annual ARCOM conference, 69–80.