. ) Preface ILLINOIS STATE UNIVERSITY GAO assists congressional decisionmakers in their deliberations by furnishing them with analytical information. Many diverse methodologies are needed to develop sound and timely answers to the questions that the Congress poses. To provide GAO evaluators with basic information about the more commonly used methodologies, GAO's policy guidance includes documents such as methodology transfer papers and technical guidelines. · This transfer paper on content analysis describes how GAO can use this methodology in its audits and evaluations. It defines content analysis and details how to decide whether it is appropriate and, if so, how to develop an analysis plan. The paper also specifies how to code documents, analyze the data, and avoid pitfalls at each stage. Several software packages useful for GAO audits and evaluations are described. Content Analysis: A Methodology for Structuring and Analyzing Written Material is one of a series of papers issued by the Program Evaluation and Methodology Division (PEMD). The purpose of the series is to provide GAO evaluators with guides to various aspects of audit and evaluation methodology, to illustrate applications, and to indicate where more detailed information is available. Page 1 GAO/PEMD-10.3.1 Content Analysis )) 0 c. A J, 13 '. Preface O_C"\'1 D - \C -::: ·) L I 1 I ! __.,Jr ...,,c, We look forward to receiving comments from the readers of this paper. They should be addressed to Joseph F. Delfi.co at (202) 512-2900. Brian P. Crowley Assist.ant Comptroller General Office of Policy Joseph F. Delfi.co Acting Assistant Comptroller General Program Evaluation and Methodology Division Page2 GAO/PEMD-10.3.1 Content Analysis ------ -~--- -------------------- --- ~-------~------ .\ Page3 GAO/PEMD-l0.3.1 Content Analysis Contents Preface 1 Chapter 1 6 A Definition of Content Analysis 6 What Content The Uses of Content Analysis 7 Analysis Is Computerized Content Analysis 9 Some Advantages of Content Analysis 9 Some Disadvantages of Content Analysis 10 How to Apply Content Analysis 11 The Organization of This Paper 12 Chapter2 14 Assignment Objectives 14 Deciding Whether Data Available or to Be Collected 15 to Use Content Kinds of Data Required 15 Analysis Kinds of Analysis Required 16 Resources Needed 17 Chapter 3 19 Defining the Variables 19 Planning a Selecting Material for Analysis 22 Content Analysis Defining the Recording Unit 24 Developing an Analysis Plan 26 Chapter4 30 Creating Codes 30 Coding the Coding Options 31 Textual Material Coder Selection and Training 32 The Potential for Coding Error 33 Selecting and Managing Documents 35 Applying Codes 36 Using a Computer to Code 39 Chapter 5 42 Preparing for Data Analysis 42 Analyzing the Estimating Reliability 42 Data Counting a Code's Frequency 43 Page4 GAO/PEMD-10.3.l Content Analysis Contents Finding Associations 43 Reporting the Methodology and Results 45 Chapter6 46 Planning 46 Avoiding Pitfalls Coding 48 Analyzing and Reporting the Data 50 Appendixes Appendix I: Analysis of Qualitative Data 52 Appendix II: Software for Content Analysis 54 Appendix III: Intercoder Reliability 64 Bibliography 68 Glossary 74 Papers in This 75 Series Tables Table 1.1: Seven Steps in Conducting a 12 Content Analysis Table 5.1: Frequency of Stars and Stripes 44 Stories on Selected Topics Table Il. l: Software Features Relevant to 55 Content Analysis Table III.l: Codes Applied by Two Coders 64 Table Ill.2: Observed Co-occurrences of 65 Codes Figures Figure 4.1: A Coded Passage 30 Figure 4.2: Overlapping Codes 37 Figure 4.3: Nested Codes 38 Abbreviations AID Agency for International Developm~nt DOD Department of Defense GAO General Accounting Office VA Department of Veterans' Affairs Page 5 GAO/PEMD-10.3.1 Content Analysis -- ------------- --- - -----------------~--~--~~~--~--~~~~~""---~~~~--- Chapter 1 What Content Analysis Is A Definition of In content analysis, evaluators classify the key ideas in a written communication, such as a report, article, Content Analysis or film. Evaluators can do content analysis of video, film, and other forms of recorded information, but in this paper, we focus on analyzing words. Here is a formal definition of content analysis: it is a systematic research method for analyzing textual information in a standardized way that allows evaluators to make inferences about that information. (Weber, 1990, pp. 9-12, and Krippendorff, 1980, pp. 21-27) Another expression of this is as follows: "A central idea in content analysis is that the many words of the text are classified into much fewer content categories." (Weber, 1990, p. 12) The classification process, called "coding," consists of marking text passages with short alphanumeric codes. This creates "categorical variables" that represent the original, verbal information and that can then be analyzed by standard statistical methods. The text passages can come from structured interviews, focus group discussions, case studies, open-ended questions on survey instruments, workpapers, agency documents, and previous evaluations. 1 Content analysis is particularly useful in GAO work because of the large quantity of written material that evaluators typically collect during a project, especially when it comes from diverse and unstructured sources. To classify a document's key ideas, the evaluator identifies its themes, issues, topics, and so on. The result might be a simple list of the topics in a series of meeting notes. Content analysis can go further if the evaluator counts the frequency of statements, detects subtle differences in their intensity, or examines issues over time, in different situations, or from different groups. 1See appendix I for a brief discussion of related forms of textual analysis. Babbie (1992) and Weber (1990) give an overview of the form of content analysis we discuss in this paper. Page6 GAO/PEMD-10.3.l Content Analysis Chapter 1 What Content Analysis Is Thus, content analysis can not only help summarize the formal content of written material; it can also describe the attitudes or perceptions of the author of that material. For example, if an evaluator wanted to assess the effects of a program on the lives of older people from their perspective, he or she could analyze open-ended interview responses to determine their outlook on life, loneliness, or security. Similarly, an evaluator could assess the effect of Voice of America broadcasts by analyzing the content of Soviet newspaper articles and radio broadcasts. (Inkeles, 1952) Here are several ways in which GAO evaluators 'have The Uses of successfully used content analysis techniques. Content Analysis 1. In Stars and Stripes: Inherent Conflicts Lead to Allegations of Military Censorship (GAO, 1988), GAO evaluators used content analysis to help assess issues of censorship, news management, and other influences on various editions of the military newspaper. Details of technique and substance from this report are used as examples throughout this · transfer paper. 2. In Student Loans: Direct Loans Could Save Billions in First 5 Years With Proper Implementation (GAO, 1992c), GAO evaluators examined transcripts of focus groups discussing the difficulty of implementing a student loan program. The participants' views on whether the Department of Education could administer a direct loan program were mixed, but the evaluators were able, through content analysis, to highlight the dominant views and the reasons for them. 3. Federal Employment: How Federal Employees View the Government As a Place to Work (GAO, Page 7 GAO/PEMD-10.3.1 Content Analysis Chapter 1 What Content Analysis Is 1992a) reported that while the majority of the survey respondents looked favorably on working for the government, many did not. The evaluators used content analysis to assess the respondent's insightful, written comments to open-ended questions. An appendix in the report is devoted to the analysis of these comments. 4. Another excellent example of the use of content analysis appears in Women in the Military: Deployment in the Persian Gulf War (GAO, 1993c). For this study, the evaluators gave a primarily positive assessment of women's performance, using content analysis to determine that while men and women endured similarly harsh encampment facilities and conditions, both men and women considered health and hygiene problems inconsequential and their cohesion in mixed-gender units effective. 5. Among other fine examples of the use of content analyis is Veterans' Health Care: Veterans' Perceptions of VA Services and VA's Role in Health Care Reform (GAO, 1994a). The report's scope and methodology section details the analysis and summary of veterans' views that changing the VA system could, among other things, diminish or eliminate their benefits as well as harm them both emotionally and in terms of their specialized health care needs. Other uses of content analysis in GAO reports include an analysis of transcripts of focus groups on people's ability to participate in food assistance programs (GAO, 1990), an analysis of descriptive text on the maintenance of aging aircraft (GAO, 1993a), and an analysis of open-ended discussions with Amerasian immigrants on their experiences in Vietnam, the Philippines, and the United States (GAO, 1994b). Page 8 GAO/PEMD-10.3.1 Content Analysis Chapter I What Content Analysis Is The increasing availability of written information on Computerized computer files, and the increasing number of Content Analysis computer programs to analyze text files, makes content analysis easier