The Microscope and the Moving Target: the Challenge of Applying Content Analysis to the World Wide Web

THE MICROSCOPE AND THE MOVING TARGET: THE CHALLENGE OF APPLYING CONTENT ANALYSIS TO THE WORLD WIDE WEB

By Sally }. McMillan

Analysis of nineteen studies t}tat apply content analysis technicfues to the World Wide Web found that this stable research technique can be applied to a dynmnic environment. However, the rapid growth and change of Web-based content present some unique challenges. Nevertheless, researchers are now using content analysis to examine themes such as diversity, commercialization, and utilization of technology on the World Wide Web. Suggestions are offered for how researchers can apply content analysis to the Web with primary focus on formulating research quest ions/hypotheses, sampling, data collection and coding, training/ reliability of coders, and analyzing/interpreting data.

Content analysis has been used for decadesas a microscope that brings communication messages into focus. The World Wide Web has grown rapidly since the technology that made it possible was introduced in 1991 and the first Web browsers became available in 1993.' The Commerce Depart- ment estimates that one billion users may be online by 2005, and the growth in users has mirrored growth in content.' More than 320 million home pages can be accessed on the Web^ and some Web sites are updated almost constantly.* This growth and change makes the Web a moving target for communication research. Caji the microscope of content analysis be applied to this moving target? The purpose of this article is to examine ways that researchers have begun to apply content analysis to the World Wide Web. How are they adapting the principles of content analysis to this evolving form of computer- mediated communication? What are the unique challenges of content analysis in this environment? This review of pioneering work in Web-based content analysis may provide researchers with insights into ways to adapt a stable research technique to a dynamic communication environment.

Krippendorff found empirical inquiry into communication content Literature dates at least to the late 1600s when newspapers were examined by the Review Church because of its concern of the spread of nonreligious matters. The first well-documented case of quantitative analysis of content occurred in eigh- teenth-century Sweden. That study also involved conflict between the church and scholars. With the advent of popular newspaper publishing at the turn of the twentieth century, content analysis came to be more widely

I&MC Qiiarierlii Sfl//i/ /. McMillan is an assistant professor in the Advertising Department at the Val77,No.l University ofTennessee-Knoxvitk. The author acknowledges M. Mark Miller'6 valuable Sprin^lOOO input oil ail earlier draft of this manuscript and Emmanuela Marcoglou's research 80-98 assistance.

80 louRfJMJSM & MASS CoMMUNioinofJ QUAXTEKLI used.^ it was witb the work of Berelson and Lazarsfeld'' in the 1940s and Berelson'' in the 1950s that content analysis came to be a widely recognized research tool that was used in a variety of research disciplines. Berelson's definition of content analysis as "a research technique for the objective, systematic and quantitative description of the manifest content of communication"** underlies much content analysis work. Budd, Thorp, and Donohew built on Berelson's definition defining content analysis as "a systematic technique for analyzing message content and message han- dling."^ Krippendorff defined content analysis as "a research technique for making replicable and valid inferences from data to their context.""' Krippendorff identified four primary advantages of content analysis: it is unobtrusive, it accepts unstructured material, it is context sensitive and thereby able to process symbolic forms, and it can cope with large volumes of data." All of these advantages seem to apply equally to the Web as to media such as newspapers and television. The capability of coping with large volumes of data is a distinct advantage in terms of Web analysis. Holsti identified three primary purposes for content analysis: to describe the characteristics of communication, to make inferences as to the antecedents of communication, and to make inferences as to the effects of communication.'^ Both descriptive and inferential research focused on Web-based content could add value to our understanding of this evolving communication environment. How-to guides for conducting content analysis suggest five primary steps that are involved in the process of conducting content analysis research.'•* Those steps are briefly enumerated below with consideration for application to the Web. First, the investigator formulates a research question and/or hypotheses. The advent of the Web has opened the door for many new research questions. The challenge for researchers who apply content analysis to the Web is not to identify questions, but rather to narrow those questions and seek to find a context for them either in existing or emerging communication theory. The temptation for researchers who are examining a "new" form of communication is to simply describe the content rather than to place it in the context of theory and/or to test hypotheses. Second, the researcher selects a sample. Thesizeofthe sample depends on factors such as the goals of the study. Multiple methods can be used for drawing the sample. But Krippendorff noted that a sampling plan must "assure that, within the constraints imposed by available knowledge about the phenomena, each unit has the same chance of being represented in tbe collection of sampling units."'"' This requirement for rigor in drawing a sample may be one of the most difficult aspects of content analysis on the Web. Bates and Lu pointed out tbat "with the number of available Web sites growing explosively, and available directories always incomplete and over- lapping, selecting a true random sample may be next to impossible."'-'' Not only do new Web sites come online frequently, but old ones are also removed or changed, McMillan found that in one year 15.4 percent of a sample of health-related Web sites had ceased to exist.'" Koehler found that 25.3 percent of a randomly selected group of Web sites had gone off-line in one year. However, be also found that among those sites that remained functioning after one year, the average size of the Web site (in bytes) more than doubled,'' The third step in the content analysis process is defining categories. Budd, Thorp, and Donohew identified two primary units of measurements:

THR MICRO'COPFANDTHE MOVING TARGFJ coding units and context units. Coding units are the smallest segment of content counted and scored in the content analysis. The context unit is the bodyofmaterialsurrounding thecodingunit. Forexample, if thecodingunit is a word, the context unit might be the sentence in which the word appears or the paragraph or the entire article. Many researchers use the term "unit of analysis" to refer to tbe context from which coding units are drawn."^ Defining the context unit/unitof analysisisa unique challenge on the World Wide Web. Ha and James argued that the home page is an icfeai unit of analysis because many visitors to a Web site decide whether they will continue to browse the site on tbe basis of their first impression of the home page. They also argued tbat coding an entire site could be extremely time- consuming and introduce biases based on Web-site size.'*^' Web Techniques estimated tbat Web sites range from one page to 50,000 pages.'" Other researchers have attempted to take a more comprehensive approach to defining the unit of analysis. For example, in a study of library Web sites Clyde coded all pages mounted by a library on its server.^' Fourth, coders are trained, they code the content,and reliability of their coding is checked. Each step of this coding process is impacted by Weh technology. The Web can facilitate training and coding. For example, McMillan trained twocoders who were living in different parts of the country to assist in coding Web sites. Instructions and coding sheets were delivered by e-mail and coders were given URLs for sites to be coded. The coding process was carried out remotely and without paper,^^ Wassmutb and Thompson noted potential problems in checking intercoder reliability for changing content. They controlled for possible changes in content by having sites that were being coded by multiple coders viewed at the same day and time by all coders.-^ Massey and Levy addressed the problem of changing contentby having all sites evaluated twicewith the second visitmade twenty- four hours after the initial visit to the site.^'' Fifth, the data collected during the coding process is analyzed and interpreted. The statistical tools used to analyze the data will depend on the type of data collected and on the questions that are addressed by the research. Similarly, the interpretation will grow out of the data analysis as well as the researcher's perspective. Tbe fact that the content that has been analyzed is Web-based is not expected to change the basic procedures of analysis and interpretation used in content analysis.

Methodology The author took several steps to identify both published and unpub- lished studies that have applied content analysis to the Web. First, a search of the electronic Social Sciences Citation Index was conducted searching for key words "Web" and "content analysis." Tbe SSCI was also searched for "Internet" and "content analysis." Selected communication journals were also reviewed for any articles that applied content analysis to the Web (see Appendix ]). Tbe author also sought out papers from communication conferences (e.g.. Association for Education injournalism and MassCommu- nication and Internationa! Communication Association) that applied content analysis to the Web. Finally, bibliographies were checked for all studies that were identified using the above techniques. Any cited study tbat seemed to apply content analysis to the Web was examined. Eleven studies were identified that applied content analysis to computer-mediated communication technologies other than the Web. Three studies analyzed content of Hstservs designed for academicians and practi- o2 JOURNALISM & MASS CuMMUNicAriDN tioners of public relations^^ and advertising.-^* Two studies analyzed newsgroups tbat served as online support/information groups on the topics of abortion^'' and eating disorders,^** Two studies analyzed pornographic materials found on Usenet.^^ Two studies focused on e-mail and electronic bulletin board postings related to public affairs: Newhagen, Cordes, and Levy examined e-mail sent to NBC Nightly News^'^ and Ogan examined content of an electronic bulletin board during the Gulf War.-'' The final two studies focused on organizational communication issues with analysis of e- mail messages related to an online training program^^ and an Internet discussion group established in conjunction with a conference.^'' In addition to the studies that examined computer-mediated communication forms such as e-mail and listservs, nineteen studies were found that focused specifically on content analysis of the World Wide Web, These nineteen studies are listed in Appendix 2 and examined in tbe Findings section of this article. Tbe five steps in content analysis identified in the literature are the basis for examination of how these nineteen studies applied tbe microscope of content analysis to the moving target of tbe World Wide Web.

Tbe first step in content analysis is to formulate research questions and /or hypotheses. Table 1 summarizes the primary purpose and questions addressed by each of the nineteen studies tbat applied content analysis to the World Wide Web. The majority of these studies were descriptive in nature. However, a few of these studies do seem to be moving more toward hypothesis testing and theory development. For example, McMillan pro- posed four models of funding for Web content based in part on level of interactivity that was determined by analysis of Web site features.-*^ Wassmu th and Thompson suggested that their study might be a first step in developing a theory that explains relationships between specificity of banner ad messages and the goal of information tasks,-^'' The second step in conducting a content analysis is to select a sample. Table 2 summarizes information about samples for the nineteen studies that analyzed content of Web sites. The sampling frame from which sites were drawn varied widely. One common way of defining a sampling frame was to use an online list of sites in a given category. A second popular technique was to use search engine(s) to identify sites that met criteria related to the purpose of the study. Two studies used off-line sources to identify Web sites^'' and two studies combined online and off-line listings of Web sites,^' Finally, one site analyzed three newspaper Web sites with no discussion of the sampling frame.-*** After definingfhe sampling frame, researchers typically draw a sample for analysis. Nine of tbe nineteen studies did not sample; instead they analyzed all sites in the sampling frame. The goals of these studies were typically to describe and/or set a benchmark for analysis of a given type of Web sites. Among studies that used sampling, use of a table of random numbers was tbe most commonsamplijig technique, McMillan pointed outchallenges of applying a table of random numbers to an online listing or search engine results. Most lists generated in this way are not numbered leaving tbe researcher a tedious hand-counting task. Search engine listings may be presented in multiple hypertext sub-menus that further complicate the task of assigning random numbers.^" However, few of the studies cited here provided much, if any discussion, of how to address sampling difficulties

THE MiCFOSCOPt: AND THE MOVING TARGET 83 TABLE 1 Study Overview Primary Puq^ose and Questions Addressed By Study Aikat To analyze academic, government, and commercial Web sites to determine their information content. Alloroet al. Compare digital journals to printed journals in order to understand if the traditional paper journal is becoming obsolete. Bar-Iian Examine what the authors of Web pages say about mathematician Paul Erdos. Bates and Lu To develop a preliminary profile of some personal home pages and thus provide an initial characterization of this new social form. Clyde To identify purposes for which a library might create a home page on the Web and of the information that might be provided. Elliott Classify sites on the Internet as family life education and compile an extensive list of sites that could be used as a reference. Esrock and Leichty Examine how corporations are making use of the Web to present themselves as socially responsible citizens and to advance their own policy positions. Frazer and McMillan Examine the structure, functional interactivity, commercial goals, marketing communication analogies, and types of businesses that have established a commercial presence on the Web. Gibson and Ward Addresses the question of whether the Internet is affecting the domi- nance of the two major parties in U.K. politics. Ha and James Explore the nature of interactivity in cyberspace by examining a sample of business Web sites. Ho Propose a framework to evaluate Web sites from a customer's perspective of value-added products and services. U Examine how Internet newspapers demonstrated a change from the convention of newspaper publishing to the new media age. Liu et al. Identify ways in which large U.S. firms have used their Web sites and home pages to conduct business. Massey and Levy Investigate online journalism's practical state through a more cohesive conception of interactivity and to contribute toward building a geographically broad perspective of how journalism is being done on the Net. McMillan Examine relationships between interactivity, perceptions of property values, audience size, and funding source of Web sites. McMillan and Campbell Evaluate current status of networked communication as a tool for building a forum for community-based participatory democracy. Peng, Tham, and Xiaoming Explore current trends in Web newspaper publishing by looking at various aspects of such operations as advertising, readership, content, and services, Tannery and Wessel To demonstrate bow medical libraries utilize the Web to develop digital libraries for access to information. Wassmuth and Thompson To examine banner ads in online editions of daily, general interest newspapers in the United States to determine if the banner ads are specific, general, or ambiguous.

that arise from using online sources to identify the sampling frame. The sample size for these nineteen studies varied dramatically from three to 2,865. The majority of the studies (fourteen) analyzed between fifty and 500 sites. The third step in content analysis is defining categories. Table 3 summarizes the categories and other details of data collection. In content analysis of traditional media, one of the first steps is to define the time period of the study (e.g., a constructed week of newspaper issues). But, in analysis

84 JfjuKNAUSM Er MASS CoMMumymoN QuARTEOy TABLE 2 Sampling Sampling Frame Sample Selection Method Sample Size

Aikat Master list of 12,687 Random selection of 1,140 WWW addresses about 10 percent of sites Alloroet a!, Online biomedical journals All found using multiple 54 search tools Bar-Ilan 6,681 documents retrieved Examined all that made 2,865 using seven search engines reference to Paul Erdos Bates and Lu The People Page Directory The first five entries under 114 http;//wvi'w.peoplepage.com each letter Clyde 413 library Web sites from Randomly selected 50 multiple sources Elliott AU World Wide Web sites All sites found using and 356 that met four criteria links search engines Esrock and Leichty Fortune 500 list of companies Every fifth site after 90 on the Pathfinder Web site. a randomstart Frazer and McMillan Web sites mentioned in first AU sites mentioned that 156 year of Web coverage in were still functioning Advertising Age at time of study Gibson and Ward U.K. political party Web sites All listed in one or more 28 of four Indexes Ha and James Web Digest for Marketers Census of all business Web 110 (online) sites during given time Ho Representative sites found Stratified random sample 1,400 using Yahoo and AltaVista of business categories Li Online editions of three Arbitrary 3 major newspapers Liu et al. Fortune 500 Companies All that could be found using 322 multiple search tools Massey and Levy Daily, general-circulation, All that could be found using 44 English-language news- multiple search tools papers in Asia that publish companion Web editions. McMillan Yahoo listing of health-related Random-number table 395 Web sites selection of sites McMillan and Campbell City.net list of Web sites Random-number table 500 selection of sites Peng, Tham, and Xiaoming Web newspapers published Stratified sample of national 80 in the United States newspapers, metropolitan dailies, and lwal dailies Tannery and Wessel Association of Academic All medical libraries listed 104 Health Science Libraries in online list of libraries Wassmuth and Thompson Editor and Publisher listing Every ninth site after a 75 of online U.S. newspapers random start of Web sites, study authors placed emphasis on the time frame (e.g., April 1997) of the study. As noted earlier, changes in the content of Web sites necessitates rapid collection of data. The most rapid data collection reported was two days and the longest was five months. Most studies collected data in one to two months. After identifying the time frame of the study, researchers need to identify context units for coding. The most common context unit used for

THE MJCRoxore. AND THI. MOVINC. TAR TABLE 3 Data Coltectiou and Coding Time Frame Context Unit Coding Unit

Aikat A two-week period Web site Content categories: news, commentary, public relations, advertising, bulletin board, service/product support information, entertainment, exhibits/picture archive, data bank/general information, miscellaneous/ other. Alloroet al. April 1997 Web site Content categories: aim/scope, author instructions, table of contents, abstracts or tables, full text, searchable index. Bar-Ilan 21 Nov. 199(1- Web site Content categories; mathematical work, 7 Jan.1997 Erdos number, in honor/memory, jokes/ quotations, math education. Bates and Lu June 1996 Home Home page internal structure and physical page features. Content categories: information about the site as a whole, purposes of home page, personal information content of pages, misc. content/elements/features. Clyde ]6-17Sept. 1996 All pages A list of 57 features. Country of Web site on server origin. Education level of Web site. Content categories: human development Elliott January Web site and sexuality; interpersonal relationships; to April 1997 family interaction; family resource management; education about parenthood, family, and society. Esrock November 1997 Web site Messages relating to 13 social responsibility and Leichty contentareas.Otherattributes: annual sales, industry sector, ranking on the Fortune list, and rank within industry. Interactive features (e.g., e-mail links). Potential use of the Web sites for agenda-setting activity. Frazer 30 hours Web site Marketing type (Hoffman/Novak typol- and McMillan in one weekend ogy), business type (e.g., entertainment, in April 1996 media), content features (e.g., text, graphics), interactive fimctions (e.g., online or- dering), selling message (price, brand, none). Gibson March Web site E-mail links to: party headquarters, mem- and Ward and April 1997 bers of parliament, regional/local parties. Also coded feedback requested as: policy- based, non-substantive. Web site design features: graphics, split screen, flashing/ moving icon, links to other sites, frequency of updating, web design, targeted youth pages. Ha and James October 1995 Home page Nature of business, dimensions of inter- to January 1996 activity, information collection. Ho May through Portions Business purpose, types of value creation. September 1996 of Web Site U 9-18 Sept, 1996 Articles Pictures, graphs, news articles, news links. on home page Table 3 cont. next page

86 &• MASS CoMMUMCAni Table 3 cont. Time Frame Context Unit Coding Unit Liu et a). 20 Sept. 1995- Web site Content categories: description, ov'en-'iew, 1 Nov. 1995, feedback, "what's new," financial, cus- updated )uly 1996 tomer service, search, employment, guest book, index/directory, online business, other sites, CEO message, FAQ. Massey 23 March - Web site Content features: complexity of choice avail- and Levy 10 April 1998 able, responsiveness to the user, ease of adding information, facilitation of interpersonal interaction McMillan January to Web site Number ofhot links, banner advertising, hit February 1997 counter, indicates last update, organizational tools (e.g., search engine, site map), interactive features (e.g., bulletin boards, chat rooms). McMillan Spring 1996 Web site Content categories: contact information, and Campbell city functions, time/place of public meet- ings, agendas/ minutes of public meet- ings, rules and regulations, budget/fi- nances, e-mail contact, voting information, "hot topics," community "forum." Peng, Tham, January 1997 Web site Not explicitly reported in the study. and Xiaoming Tannery January -March 1997 Web site Level I: basic information such as hours, and Wessel l(x:ation and policy. Level II: adds interactive services such as databases and Internet resources. Level III: adds unique information such as original publishing, online tutorials, and digital applications. Wassmuth One week Banner ads Banner content (specific, general, ambigu- and Thompson in September 1998 found in ous), banner on home page, banner in clas- search task sified, "trick" banner, IPT tools {e.g., Java}, animation. these studies was the "Web site." Many of the studies did not specify what was meant by the Web site. However some did limit analysis to the "home page" or initial screen seen upon entering the site while others specified examination of all pages that could be found at a given site and others indicated that they had searched sites in "sufficient detail" to identify key features. Given the magnitude and changing nature of sites, some creativity would seem to be needed in defining the context unit. Wassmuth and Thompson devised one creative approach in their analysis of banner ads that were encountered when the coder performed a specific task at a newspaper Web site (find a classified ad for a BMW).'"' Another approach was to have coders analyze all sites for a specific length of time—about ten minutes.^' This helps to reduce bias that might be involved in coding sites of varying lengths. However, it might introduce "error" in that different coders may choose to examine different parts of a Web site during the ten-minute examination period. These nineteen studies varied widely in terms of coding units used by the researchers. The most common coding unit was "content categories." However, no stanciard list of categories emerges from these studies. Nor do they rely on established lists such as the fifty categories identified by Bush." Instead, content categories seem to be specifically related to the goals of the

THI: MICROSCOPE AND THF MOVING TARGFT given study. For example, Bar-Ilan developed content categories related to a specific historical figure,''^ and McMillan and Campbell developed content categories related to tbe community-building features of city-based Web sites.'" Another common coding unit is structural features of the Web site (e.g., links, animation, video, sound, etc.). Two studies used Heeter's''^ conceptual definition of interactivity as a frame for organizing analysis of Web site features.""" One study conducted an in-depth analysis of e-mail links.'''' Many of the other sites also coded e-mail links as a structural element. Tannery and Wessel developed a three-level categorization system for evaluating overall sophistication of Web sites based on both content categories and structural features.'"* Some studies also reported on "demo- graphic" characteristics of sites such as country of origin and type of institution that created the site while others explored the nature and/or purpose of the sponsoring organization in more detail. The Wassmuth and Thompson study is unique in itsanalysisof banner advertisements in Web sites.'*'' The fourth step in content analysis is training coders and checking the reliability of their coding skills. One of the primary reasons for this attention to training and checking coders is that, as Budd, Thorp, and Donohew noted, an important requirement for any social-science based research is tbat it be carried out in such a way that its results can be verified by other investigators who follow the same steps as the original researcher."^' Krippendorff wrote that at least two coders must be used in content analysis to determine reliability of the coding scheme by independently analyzing content.^' Table 4 summarizes information about bow many coders were used for each study, how they were trained, how mucb of the data was cross-coded (independently coded by two or more individuals), and the method used for calculating reliability of the coding. Eight of the nineteen studies did not report any information about coders. Of those that did report on coders, thenumber of coders used ranged from two to twelve with an average of about five coders. Only seven studies reported on training of coders and most said little about how training was done. Eleven studies reported cross-coding techniques. Most cross-coded a percentage {10 to 20%) of all sampled sites. Elliott had all sites cross-checked by two coders^^ while Li had two coders cross-check all the data collected in the first three days of a ten-day sample."^^ Ha and James used a pre-test/post- test design that enabled them to correct coding problems prior to the start of data collection as well as to check reliability of collected data.^ Ten studies reported reliability of the coding. Formulas used for testing reliability included Holsti's reliability formula, Perreault and Leigh's reliability index. Spearman-Brown Prophesy Formula, and Scott's Pi. Bar-Ilan did not indicate what type of test was used to generate a 92% reliability score for her study.^-"^ Reliability scores ranged from .62 to 1.00. The fifth stage of content analysis is to analyze and interpret data. Table 5 summarizes key findings. As noted earlier, the purpose of most of these studies was descriptive in nature. Therefore, it is not surprising that key findings are also descriptive in nature. One theme that emerged from several of these studies was the diversity found at Web sites. For example. Bates and Lu found that the "personal home page" is an evolving communication form,^ Clyde found variety in the structure and function of library Web sites,'^^and McMillan concluded TABLE 4 Coders and Coding How Many How Trained Cross-Coding Reliability

Aikat 2 Coded sites not 16 percent coded .98 using HolsH's in the sample by both coders reliability formula AUoro et al. Not stated Not stated Not stated Not stated Bar-Ilan 2 Not stated 287 sites (10%) 92% Bates and Lu Not stated Not stated Not stated Not stated Clyde Not stated Not stated Not stated Not stated Elliott 9 Trained to use All sites were cross- Not stated search engines checked by two and recognize coders search criteria Esrockand Leichty Not stated Not stated 20% of all sites Scott's Pi index ranging from .62 to 1.0 Frazer 7 Trained in coding lQTo of all sites were .88 using Holsti's and McMillan procedures and coded by two coders intercoder variables reliability formula Gibson and Ward Not stated Not stated Not stated Not stated Ha and James 7 Trained in using the All coders coded From .64 to .96 us- coding instrument. two pre-test and ing Perreauit and three post-test sites Leigh's reliability index Ho Not stated Not stated Not stated Not stated U 2 Not stated All data collected Spearman - Brown in first three days Prophecy For- of ten-day sample mula Standard- ized item score of .92 Liu et al. Not stated Not stated Not stated Not stated Massey and Levy 12 Not stated 11% of all sites .80 to 1-0 using Holsti's Inter- coder reSiability formula McMillan 3 Trained using non- 10% of all sites .92 using Holsti's sampled sites Intercoder reliability formula McMillan 2 Trained using non- 10% of all sites .97 using Holsti's and Campbell sampled sites Intercoder reliability formula Peng, Tham, Not stated Not stated Not stated Not stated and Xiaoming Tannery Not stated Not stated Not stated Not stated and Wessel Wassmuth 2 Trained in mean- About 107o Rangingfrom.70to and Thompson ing of primary coded cross-coded; 1.0 using Scott's Pi variable tallied by indepen- for each variable dent researcher that the Web has diversity of content, funding sources, and communication models.''" A second theme found in these studies is commerciali7,ation of the Web. Some studies view commercialization as positive for marketers and/ or consumers while others express concern about impacts of commercializa-

TffE MtcsascoPi. AND THF. MOVING T.iRGtT 89 TABLE 5 Summary of Findings Primary Findings of the Study

Aikat Public relations and advertising were the dominant information categories for the three types of WWW sites studied. Alloro The Internet seems useful only for publishers seeking to market their products or for et al. authors needing information for submission of papers. The replacement of printed journals by their electronic counterparts in the field of biomedicine does not seem possible in the short term. Bar-llan At this time, the Web cannot be considered a good quality data base for learning about Paul Erdos, but it may serve as an adequate reference tool once the noise introduced by duplicates, structural links, and superficial information is removed. Bates The home pages reviewed displayed a great variety of content and of specific types of and Lu formatting within broader formatting approaches. The home page as a social institution is still under development. Clyde Although there is some agreement about some of the features that are necessary on a library's home page there is nevertheless great diversity, too, and not all library Web pages contained even basic information to identify the library. Elliott Of the key concepts analyzed in this study, \7% were covered by only one site, and 18% were not covered at all. There are many concepts that family life educators could focus on in on-line education. Esrock 82'^i of the Web sites addressed at least one corporate social responsibility issue. More and Leichty than half of the Web sites had items addressing community involvement, environmental concerns, and education. Few corporations used their Web pages to monitor public opinion. The number of social responsibility messages was positively correlated with the size of an organization. Frailer and Thus far, itdoesnotseem that many marketers are taking full advantageoftheinteractive McMillan capabilities of the World Wide Web. Gibson The Internet allows minor parties in the United Kingdom to mount more of a challenge and Ward to their major counterparts than in other media. Ha and The generally low use of interactive devices reveals a discrepancy between the interactive James capability of the medium and the actual implementation of interactivity in a business setting. Ho Sites are primarily promotional and the marketing approach taken is conventional: product news, catalogs and portfolios, previews and samples, special offers and discounts, contests and sweepstakes. Liu et al. Web sites can be used to support pre-sales, sales, and after-sales activities. Companies that have higher market performances will more likely use Web sites to reach their customers. Using Web sites to reach potential customers is not limited to certain industries. Massey A fuller portrait of online journalism can be developed by applying a more unified and Levy conception of interactivity to news-making on the Web. McMillan The computer-mediated communication environment is currently robust enough for diversity of content, funding sources, and communication models to co-exist. McMillan It is possible to use the Web to implement activities that have been designed to increase and citizen participation in public life; it is also possible to use this technology to further the Campbell commercial interests that define consumption communities. Peng, Tham, New technologies can bring new opportunities as well as threats to existing media, and Xiaoming Tannery Most libraries are at Level U (basic information plus interactive resources), but the future and Wessel lies with Level III sites that use digital technology as a distinct medium with online tutorials, locally created databases, and digital archives. Wassmuth As a user navigates deeper into the site and closer to the goal of the information location and task, banner ads become progressively more specific. Thompson

90 tion on public life and/or academic expression. As a counterpoint to these concerns, Gibson and Ward found the Web offered a forum for minor political parties to have a voice.^'' A third major theme was the fact that many site developers are not using the Web to its full potential as a multi-media interactive environment. Researchers found many sites made limited use of interactivity, graphicsand motion video, and search functions. And Elliott's study suggested that there is still significant room for development of new content in important areas such as family life education.''"

The studies reviewed above found that the stable research technique of Recommend- content analysis can be applied in the dynamic communication environment ations of the Web. But using content analysis in this environment does raise for Future potential problems for the researcher. Some of these studies exhibit creative ways of addressing these problems. Unfortunately, others seem to have Researchers failed to build rigor into their research designs in their haste to analyze a new medium. Future studies that apply content analysis techniques to the Web need to consider carefully each of the primary research steps identified in the literature: formulating research questions and/or hypotheses, sampling, data collection and coding, training coders and checking the reliability of their work, and analyzing and interpreting data. For the first step, formulating the research questions and/or hypotheses, content analysis of the Web is both similar to and different from traditional media. Content analysis of traditional media, such as newspapers and broadcast, assume some linearity or at least commonly accepted se- quencing of messages. Hypertext, a defining characteristic of the Web, defies this assumption. Each individual may interact with content of a Website in different ways. Furthermore, the Web is both "like" and "unlike" print and broadcast as it combines text, audio, still images, animation, and video. These unique characteristics of the medium may suggest unique research questions. However, in some fundamental ways, the first step in the research process remains similar. Researchers should build on earlier theoretical and empirical work in defining their Web-based research. Approaches taken by the authors cited above tend to be consistent with Holsti's first purpose of content analysis: describing the characteristics of communication. This is appropriate forearly studies ofan emerging medium. However, researchers also need to move on to the other two purposes identified by Holsti: making inferences as to antecedents of communication and making inferences as to the effects of communication.''' The second step in content analysis research, sampling, presents some unique challenges for Web-based content analysis. As noted earlier, a key concern in sampling is that each unit must have the same chance as all other units of being represented. The first challenge for the researcher is to identify the imits to be sampled. This will be driven by the research question. For example, if the researcher wants to examine a sample of Web sites for Fortune 500 companies, it may be fairly simple to obtain a list of these sites and apply a traditional sampling method (e.g., a table of random numbers, every nth item on the list, etc.) But, if one seeks a different kind of sample (e.g., all business Web sites) the task may become more difficult. Essentially, the researcher has two primary sources from which to develop a sampling frame: offline sources and online sources.

THE MICROSCOPE AND THE MOVING TARC 91 If the researcher draws a sample from offline sources (e.g., directories, lists maintained by industry groups, etc.), then drawing a random sample is fairly nonproblematic. Because the sample universe exists in a "set" medium, traditional methods of sampling can be used. However, a major problem with offline sources is that they are almost certainly out of date. Given the rapid growth and change of the Web, new sites have probably been added since the printed list was created while others have moved or been removed. Online sources can be updated more frequently, and thus help to eliminate some oftheproblemsassociated with offline sources. And, in many cases (e.g., directories, lists maintained by industry groups, etc.), the list can appear in the same kind of format that is found in an offline source making sampling fairly straightforward. But, human list-compilers are less efficient than search engines for identifying all potential Web sites that meet specific criteria. And in some cases no compiled list may exist that directly reflects the criteria of a study. In these cases, search engines may be the best way of generating a sample frame. However, care must be taken in using search engine results. First, authors must make sure that they have defined appropriate key words and that they understand search techniques for the search engine that they are using. If they do not, they run the risk of either over registration or under registration. Human intervention may be needed to insure that the computer-generated list does not include any spurious and/ or duplicate items. Once a list has been generated, the researcher needs to determine the best way to draw a random sample. One way to ease the problem of random assignment may be to generate a hardcopy of the list. Then the researcher can assign random numbers to the list and/or more easily select every nth item. However, if a hierarchical search engine (such as Yahoo!) is used, special care must be taken to ensure that all items in sub- categories have equal chance of being represented in the sample. In some cases, stratified sampling may be required. Research needs to be done to test the validity of multiple sampling methods. For example, does the use of different search engines result in empirically different findings? What sample size is adequate? How can sampling techniques from traditional media (e.g., selection of "representative" newspapers or broadcast stations) be applied to the Web? A key concern is that sampling methods for the Web not be held to either higher or lower standards than have been accepted for traditional media. The rapid growth and change of the Web also leads to potential problems in tbe third stage of content analysis: data collection and coding. The fast-paced Web almost demands that data be collected in a short time frame so that all coders are analyzing the same content. Koehler has suggested some tools that researchers can use to download Web sites to capture a "snapshot" of content.''- However, as Clyde noted, copyright laws of some countries prohibit, among other things, the storage of copyrighted text in a database system.''-' Whether or not the site was downloaded prior to analysis, researchers must specify when they examined a site. Just as newspaper-based research must indicate the date of publication, and broadcast-based research must indicate which newscast is analyzed, so Web-based content analysis must specify tbe timeframe of the analysis. For sites that change rapidly, exact timing may become important. Researchers must also take care in defining units of analysis. The coding unit can be expected to vary depending on the theory upon which the study is based, the research questions explored, and the hypotheses tested.

92 louKMAijUM & MASS QiMMiiMomoN QuAfnr.Kiy However, some standardization is needed for context units. For example, analyses of traditional media have developed traditional context units (e.g., the column-inch and/or a word count for newspapers, time measured in seconds for broadcast), but no such standard seems to have yet emerged for the Web. The fact that multiple media forms are combined on the Web may be one of the reasons for this lack of a clear unit of measurement. The majority of the studies reviewed above simply defined the context unit as the "Web site." Future studies .should specify how much of the Web site was reviewed (e.g., "home page" only, first three levels in the site hierarchy, etc.). But as analysis of the Web matures, entirely new context units may need to be developed to address phenomena that are completely nonexistent in traditional media. For example, Wassmuth and Thompson measured the ways in which the content of the Web was "adapted" in real-time to become more responsive to site visitors' needs.*^ The fourth step in content analysis, training coders and checking the reliability of their work, involves both old and new challenges on the Web. Given the evolving nature of data coding described above, training may involve "teaming together" with coders how to develop appropriate context and coding units. But the rapid change that characterizes the Web may introduce some new problems in checking intercoder reliability.'''' This does not mean that well-developed tools for checking reliability (e.g., Holsti's index, Scott's Pi, etc.) should be abandoned. Rather, the primary challenge is to make sure that coders are actually cross-coding identical data. If Web sites are checked at different times by different coders and/or if the context unit is not clearly defined, false error could be introduced. The coders might not look at the same segment of the site, or data coded by the first coder might he changed or removed before the second coder examines the site. Only one of the studies reviewed above directly addressed this issue. Wassmuth and Thompson carefully defined a task to be preformed at a site and had two coders perform that identical task at exactly the same date and time.** This is one viable alternative. However, if the content of the site is changing rapidly, this control may not be sufficient. Another alternative is to have coders evaluate sites that have been downloaded. These downloaded sites are "frozen in time" and will not change between the coding times. However, as noted earlier, there may be some legal problems with such downloaded sites. Furthermore, depending on the number of sites being examined, the time and disk-space requirements for downloading all of the studied sites may be prohibitive. The Web does not seem to pose any truly new challenges in the final step in content analysis: analyzing and interpreting the data. Rather, researchers must simply remember that rigor in analyzing and interpreting the findings are needed as much in this environment as in others. For example, some of the studies reported above used statistics that assume a random sample to analyze data sets that were not randomly generated. Findings and conclusions of such studies must be viewed with caution. Such inappropriate use of analytical tools would probably not be tolerated by reviewers had not the Web-based subject mater of the studies been perceived as "new" and "innovative." But new communication tools are not an excuse for ignoring established communication research techniques. In conclusion, the microscope of content analysis can be applied to the moving target of the Web. But researchers must use rigor and creativity to make sure that they don't lose focus before they take aim.

THF MKROSCOK AND THF MOVINC. TARCFT 93 APPENDIX 1

The following journals were searched for articles that reference content analysis of the World Wide Web. The search began with January 1994 issues and continued through all issues that were available as of early July 1999.

Coiiinninicatioii Research Human Coinmiinicatiati Research lournal of Advertisiii;^ fournal of Advertising Research Journal of Broadcasting & Electronic Media Journal of Communication Journal of Computer Mediated Communication Journal of Public Relations Research Journalism & Mass Communication Quarter!}/ Journalism &• Mass Communication Educator Newspaper Research Journal Public Relations Review

NOTES

1. Barry M. Leiner, Vinton G. Cerf, David D. Clark, Robert E. Kahn, Leonard Kleinrock, Daniel C. Lynch, Jon Postel, Larry G. Roberts, and Stephen Wolff, A Brief History of the Internet (Available online at: http:// www.isoc.org/intemet-history/brief.htmt). 2. Commerce Department, The Emerging Digital Economy (available online at: http://www.ecommerce.govdanintro.htm). 3. Internet Index (available online at: http://www.openmarket.com/ intindex/content.html). 4. Wallace Koehler, "An Analysis of Web Page and Web Site Constancy and Permanence," Journal of the American Society for Information Science 50 (February 1999): 162-80. 5. Klaus Krippendorff, Content Analysis: An Introduction to Its Method- ology (Beverly Hills, CA, 1980), 13-14. 6. Bernard Berelsonand Paul F. Lazarsield, The Analysis of Conimuuica- tion Content (Chicago and New York: University of Chicago and Columbia University, 1948). 7. Bernard Berelson, Content Analysis in Communication Research (New York: Free Press, 1952). 8. Bereison, Content Analysis, 18. 9. Richard W. Budd, Robert K. Thorp, and Lewis Donohew, Content Analysis of Communications (New York: The Macmillan Company, 1967), 2. 10. Krippendorff, Content Analysis, 21. lL Krippendorff, Content Analysis, 29-31. 12. Ole R. Holsti, Content Analysis for the Social Sciences and Humanities (Reading, MA: Addison-Wesley, 1969). 13. See for example Budd, Thorp, and Donohew, Content Analysis, and Krippendorff, Content Analysis. 14. Krippendorff, Content Analysis, 66. 15. Marcia J. Bates and Shaojun Lu, "An Exploratory Profile of Personal Home Pages: Content, Design, Metaphors," Online and CDROM Rei'iew 21 Qune 1997): 332.

94 loURNAUSM & M^SS COMMUNICAnLVJ Ql APPENDIX 2 Following is a list of the nineteen studies analyzed in this article:

Aikat, Debashis. 1995. Adventures in Cyberspace: Exploring the Information Content of the World Wide Web Pages on the Internet. Ph.D. diss.. University of Ohio. AUoro, G. C. Casilli, M. Taningher, and D. Ugolini. 1998. Electronic biomedical journals: How they appear and what they offer. European journal of Cancer 34, no. 3: 290-95. Bar-Ilan, Judit. 1998. The mathematician, Paul Erdos (1913-1996) in the eyes of the Internet. Scientometrics 43, no 2: 257-67. Bates, MarciaJ.and Shaojun Lu. 1997. An exploratory profileofpersonal homepages: Content, design, metaphors. Online and CDROM Rci>ieu> 21, no. 6: 331-40. Clyde, Lauren A. 1996. The library as information provider: The home page. The Electronic Library 14, no. 6: 549-58. Elliott, Mark. 1999. Classifyingfamily life education on the World Wide Web. Family Relations 48, no. 1: 7-13. Esrock, Stuart L. 1998. Social responsibility and corporate Web pages: Self-presentation or agenda-setting? Public Relations Review, 24, no. 3: 305-319. Frazer, Charles and McMillan, Sally J. (1999). Sophistication on the World Wide Web: Evaluating structure, function, and commercial goals of Web sites. ]n Advertising and the World Wide Web, ed. Esther Thorson and David Schumann, 119-34. Hillsdale, NJ: Lawrence Erlbaum. Gibson, Rachel K. and Stephen J. Ward. 1998. U.K. political parties and the Internet: "Politics as usual" in the new media? PressjPolitia 3, no. 3: 14-38. Ha, Louisa and E. Lincoln James. 1998. Interactivity Re-examined: A Baseline Analysis of Early Business Web Sites. Journal of Broadcasting & Electronic Media 42 (fall): 457-74. Ho, James. 1997. Evaluating the World Wide Web: A global study of commercial sites, journal of Computer Mediated Couimunication 3, no. 1: Available online at: http:/www.ascusc.org/jcmc/vol3/ issuel/ho.html. Li, Xigen. 1998. Web page design and graphic use of three U.S. newspapers. Journalism & Mass Communication Quarterly 75, no. 2: 353-65. Liu, Chang, Kirk P. Amett, Louis M.Capella, and Robert C.Beatty. 1997. Web sites of the Fortune 500 companies: Facing customers through home pages. Information & Management 31: 335-45. Massey, Brian L. and Mark R. Levy, 1999. Interactivity, online journalism, and English-language Web newspapers in Asia, journalism & Mass Communication Quarterly 76, no. 1: 138-51. McMillan, Sally J. 1998. Who pays for content? Funding in interactive media. Journal of Computer-Mediated Communication 4, no. 1: Available online at: http:/www.ascusc.org/icmc/vol4/ issuel/mcmillan.html. McMillan, Sally J. and Katherine B. Campbell. 1996. Online cities: Are they building a virtual public sphere or expanding consumption communities? Paper presented at the AEJMC annual conference, Anaheim, CA. Peng, Foo Yeuh, Naphtali Irene Tham, and Hao Xiaoming. 1999. Trends in online newspapers: A look at the US Web. Newspaper Research Journal 20, no. 2: 52-63. Tannery, Nancy and Charles B. Wessel. 1998. Academic medical center libraries on the Web. Bulletin of the Medical Library Association 86, no 4: 541-44. Wassmuth, Birgit and David R. Thompson. 1999. Banner ads in online newspapers: Are they specific? Paper presented at the American Academy of Advertising annual conference, Albuquerque, New Mexico.

16. Sally J. McMillan, "Virtual Community: The Blending of Mass and Interpersonal Communication at Health-Related Web Sites" (paper presented at the International Communication Association Annual Conference, Israel, 1998). 17. Wallace Koehler, "An Analysis of Web Page and Weh Site Constancy and Permanence, journal of the American Society for Information Science 50 (February 1999): 162-80.

THE MKROSCOPF- At'in TTJ/; MOVINC, TARcn 18. Budd, Thorp, and Donohew, Coulciit Anali/sis, 33-36. 19. Louisa Ha and E. Lincoln James, "InteracHvity Re-examined: A Baseline Analysis of Early Business Web Sites," Journal of Broaiicastiii^ & Electronic Media 42 (fall 1998): 457-74. 20. Web Techniques, "Web Statistics," Cyberatlas 2, no. 5 {available online at: http://www.cyberatlas.com. 21. Lauren A. Clyde, "The Library as Information Provider: The Home Page," The Electronic Library 14 (December 1996): 549-58. 22. Sally J. McMillan, "Who Pays for Content? Funding in Interactive Media," lournal of Computer-Mediated Communication 4 (September 1998): available online at: http://www.ascusc.org/jcmc/vol4/issuel/ mcmillan.html. 23. Birgit Wassmuth and David R. Thompson, "Banner Ads in Online Newspapers: Are They Specific?" (paper presented at the American Acad- emy of Advertising annual conference, Albuquerque, NM, 1999). 24 . Brian L. Massey and Mark R. Levy, "Interactivity, Online Journal- ism, and English-Language Web Newspapers in Asia," ]ournalisnt & Mrtss Communication Quarterly 7b (spring 1999): 143. 25. Bonita Dostal Neff, "Harmonizing Global Relations: A Speech Act Theory Analysis of PRForum," Public Relations Review 24 (3, 1998): 351-76; Steven R. Thomsen, "@ Work in Cyberspace: Exploring Practitioner Use of the PRForum," Public Relntion^ Revinv 22 (2, 1996): 115-31. 26. Louisa Ha, "Active Participation and Quiet Observation of Adforum Subscribers, journal of Adzvrtising Education 2 (fall 1997): 4-20. 27. Steven M. Schneider, "Creatinga Democratic Public Sphere through Political Discussion: A Case Study of Abortion Conversation on the Internet," Social Science Computer Review 14 (winter 1996): 373-93. 28. Andrew Winzelberg, "The Analysis of an Electronic Support Group for Individuals with Eating Disorders," Computers in Human Behavior 13 (summer 1997): 393-407. 29. Michael D. Mehta and Dwaine Plaza, "Content Analysis of Porno- graphic Images Available on the Internet," The Information Sfjdcfi/13 (spring 1997): 153-61; Marty Rimm, "Marketing Pornography on the Information Superhighway: A Survey of 917,410 Images, Descriptions, Short Stories, and Animations Downloaded 8.5 Million Times by Consumers in over 2000 Cities in Forty Countries, Provinces, and Territories," The Georgetown Law Journal 83 (May 1995): 1849-1934. 30. John E. Newhagen, John W. Cordes, and Mark. R. Levy, "[email protected]: Audience Scope and the Perception of Interactivity in Viewer Mail on the Internet," lournal of Communicafion 45 (summer 1995): 164-75. 31. Christine Ogan, "Listserver Communication during the Gulf War: What Kind of Medium is the Electronic Bulletin Board?" Journal of Broadcast- ing & Eh-ctronic Media 37 (spring 1993): 177-97. 32. Louis j. Kruger, Steve Cohen, David Marca, and Lucia Matthews, "Using the Internet to Extend Training in Team Problem Solving/' Behavior Research Methods, Instruments, & Computers 28 (spring 1996): 248-52. 33. R. Alan Hedley, "The Information Age: Apartheid, Cultural Impe- rialism, or Global Village?" Social Science Computer Review 17 (spring 1999): 78-87. 34. McMillan, "Who Pays for Content?" 35. Wassmuth and Thompson, "Banner Ads in Online Newspapers." 36. Charles Frazer and Sally J. McMillan, "Sophistication on the World

96 lauKNAiJSM & MASS CfiMMUNiCAnof^ QuARTFUiy Wide Web: Evaluating Structure, Function, and Commercial Goals of Web Sites," in Adz'ertising and the World VJide Web. ed. Esther Thorson and David Schumann (Hillsdale, NJ: Lawrence Erlbaum) 119-34; Chang Liu, Kirk P. Arnett, Louis M. Capelia, and Robert C. Beatty, "Web Sites of the Fortune 500 companies; Facing Customers through Home Pages," Information & Manage- ment 31 (1997): 335-45. 37. Clyde, "The Library as Information Provider"; Massey and Levy, "Interactivity, Online Journalism," 138-51. 38. Xigen Li, "Web Page Design and Graphic Use of Three Ll.S. News- papers," Journalism & Mass Communication Quarterly 75 (summer 1998): 353- 65. 39. McMillan, "Who Pays for Content?" 40. Wassmuth and Thompson, "Banner Ads in Online Newspapers." 41. See Frazer and McMillan, "Sophistication on the World Wide Web"; McMillan, "Who Pays for Content?"; Sally J. McMillan and Kathryn B. Campbell, "Online Cities; Are They Building a Virtual Public Sphere or Expanding Consumption Communities?" (paper presented at the AEJMC Annual Conference, Anaheim, CA, 1996). 42. Chilton R, Bush, "The Analysis of Political Campaign News," journalism Quarter}]/ 28 (spring 1951): 250-52. 43. Judit Bar-Ilan, "The Mathematician, Paul Erdos (1913-1996) in the Eyes of the Internet," Scicntomctrics 43 (March 1998); 257-67. 44. McMillan and Campbell, "Online Cities." 45. Carrie Heeter, "Implications of New Interactive Technologies for ConceptualizingCommunication,"inM('diflUsem//ipiM/t>rmflfiOMA;?('; Emerg- ing Patterns of Adoption and Consumer Use, ed, Jerry L. Saivaggio and Jennings Bryant (Hillsdale, NJ: Lawrence Erlbaum, 1989), 217-35. 46. Massey and Levy, "Interactivity, Online Journalism," and McMillan, "Who Pays for Content?" 47. Rachel K. Gibson and Stephen J. Ward, "U.K. Political Parties and the Internet; 'Politics as usual' in the new media?" Press/Politics 3 (fall 1998); 14- 38. 48. Nancy Tannery and Charles B. Wessel, "Academic Medical Center Libraries on the Web," Bulletin of the Medical Library Association 86 (October 1998): 541-44. 49. Wassmuth and Thompson, "Banner Ads in Online Newspapers." 50. Budd, Thorp, and Donohew, Content Analysis, 66. 51. Krippendorff, Content Analysis, 74 52. Mark Elliott, "Classifying Family Life Education on the World Wide Web," Family Relations 48 (January 1999): 7-13. 53. Li, "Web Page Design." 54. Ha and James, "Interactivity Re examined." 55. Bar-Man, "The Mathematician, Paul Erdos." 56. Bates and Lu, "An Exploratory Profile of Personal Home Pages." 57. Clyde, "The Library as Information Provider." 58. McMillan, "Who Pays for Content?" 59. Gibson and Ward, "U.K. Political Parties and tbe Internet," 60. Elliott, "Classifying Family Life Education," 61. Holsti, Content Analysis for the Social Sciences ami Humanities. 62. Koehler, "An Analysis of Web Page and Web Site Constancy." 63. Clyde, "The Library as Information Provider." 64. Wassmuth and Thompson, "Banner Ade. in Online Newspapers." 65. A separate issue, not directly addressed in this .study, is the use of

THE MtcRoscopE AND THE MOVING TARGET 97 computer-based content analysis for examining the digital content of Web sites. None of the studies examined in this article utilized this technique. With computeri;2ed analysis of content, reliability is assured—the computer will obtain the same results every time the data is analyzed. With this form of content analysis, emphasis must be placed on validity rather fhan on reliability, 66. Wassmuth and Thompson, "Banner Ads in Online Newspapers."

98 jniKNAi KM & MA^^ OmMUNiCAnoN QuM