State of the Method: an Examination of Level of Analysis, Methodology

University of Kentucky

From the SelectedWorks of Patric R. Spence

2007

State of the Method: An Examination of Level of Analysis, Methodology, Representation and Setting in Current Organizational Communication Research Patric R Spence, University of Kentukcy Colin R Baker

Available at: https://works.bepress.com/patric_spence/79/ Journal of the Northwest Communication Association, 36 (Spring 2007), 111–124.

State of the Method: An Examination of Level of Analysis, Methodology, Representation and Setting in Current Organizational Communication Research Patric R . Spence Calvin College Colin R . Baker Wayne State University

This study examines the frequency and content of organizationally related studies in six communication journals (Communication Monographs, Communication Studies, Communication Research Reports, Human Communication Research, Journal of Applied Communication Research and The Western Journal of Communication) over a seven-year period, from 1998–2004, for the purposes of describing current organizational research. The authors report that 14% of the journal articles were classified as “organizational” and of those, half were quantitative. The most common method used was the field study (46%), followed by surveys (29%). Few studies used multiple methodologies, and the authors discuss these implications.

This study describes the characteristics of studies concerning organizational communication that are published in select communication journals. We are particularly interested in the types of methods and settings explored by researchers, and we ground our examination in prior studies that have

Patric R. Spence (PhD, Wayne State University) is an Assistant Professor in the Department of Communication Arts and Sciences at Calvin College. Spence’s research interests include organizational crisis, risk communication and the role of new media in emergency response. His recent research has appeared in Journal of Applied Communication Research, Sociological Spectrum and the Journal of Emergency Management. Direct inquiries to [email protected].

Colin R. Baker (Ph.D., Michigan State University) is an Assistant Professor in the Department of Communication at Wayne State University. His research interests include communication during the organizational entry process, the impact of leadership on group communication, and social influence in applied contexts.

111 Journal of the Northwest Communication Association examined the history of organizational communication. Several published studies have examined the history of organizational communication research (Greenbaum, Hellweg & Walker, 1985; Tompkins & Redding, 1988) and topics researched (Allen, Gotcher, & Seibert, 1993). Journals have published issues devoted specifically to organizational communication [Hu- man Communication Research, 32(4) and Communication Research, 18(2)]. Yet, many questions concerning organizational research remain unexplored. For example, although a special issue of Human Communication Research was devoted to multi-level research, the issue did not identify the representation of multi-level research in the field. An article identifying research methodologies in organizational communication studies was published in Communication Studies which covered years 1979–1989 (Wert-Gray, Center, Brashers, & Meyers, 1991). However the study did not examine the setting of data collection or type of analysis employed. Moreover, the data presented in the Communication Studies article are more than two decades old and we might expect that much has changed. This study addresses select issues which previously have been unexamined in the literature. First, the representation of organizational communication articles within communication journals is identified. Next, the extent to which authors examine communication phenomena in organizations at multiple levels of analysis is reviewed, followed by investigations into the preferred methods and settings used by organizational communication researchers. Organizational Communication and Multi-level Research

As noted by Smith, Schneider and Dickson (2005), in organizational research the belief that everything interacts with everything else has been a consistent theme. Several behavioral researchers have suggested that the most significant impediment to the advancement of organizational research knowledge is the failure to simultaneously incorporate individual and organizational influences interacting across levels of analysis (Johns, 2001). The problem is called the micro-macro distinction to organizational scientists, contextualization to sociologists, and multi-level research to others. Cappelli and Sherer (1991) argue that it is impossible to develop a common paradigm for organizational behavior without incorporating multiple levels of analysis in research.

112 State of the Method

Scholars generally conduct research at only one level of analysis (often at the individual level, but other times at the group, organizational or industrial levels). Regardless, researchers often look to the next lowest level for explanatory purposes (Hackman, 2003). The dominant methodological approach to multi-level research is based on what Klein, Dansereau and Hall, (1994) call the mixed determinants model. In this model, both context and individual variables are hypothesized to affect an outcome variable at the individual level of analysis. However, other models do exist (Chan, 1998; Earley & Brittain, 1993). The motivation for conducting multi-level research is to produce results that are valid on multiple levels, avoiding the impulse to simply generalize from one level of analysis to another. Results from one level of analysis do not generalize well or exactly to other levels except under the most restric- tive circumstances (Firebaugh, 1979). Therefore, generalizing data that have been aggregated to the group level, back down to the individual level at which the data were collected, is a well known ecological fallacy (Kozlowski & Klein, 2000). For example, aggregate team problem-solving ability may be highly related to success at the team level but does not necessarily generalize to the problem solving abilities of the individual. Ecological correlations based on aggregate data are generally inflated estimates of lower level relationships (Robinson, 1950). This is true for findings at the lower level of analysis when generalization to a higher level is attempted. Such an attempt is called an atomistic fallacy. Unlike many areas of communication inquiry, studies of organizational communication allow for investigation of constructs at multiple levels of analysis. A study’s measurement may be at the organizational, group, or individual level, or a combination of several levels simultaneously. Klein and Kozlowski (2000) argue that although organizations exist at multiple levels, organizational researchers [in general] do not treat them as such. Alternatively, organizational communication scholars may be more attuned to these arguments, as the study of organizational communication has come a long way from being interpersonal communication “in a box.” Through quantitative methods that enable us to observe information linkages at multiple levels simultaneously, as well as qualitative case studies that allow

113 Journal of the Northwest Communication Association us to acquire an understanding of multiple influences on organizational behaviors, we may see a fair amount of articles investigating communicative behaviors at multiple levels of analysis. Therefore, scholars attune to these issues suggest a need to engage in multi-level research in both the theoretical and methodological realms. Moreover, organizational communication scholars may be uniquely advantaged over other scholars to look at context characteristics in research. Thus, the extent to which organizational communication research examines communication phenomena at multiple levels of analysis is examined. Organizational Communication Research and Methodological Choice

Organizational researchers have an advantage when it comes to levels of analysis issues in that while quantitative multilevel research is often statisti- cally complex, qualitative researchers may find it comparatively less taxing to address the impact of certain variables at multiple levels of analysis. As compared to other disciplines (e.g., business, management and industrial/ organizational psychology), communication researchers perhaps engage in more qualitative studies allowing these researchers to more easily and more often examine important issues at multiple levels of analysis. In addition to levels of analysis issues, questions related to methodologies and settings utilized by active researchers are important to our understanding of the present direction of the field. Organizational scholars hold many varying assumptions regarding the nature of scientific inquiry into the study of communication research. Mea- surement can be distinguished with respect to whether meaningful numeric symbols are used (quantitative measurement) or other non-numeric symbols are used to determine meaning (qualitative measurement) (Babbie, 1995; Frey, Botan & Kreps, 2000). Although some scholars advocate for one perspective over the other, many perceive value in seeking to understand organizational communication from a multiplicity of perspectives. In all, organizational communication scholars have a variety of perspectives from which they understand organizations. Examined next are the broad methodological preferences of organizational communication scholars as represented in communication journals.

114 State of the Method

Research Setting

Choice of setting for research is often based on idiosyncrasies of particular researchers, their training, and/or the research topic. However there are well known implications of these basic research decisions. Although most researchers attempt to make choices which propagate internal validity, there has been some debate in the social sciences over issues concerning both the internal and external validity of findings depending on where the data collection occurred. For example Sears (1986) suggested that the frequent use of students as a sample has created a genre of research labeled the “science of the sophomore.” However, alternative arguments have been articulated in defense of laboratory research that suggest that even though the external validity of a study may be limited to certain populations or certain unusual circumstances, there is great value in knowing what is possible through the direct manipulation of variables (Mook, 1983). Because laboratory experiments can be readily conducted, internal review boards may feel there is less at stake than collecting data from organizations. It is often more difficult to gain access to organizations to students and organizational researchers may be more inclined to use the laboratory setting to collect data. Thus the current study examines the setting of data collection. In sum, the aim of this research is to determine the number of organizational articles relative to the total number of articles published, and to describe in greater detail the methodological choices of organizational researchers in selected communication journals. Our research questions include the following: RQ1: What percentage of published articles are devoted to organizational communication? RQ2: Among organizational communication studies, what is the prevalence of different types of measurement? RQ3: Among organizational communication studies, what are the levels of analyses most often investigated? RQ4: Among organizational communication studies, what settings are most often used in research?

115 Journal of the Northwest Communication Association

Methodology

Sample A census of articles in six communication journals over a seven year period (1998–2004) was conducted, resulting in the examination of 1,108 articles. In order to give organizational articles the best possible chance for representation in the published communication literature, journals were selected based on the following criteria: (1) the journal must be a communication journal as evidenced by an affiliation with a communication association, (2) the journals selected must not restrict submissions to a particular theoretical or methodological perspective; (3) selected journals must publish articles concerning a variety of topics of communication inquiry or which limit submissions to organizational communication research in particular; and (4) priority was given to journals that explicitly stated that they accepted organizational articles. The six communication journals that were selected for analysis were: Communication Monographs, Communication Studies, Communication Research Reports, The Western Journal of Communication, Human Communication Research and the Journal of Applied Communication 1 Research. Procedure and Measures

Our first step was to create categories of journal articles by conducting a pilot test using the six journals noted above. For the pilot test, two graduate students were instructed to determine whether an article was organizational in nature. The students then read through the article making note of the type of measurement, the level of analysis, and the setting. Based on the pilot test (the two independent coders attained a percentage agreement of .82) and sub- sequent discussions, the categories were refined, and a codebook and coding sheet were completed. The final coding categories are as follows. Article Classification. In order to address Research Question 1 (the percentage of journal articles devoted to organizational studies) we needed to develop a method for classifying each article. Coders were instructed to first determine whether the article investigated a topic that they determined to be organizational. An article was considered organizational if data were collected in order to explain or predict an organizational communication

116 State of the Method phenomena, or where an article directly focused on an organizational context. When an article met these requirements, the coder was to continue coding, moving on to the next category. If the article was not considered organizational, then coder moved on to the next article. Type of Measurement. The coder then determined whether the article used qualitative measures, quantitative measures, a combination of measures, or no measures. An article was categorized as quantitative if it used meaningful numeric symbols while an article was considered qualitative if it employed meaningful non-numeric symbols such as text or ethnography. The study was coded as a combination if measurement triangulation occurred (e.g., a study used a scale and an interview), or it the article used multiple measures. A study was coded as no measurement if none was used. Level of analysis. An article was coded according to the following levels of analysis: individual if there were one data collection point (or as many data points as there are people); group level., or organizational level. If the article studied a macro-level entity, such as society, it was coded as other. The following categories were developed to examine multiple levels of analysis: individual and group; individual and organizational; individual and other; group and organizational; group and other; organizational and other; individual, group and organizational; individual, group and other; individual, organizational and other; group organizational and other; and individual, group, organizational and other. Data collection setting. The research setting was the final aspect analyzed. If the data collection occurred in the field, the participants’ natural setting or collected in an environment that the researcher did not manu- facture it was classified as such. Other classifications included a laboratory setting; distribution of surveys outside of the organization or laboratory setting; and the use of multiple settings also occur. An article that has data collected from any combination of the field, laboratory or surveys was classified as multiple settings. The final category included settings such as non-organizational archival records.

117 Journal of the Northwest Communication Association

Coding

Two doctoral students and one undergraduate student served as coders. These individuals underwent approximately five hours of training on coding procedures. Intercoder reliability was assessed using Cohen’s kappa (Cohen, 1960). Reliability for overall article classification was .86 and was deemed acceptable using the criteria of Banerjee, Capozzoli, McSweeney and Sinha (1999), who suggest that levels of agreement of .75 and greater demonstrate excellent agreement beyond the role of chance. Results

The number of organizational articles was first determined in the pool of 1,108. Approximately 14% of the articles were classified as organizational (N = 153). In addition we found that the year 2000 had the highest propor- tion of organizational articles: 19% (N = 30). Among publications in the observed time period (between 1998 and 2004), the Journal of Applied Communication Research published the largest percentage of organizational communication articles at 32% (N = 46) and Communication Research Reports was second at 11% (N = 35). (See Table 1).

Table 1 Reliability and representation of organizational communication articles in six communication journals

Number of Journal N Organizational % Articles

HCR 151 14 9%

Mono 154 19 12%

CRR 306 35 11%

WJC 153 17 11%

CS 204 22 11%

JACR 140 46 33%

Overall 1108 153 14%

118 State of the Method

Most of the articles used empirical data (86%). Over half (55%) of the organizational articles focused on the individual level (N = 85). There were only 29 (19%) multilevel studies in the seven year period. Half of the organizational articles were quantitative (N = 76), with 32% (N = 49), using qualitative methods. In addition, 5% (N = 7 articles) utilized a combination of qualitative and quantitative measures, and the remaining 14% (N = 21 articles) had no measurement. (See Table 2).

Table 2 Measurement type in organizational communication articles in six communication journals

Journal N Quantitative Qualitative Multiple None

HCR 16 10 4 0 2

Mono 19 6 7 1 5

CRR 33 29 2 1 1

WJC 17 8 6 1 2

CS 22 4 9 1 8

JACR 46 19 21 3 3

Overall 153 76 49 7 21

Among the qualitative studies, 37 out of 45 (82%) took place in a field setting, whereas, for the quantitative studies, 53 of the 76 (70%) collected data outside the organization through either surveys or laboratory experiments. Methodologies employed included the following: 46% (N = 70) were field studies; 29% (N = 45) used surveys; 5% (N = 9) collected data from multiple setting; and the remaining 4% (N = 7) were conducted in a laboratory. (See Table 3).

119 Journal of the Northwest Communication Association

Table 3 Setting of data collection for organizational communication articles in six communication journals

Multiple Journal N Field Laboratory Survey None Settings

HCR 16 4 1 3 5 3

Mono 19 8 1 4 1 5

CRR 33 10 2 20 0 1

WJC 17 6 1 7 1 2

CS 22 8 0 5 1 8

JACR 46 34 2 6 1 3

Overall 153 70 7 45 9 22

Discussion

Results indicate that there are few organizational multi-level studies represented within selected communication journals from 1998 through 2004. This finding is surprising given that organizational communication scholars have an advantage over other communication researchers in their ability to collect these types of data. We argue that the utility of gaining understanding of communication phenomena at multiple levels within organizations is essential to our complete understanding of communicative behaviors and motivations. Findings also indicated that organizational communication articles are most often quantitative. This finding offers a contrast to the papers presented at annual meetings of the organizational communication divisions such as the National Communication Association (NCA) and Interna- tional Communication Association (ICA), which typically have a higher percentage of research using qualitative methodologies. For example, in 2003, the Organizational Communication Division of NCA reported that

120 State of the Method more than 80% of submitted manuscripts for conference paper review were qualitative. The journals chosen for this study suggest greater interest in quantitative work, with qualitative methodologies underrepresented. It is not readily apparent as to why this may be the case. It is possible that there is some bias against qualitative research in general in the publication and editorial process. It may also be the case that qualitative research is published in journals other than those studied here. It could be that although there is a significant amount of excellent qualitative research being conducted and published, that it is more often less publishable for a variety of reasons. Most likely however is that qualitative researchers may be publishing in a wider range of journals or in journals that publish only qualitative work, whereas the criteria for inclusion in this study was that a journal must be willing to publish a variety of methodologies. When we compare our findings to those of Wert-Gray et. al., who analyzed research methods in similar organizational communication journals during the 1980s, quantitative methods comprised 58% of the studies, compared to our finding of 50%, a relatively stable pattern. More interesting, perhaps, is the relative absence of organizational communication articles using mixed or triangulated methods. Of the total articles examined only seven could be categorized as such. It appears that researchers are reluc- tant, unable, or do not desire to work through the inherent differences in epistemological and ontological assumptions from which the prototypi- cal researcher from each of these perspectives may operate. Unfortunately, quantitative researchers are not addressing the issues of multilevel analysis in organizational communication. Statistical techniques exist that allow researchers to look at the simultaneous effects of the group and organization on individual behaviors. Regarding the setting of communication studies, it is promising to see that a majority of organizational researchers in this sample are interested in examining actual organizations and that researchers have gained access to a diverse set of contexts. This analysis indicates that researchers in organizational communication are conducting more field studies than surveys or laboratory studies. Of significant importance is that work conducted in the organization allows the researcher to examine communication behaviors in

121 Journal of the Northwest Communication Association context and at multiple levels. Indeed it was found that those conducting research in the field setting were more likely (and able) to conduct analysis at multiple levels. Although some have argued that organizational communication studies are moving toward the study of macro-level variables (see, for example, Wert-Gray, et. al., 1991), this trend seems unlikely. A majority of organizational communication studies investigated data at the individual level of analysis rather than the group or organizational level. It is possible that since the 1980s, researchers have realized the difficulty in quantitatively and qualitatively measuring macro issues in communication and have therefore focused instead on individual traits and behaviors. Our analysis revealed that organizational communication studies in general were less represented overall: only 14% of studies published in the journals selected over the seven-year period were considered organizational in nature. Limitations and Future Research

The sample of communication journals in this study is by no means exhaus- tive; however, the sample is representative of communication journals in which organizational scholars are likely to publish their findings. A more comprehensive study of methods utilized in organizational communication articles could involve the analysis of more years of published work, and a broader spectrum of communication journals. Analyzing data from a longer time span will enable the use of trend analysis as well as reduce the chances of findings being influenced by editorial idiosyncrasies. Conclusion

There still exists a need for multi-level research in organizational communication. Among communication journals, there is a failure to simultaneously incorporate individual and organizational influences interacting across levels of analysis. This is surprising given that the various methods and theory of organizational communication are bound together by the common conceptual problem of analyzing the interaction between sym- bolic action and social/organizational structures (Conrad & Haynes, 2001).

122 State of the Method

Do these structures to which they refer predominately exist at a single level? Clearly multi-level research is underrepresented in the literature, which begs the question: why? Endnotes

1 Because of the four narrowly defined selection criteria, journals such asManage - ment Communication Quarterly, Journal of Communication and Communication Research were omitted from consideration. It is worth noting that a cursory glance of the literatures represented in these journals yields relatively low levels of organizational communication articles. As such, the authors were of the opinion that inclusion of these and other journals would have led to artificially deflated levels of organizational representation across journals. References

Allen, M. W., Gotcher, J. M. & Seibert, J. H. (1993). A decade of organizational communication research: Journal Articles 1980–1991. In S. Deetz, (Ed.), Communication Yearbook 16, London: Sage. Babbie, E. (1995). The practice of social research (7th ed). Boston: Wadsworth. Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D. (1999). Beyond kappa: A review of interrater agreement measures. Canadian Journal of Statistics, 27, 2–23. Cappelli, P. & Sherer, P. D. (1991). The missing role of context in OB: The need for a meso-level approach. Research in Organizational Behavior, 13, 55–110. Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: a typology of composition models. Journal of Applied Psychology, 83, 234–246. Cohen, J. (1960). A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20, 37–46. Conrad, C., & Haynes, J. (2001). Development of key constructs. In F.M. Jablin and L.L. Put- nam (Eds.), The new handbook of organizational communication: Advances in theory, research, and methods. (pp. 47–77). Thousand Oaks: Sage. Earley, P. C., & Brittain, J. (1993). Cross-level analysis of organizations: social resource management model. Research in Organizational Behavior, 15, 357–408. Firebaugh, G. (1979). Assessing group effects: A comparison of two methods. Sociological Methods and Research, 7, 384–395.

123 Journal of the Northwest Communication Association

Frey, L. R., Botan, C. H., & Kreps, G. L. (2000). Investigating communication: An introduction to research methods (2nd ed.). New York: Allyn and Bacon. Greenbaum, H, H., Hellweg, S. A., & Walter, J. W. (1985). Organizational communication: Abstracts, analysis and overview. Beverly Hills, CA: Sage. Hackman, J. R. (2003). Learning more by crossing levels: Evidence from airplanes, hospitals, and orchestras. Journal of Organizational Behavior, 24, 905–922. Johns, G. (2001). In praise of context. Journal of Organizational Behavior, 22, 31–42. Klein, K. J., Dansereau, F., & Hall, R. J. (1994). Levels issues in theory development, data collection, and analysis. Academy of Management Review, 19, 195–229. Klein, K. J., & Kozlowski, S. W. J. (2000). From micro to meso: Critical steps in conceptualizing and conducting multi-level research. Organizational Research Methods, 3, 211–236. Kozlowski, S. W. J., & Klein, K. J. (2000). A multi level approach to theory and research in organizations: Contextual, temporal, and emergent processes. In K.J. Klein & S.W.J. Kozlowski (Eds.) Multilevel theory, research and methods in organizations: Foundations, ex- tentions, and new directions (pp. 3–90). San Francisco, CA: Jossey-Bass. Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 40, 379–387. Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351–357. Sears, D. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology’s view of human nature. Journal of Personality and Social Psychology, 51, 515–530. Smith, D. B., Schneider, B., & Dickson, M. W. (2005). Meso Organizational Behavior: Com- ments on the Third Paradigm. To appear in W. Nord, (Ed.), Handbook of Organizational Studies. Sage. Tompkins, P. K., & Redding, W. C. (1988). Organizational communication: Past and present tenses. In G. M. Goldhaber & G. A. Barnett (Eds.), Handbook of organizational communication (pp. 5–33). Norwood, NJ: Ablex. Wert-Gray, S., Center, C., Brashers, D. E., & Meyers, R. A. (1991). Research topics and methodological orientations in organizational communication: A decade in review. Com- munication Studies, 42, 141–154.

124