CONCEPTUALIZATION, F OPERATIONALIZATION, 3 AND MEASUREMENT

What You'll Learn in This Chapter

You'll see how the interrelated steps of conceptualization, operationalization, and measurement allow researchers to turn a general idea for a research topic into useful and valid measurements in the real world. You'll also see that an essential part of this process involves transforming the relatively vague terms of ordinary language into precise objects of study with well-defined and measurable meanings. In this chapter. . .

An Opening Quandary eople sometimes doubt the social re- Introduction P searcher's ability to measure and study Measuring Anything That Exists things that matter. For many people, for ex- Conceptions, Concepts, and Reality ample, religious faith is a deep and impor- Conceptions as Constructs tant part of life. Yet both the religious and the nonreligious might question the social re- Conceptualization searcher's ability to measure how religious a Indicators and Dimensions given person or group is and-even more The Interchangeability of Indicators difficult a measurement-why some people Real, Nominal, and Operational Definitions are religious and others not. Creating Conceptual Order This chapter will show that social re- An Example of Conceptualization: The Concept searchers can't say definitively who is reli- of Anomie gious and who is not, nor what percentage of Definitions in Descriptive and Explanatory people in a particular population are reli- Studies gious, but they can to an extent determine Operationalization Choices the causes of religiosity. How can that be? Range of Variation Variations between the Extremes A Note on Dimensions Defining Variables and Attributes Levels of Wleasurement INTRODUCTION Single or Multiple Indicators Some Illustrations of Operationalization Choices This chapter and the next deal with how re- Operationalization Goes On and On searchers move from a general idea about what to Criteria of Measurement Quality study to effective and well-defined measurements Precision and Accuracy in the real world. This chapter discusses the inter- Reliability related processes of conceptualization, opera- Validity tionalization, and measurement. Chapter 6 builds Who Decides What's Valid? on this foundation to discuss types of measure- Tension between Reliability and Validity ments that are more complex. A Quandary Revisited \Me begin this chapter by confronting the hidden concern people sometimes have about whether it's Main Points truly possible to measure the stuff of life: love, hate, Key Terms prejudice, religiosity, radicalism, alienation. Over Review Questions the next few pages, we'll see that researchers can measure anything that exists. Once that point has Additional Readings been established, we'll turn to the steps involved Multimedia Resources in actual measurement. 122 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMEN1

MEASURING ANYTHING THAT EXISTS Similar points apply to relgious afllilinlion. Some- times this variable refers to official membership in a particular church, temple, mosque, and so forth; Earlier in this book, 1 said that one of the two pillars other times it simply means whatever religion, if of science is observation. Because this word can any, one identifies oneself with. Perhaps to you it suggest a casual, passive activity, scientists often means something else, such as attendance at reli- use the term measurement instead, meaning care- gious services. ful, deliberate observations of the real world for the The truth is that neither party aflliation nor reli- purpose of describing objects and events in terms gious aflliation has any real meaning, if by "real" of the attributes composing a variable. we mean corresponding to some objective aspect Like the people in the Opening Quandary, you of reality. These variables do not exist in nature. may have some reservations about the ability of They are merely terms we have made up and as- science to measure the really important aspects of signed specific meanings to for some purpose, human social existence. If you've read research re- such as doing social research. ports dealing with something like liberalism or re- But, you might object, "political affiliation" and ligion or prejudice, you may have been dissatisfied "religious affiliation"-and a host of other things with the way the researchers measured whatever social researchers are interested in, such as preju- they were studying. You may have felt that they dice or compassion-have some reality. After all, were too superficial, missing the aspects that really we make statements about them, such as "In Hap- matter most. Maybe they measured religiosity as pytown, 55 percent of the adults affiliate with the the number of times a person went to religious Republican Party, and 45 percent of them are Epis- services, or liberalism by how people voted in a copalians. Overall, people in Happytown are low in single election. Your dissatisfaction would surely prejudice and high in compassion." Even ordinary have increased if you had found yourself being people, not just social researchers, have been misclassified by the measurement system. known to make statements like that. If these things Your dissatisfaction reflects an important fact do not exist in reality, what is it that we're measur- about social research: Most of the variables we ing and talking about? want to study do not actually exist in the way What indeed? Let's take a closer look by co11- that rocks exist, Indeed, they are made up. More- sidering a variable of interest to many social over, they seldom have a single, unambiguous researchers (and many other people as well)- meaning. pr-ejudice. To see what I mean, suppose we want to study political party aJ1iation. To measure this variable, we might consult the list of registered voters to Conceptions, Concepts, and Reality note whether the people we were studying were As you and I wander down the road of life, we have registered as Democrats or Republicans and take already observed a lot of things and known they that as a measure of their party affiliation. But we were real through our observations, and we have could also simply ask someone what party they heard reports from other people that have seemed identify with and take their response as our mea- real. For example: sure. Notice that these two different measurement possibilities reflect somewhat different definitions We personally heard people say nasty things of "political party affiliation." They might even pro- about minority groups. duce different results: Someone may have regis- We heard people said women were inferior tered as a Democrat years ago but gravitated more to men. and more toward a Republican philosophy over We read about African Americans being time. Or someone who is registered with neither lynched. political party may, when asked, say she is affiliated We read that women and minorities earned with the one she feels the most kinship with. less for the same work. MEASURING ANYTHING THAT EXISTS 123

We learned about "ethnic cleansing" and wars our conceptions and the things we observe that are in which one ethnic group tried to eradicate related to those conceptions. These terms make another. it possible for us to communicate and eventually agree on what we will specifically mean by those With additional experience, we noticed some- terms. In social research, the process of coming to thing more. People who participated in lynching an agreement about what terms mean is conceptu- were also quite likely to call African Americans alization, and the result is called a concept. ugly names. A lot of them, moreover, seemed to Let's take another example of a conception. want women to "stay in their place." Eventually it Suppose that I'm going to meet someone named dawned on us that these several tendencies often Pat, whom you already know. I ask you what Pat is appeared together in the same people and also had like. Now suppose that you've seen Pat help lost something in common. At some point, someone children find their parents and put a tiny bird back had a bright idea: "Let's use the wordprejudicecf for in its nest. Pat got you to take turkeys to poor fam- people like that. We can use the term even if they ilies on Thanksgiving and to visit a children's hos- don't do all those things-as long as they're pretty pital at Christmas. You've seen Pat weep through a much like that." movie about a mother overcoming adversities to Being basically agreeable and interested in save and protect her child. As you search through efficiency, we agreed to go along with the system. your mental files, you may find all or most of those That's where "prejudice" came from. We never ob- phenomena recorded on a single sheet labeled served it. We just agreed to use it as a shortcut, a "compassionate." You look over the other entries name that represents a collection of apparently re- on the page, and you find they seem to provide lated phenomena that we've each observed in the an accurate description of Pat. So you say, "Pat is course of life. In short, we made it up. compassionate." Here's another clue that prejudice isn't some- Now I leaf through my own mental file drawer thing that exists apart from our rough agreement to until I find a sheet marked "compassionate." I then use the term in a certain way. Each of us develops look over the things written on my sheet, and I say, our own mental image of what the set of real phe- "Oh, that's nice." I now feel I know what Pat is like, nomena we've observed represents in general and but my expectations reflect the entries on my file what these phenomena have in common. When I sheet, not yours. Later, when I meet Pat, I happen say the word pi-ejudice, it evokes a mental image in to find that my own experiences correspond to the your mind, just as it evokes one in mine. It's as entries 1 have on my "compassionate" file sheet, though file drawers in our minds contained thou- and I say that you sure were right. But suppose my sands of sheets of paper, with each sheet of paper observations of Pat contradict the things I have on labeled in the upper right-hand corner. A sheet of my file sheet. I tell you that I don't think Pat is very paper in each of our minds has the term prejudice compassionate, and we begin to compare notes. on it. On your sheet are all the things you've been You say, "I once saw Pat weep through a movie told about prejudice and everything you've ob- about a mother overcoming adversity to save and served that seems to be an example of it. My sheet protect her child." I look at my "compassionate has what I've been told about it plus all the things sheet" and can't find anything like that. Looking I've observed that seem examples of it-and mine elsewhere in my file, I locate that sort of phenome- isn't the same as yours. non on a sheet labeled "sentimental." I retort, The technical term for those mental images, "That's not compassion. That's just sentimentality." those sheets of paper in our mental file drawers, is To further strengthen my case, I tell you that I coi~ception.That is, I have a conception of preju- saw Pat refuse to give money to an organization dice, and so do you. We can't communicate these dedicated to saving whales from extinction. "That mental images directly, so we use the terms written represents a lack of compassion," I argue. You in the upper right-hand corner of our own mental search through your files and find saving the sheets of paper as a way of communicating about whales on two sheets-"environmental activism" 124 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION. AND MEASUREMENT and "cross-species datingr'-and you say so. Even- tually we set about comparing the entries we have on our respective sheets labeled "compassionate." We then discover that many of our mental images corresponding to that term differ. In the big picture, language and communication work only to the extent that our mental file-sheet entries overlap considerably The similarities on those sheets represent the agreements existing in our society As we grow up, we're all taught ap- proximately the same thing when we're first intro- duced to a particular term. Dictionaries formalize the agreements our society has about such terms. Each person, then, shapes his or her mental images to correspond with such agreements. But because all of us have different experiences and observa- tions, no t\uo people end up with exactly the same Conceptualization and operationalization involve the search for ways to measure the variables that set of entries on any sheet in their file systems. If we interest us. want to measure "prejudice" or "compassion," we must first stipulate what, exactly, counts as preju- dice or compassion for our purposes. served that person's gender. History books or min- Returning to the assertion made at the outset of utes of corporate board meetings provide indirect this chapter, we can measure anything that's real. observations of past social actions. Finally, the We can measure, for example, whether Pat actually third class of observables consists of collsti-uc&- puts the little bird back in its nest, visits the hospi- theoretical creations that are based on observa- tal on Christmas, weeps at the movie, or refuses to tions but that cannot be observed directly or indi- contribute to saving the whales. All of those be- rectly A good example is intelligence quotient, or haviors exist, so we can measure them. But is Pat JQ. It is constructed mathematically from observa- really compassionate? We can't answer that ques- tions of the answers given to a large number of tion; we can't measure compassion in any objec- questions on an IQ test. No one can directly or in- tive sense, because compassion doesn't exist the directly observe IQ. It is no more a "real" charac- way those things I just described exist. Compassion teristic of people than is compassion or prejudice. exists only in the form of the agreements we have Kaplan (1 964 : 49) defines concept as a "family of about how to use the term in communicating conceptions." A concept is, as Kaplan notes, a con- about things that are real. struct, something we create. Concepts like corn-- passion and prejudice are constructs created from your conception of them, my conception of them, Concepts as Constructs and the conceptions of all those who have ever Abraham Kaplan (1964) distinguishes three classes used these terms. They cannot be observed directly of things that scientists measure. The first class is or indirectly, because they don't exist. We made direct observnbles: those things we can observe them up. rather simply and directly, like the color of an apple To summarize, concepts are constructs derived or a check mark on a questionnaire. The second by mutual agreement from mental images (con- class, Indirect observables, requires "relatively more ceptions). Our conceptions summarize collec- subtle, complex, or indirect observations" (1964: tions of seemingly related observations and expe- 55). When we note a person's check mark beside riences. The observations and experiences are "female" in a questionnaire, we've indirectly ob- real, at least subjectively but conceptions, and the CONCEPTUALIZATION 125 concepts derived from them, are only mental cre- CONCEPTUALIZATION ations. The terms associated with concepts are merely devices created for the purposes of filing and communication. A term like prejudice is, ob- As we've seen, day-to-day communication usually jectively speaking, only a collection of letters. It has occurs through a system of vague and general no intrinsic reality beyond that. It has only the agreements about the use of terms. Although you meaning we agree to give it. and I do not agree completely about the use of the Usually, however, we fall into the trap of believ- term compassionnte, I'm probably safe in assuming ing that terms for constructs do have intrinsic that Pat won't pull the wings off flies. A wide range meaning, that they name real entities in the world. of misunderstandings and conflict-from the inter- That danger seems to grow stronger when we be- personal to the international-is the price we pay gin to take terms seriously and attempt to use them for our imprecision, but somehow we muddle precisely. Further, the danger is all the greater in through. Science, however, aims at more than the presence of experts who appear to know more muddling; it cannot operate in a context of such than we do about what the terms really mean: imprecision. Yielding to authority is easy in such a situation. The process through which we specify what we Once we assume that terms like prejudice and mean when we use particular terms in research is compnssion have real meanings, we begin the tor- called conceptualization. Suppose we want to tured task of discovering what those real meanings find out, for example, whether women are more are and what constitutes a genuine measurement compassionate than men. I suspect many people of them. Regarding constructs as real is called assume this is the case, but it might be interesting 1-eification. The reification of concepts in day-to-day to find out if it's really so. we can't meaningfully life is quite common. In science, we want to be study the question, let alone agree on the answer, quite clear about what it is we are actually measur- without some working agreements about the ing, but this aim brings a pitfall with it. Settling on meaning of compassion. They are "working" agree- the "best" way of measuring a variable in a particu- ments in the sense that they allow us to work on lar study may imply that we've discovered the "real" the question. We don't need to agree or even pre- meaning of the concept involved. In fact, concepts tend to agree that a particular specification is ulti- have no real, true, or objective meanings-only mately the best one. those we agree are best for a particular purpose. Conceptualization, then, produces a specific, Does this discussion imply that compassion, agreed-on meaning for a concept for the purposes prejudice, and similar constructs cannot be mea- of research. This process of specifying exact mean- sured? Interestingly, the answer is no. (And a good ing involves describing the indicators we'll be us- thing, too, or a lot of us social researcher types ing to measure our concept and the different as- would be out of work.) I've said that we can mea- pects of the concept, called dimensions. sure anything that's real. Constructs aren't real in the way that trees are real, but they do have an- Indicators and Dimensions other important virtue: They are useful. That is, they help us organize, communicate about, and Conceptualization gives definite meaning to a con- understand things that are real. They help us make cept by specifying one or more indicators of what predictions about real things. Some of those pre- dictions even turn out to be true. Constructs can work this way because, while not real or observ- able in themselves, they have a definite relationship conceptualization The mental process whereby fuzzy and imprecise notions (concepts) are made more specific and to things that are real and observable. The bridge precise. So you want to study prejudce. What do you mean from direct and indirect observables to usef~llcon- by prejud~ce?Are there different kinds of prejudce? What structs is the process called conceptualization. are they? we have in mind. An indicator is a sign of the pres- know them and learning more about their views. ence or absence of the concept we're studying. You could tell them you were a social researcher Here's an example. interested in learning about their group, or perhaps We might agree that visiting children's hospitals you would just express an interest ~nlearning more during Christmas and Hanukkah is an indicator of without saying why. compassion. Putting little birds back in their nests In the course of your conversations with group might be agreed on as another indicator, and so members and perhaps attendance of religious ser- forth. If the unit of analysis for our study is the vices, you would put yourself in situations where individual person, we can then observe the pres- you could come to understand what the cult mem- ence or absence of each indicator for each person bers mean by compassion.You might learn, for ex- under study. Going beyond that, we can add up the ample, that members of the group were so deeply number of indicators of compassion observed for concerned about sinners burning in hell that they each individual. We might agree on ten specific were willing to be aggressive, even violent, to indicators, for example, and find six present in our make people change their sinful ways. Within their study of Pat, three for John, nine for Mary, and own paradigm, then, cult members would see so forth. beating up gays, prostitutes, and abortion doctors Returning to our question about whether men as acts of compassion. or women are more compassionate, we might cal- Social researchers focus their attention on the culate that the women we studied displayed an av- meanings that people under study give to words erage of 6.5 indicators of compassion, the men an and actions. Doing so can often clarify the behav- average of 3.2. On the basis of our quantitative iors observed: At least now you understand how analysis of group difference, we might therefore the cult can see violent acts as compassionate. On conclude that women are, on the whole, more the other hand, paying attention to what words and compassionate than men. actions mean to the people under study almost al- Usually, though, it's not that simple. Imagine ways complicates the concepts researchers are in- you're interested in understanding a small funda- terested in. (We'll return to this issue when we dis- mentalist religious cult, particularly their views on cuss the validity of measures, toward the end of various groups: gays, nonbelievers, feminists, and this chapter.) others. In fact, they suggest that anyone who re- Whenever we take our concepts seriously and fuses to join their group and abide by its teachings set about specifying what we mean by them, we will "burn in hell." In the context of your interest in discover disagreements and inconsistencies. Not compassion, they don't seem to have much. And only do you and I disagree, but each of us is likely yet, the group's literature often speaks of their to find a good deal of muddiness within our own compassion for others. You want to explore this mental images. If you take a moment to look at seeming paradox. what you mean by compassion, you'll probably To pursue this research interest, you might ar- find that your image contains several kinds of range to interact with cult members, getting to compassion. That is, the entries on your mental file sheet can be combined into groups and subgroups, say, compassion toward friends, coreligionists, hu-

-- mans, and birds. You can also find several different indicator An observation that we choose to conslder as strategies for making combinations. For example, a reflection of a variable we wish to study Thus, for ex- you might group the entries into feelings and ample, attending religious servlces mlght be considered actions. an ind~catorof rel~glosity The technical term for such groupings is di- dimension A specifiable aspect of a concept. "Religios- ity," for example, might be specified in terms of a belief mension: a specifiable aspect of a concept. For in- dimension, a ritual dimension, a devotional dimension, a stance, we might speak of the "feeling dimension" knowledge dimension, and so forth. of compassion and its "action dimension." In a CONCEPTUALIZATION 127 different grouping scheme, we might distinguish phenomena that were all called "genocide" (2003: "compassion for humans" from "compassion for 15-1 8): animals." Or we might see compassion as helping Co~wenience.Sometimes the attempt to eradi- people have what we want for them versus what cate a group of people serves a function for the they want for themselves. Still differently, we might eradicators, such as Julius Caesar's attempt to distinguish compassion as forgiveness from com- eradicate tribes defeated in battle, fearing they passion as pity. would be difficult to rule. Or when gold was Thus, we could subdivide compassion into sev- discovered on Cherokee land in the Southeast- eral clearly defined dimensions. A complete con- ern United States in the early nineteenth cen- ceptualization involves both specifying dimensions tury, the Cherokee were forcibly relocated to and identifying the various indicators for each. Oklahoma in an event known as the "Trail of Sometimes conceptualization aimed at identi- Tears," which ultimately killed as many as half fying different dimensions of a variable leads to of those forced to leave a different kind of distinction. We may conclude Revenge: When the Chinese of Nanking that we've been using the same word for meaning- bravely resisted the Japanese invaders in fully distinguishable concepts. In the following the early years of World War 11, the con- example, the researchers find, first, that "violence" querors felt they had been insulted by those is not a sufficient description of "genocide" and, they regarded as inferior beings. Tens of second, that the concept "genocide" itself com- thousands were slaughtered in the "Rape prises several distinct phenomena. Let's look at of Nanking" in 1937-1938. the process they went through to come to this Fear. The ethnic cleansing that recently conclusion. occurred in the former Yugoslavia was at When Daniel Chirot and Jennifer Edwards at- least partly motivated by economic com- tempted to define the concept of "genocide," they petition and worries that the growing Al- found that existing assumptions were not precise banian population of Kosovo was gaining enough for their purposes: political strength through numbers. Simi- The United Nations originally defined it as an at- iarly, the ~utu-attemptto eradicate the Tut- tempt to destroy "in whole or in part, a national, sis of Rwanda grew out of a fear that return- ethnic, racial, or religious group." If genocide is ing Tutsi refugees would seize control of distinct from other types of violence, it requires the country. Often intergroup fears such as its own unique explanation. - (2003:14) these grow out of long histories of atroci- ties, often inflicted in both directions. Notice the final comment in this excerpt, as it Purflcation: The Nazi Holocaust, probably provides an important insight into why researchers the most publicized case of genocide, was are so careful in specifying the concepts they study. intended as a purification of the "Aryan race." If genocide, such as the Holocaust, were simply an- While Jews were the main target, gypsies, other example of violence, like assaults and homi- homosexuals, and many other groups were cides, then what we know about violence in gen- also included. Other examples include the eral might explain genocide. If it differs from other Indonesian witch-hunt against communists forms of violence, then we may need a different ex- in 1965-1966 and the attempt to eradicate planation for it. So, the researchers began by sug- all non-Khmer Cambodians under Pol Pot gesting that "genocide" was a concept distinct from in the 1970s. "violence" for their purposes. Then, as Chirot and Edwards examined histori- No single theory of genocide could explajn these cal instances of genocide, they began concluding various forms of mayhem. Indeed, this act of con- that the motivations for launching genocidal may- ceptualization suggests four distinct phenomena, hem differed sufficiently to represent four distinct each needing a different set of explanations. 128 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMEN1

Specifying the different dimensions of a concept, The logic works like this. If we disagree totally then, often paves the way for a more sophisticated on the value of the indicators, one solution would understanding of what we're studying. To take be to study all of them. Suppose that women turn another example, we might observe that women out to be more compassionate than men on all 100 are more compassionate in terms of feelings, and indicators-on all the indicators you favor and on men more so in terms of actions-or vice versa. all of mine. Then we would be able to agree that Whichever turned out to be the case, we would not women are more compassionate than men even be able to say whether men or women are really though we still disagree on exactly what compas- more compassionate. Our research would have sion means in general. shown that there is no single answer to the ques- The interchangeability of indicators means that tion. That alone represents an advance in our un- if several different indicators all represent, to some derstanding of reality. degree, the same concept, then all of them will be- have the same way that the concept would behave if it were real and could be observed. Thus, given For an excellent sampling of concepts, variables, and indicators, go to the Gen- a basic agreement about what "compassion" is, if eral Social Survey codebook and explore women are generally more compassionate than some of the ways the researchers have men, we should be able to observe that difference measured various concepts: by using any reasonable measure of compassion. http://www.icpsr.umich.edu/GSS99/ If, on the other hand, women are more compas- subjectls-index.htm. * sionate than men on some indicators but not on others, we should see if the two sets of indicators represent different dimensions of compassion. You have now seen the fundamental logic of The Interchangeability of Indicators conceptualization and measurement. The discus- There is another way that the notion of indicators sions that follow are mainly refinements and ex- can help us in our attempts to understand reality by tensions of what you've just read. Before turning to means of "unreal" constructs Suppose, for the mo- a technical elaboration of measurement, however, ment, that you and I have compiled a list of 100 in- we need to fill out the picture of conceptualization dicators of compassion and its various dimensions. by looking at some of the ways social researchers Suppose further that we disagree widely on which provide the meanings of terms with standards, indicators give the clearest evidence of compas- consistency, and commonality. sion or its absence. If we pretty much agree on some indicators, we could focus our attention on those, and we would probably agree on the answer Real, Nominal, and they provided. We would then be able to say that Operational Definitions some people are more compassionate than others As we have seen, the design and execution of so- in some dimension. But suppose we don't really cial research requires us to clear away the confu- agree on any of the possible indicators. Surpris- sion over concepts and reality. To this end, logi- ingly, we can still reach an agreement on whether cians and scientists have distinguished three kinds men or women are the more compassionate. How of definitions: real, nornirial, and operational. we do that has to do with the interchangeability of The first of these reflects the reification of terms. indicators. As Carl Hempel has cautioned,

A "real" definition, according to traditional logic, *Each time the Internet icon appears, you'll be given help- is not a stipulation determining the meaning of ful leads for searching the World Wide Web; whenever you see a GSS reference, you'll be given some tips on using this some expression but a statement of the "essen- valuable resource. tial nature" or the "essential attributes" of some CONCEPTUALIZATION 129

entity. The notion of essential nature, however, terest in social activism could narrow to "individu- is so vague as to render this characterization als who are committed to empowerment and useless for the purposes of rigorous inquiry. social change" and further focus on discovering - (1952 : 6) "what experiences shaped the development of fully committed social activists." This focusing process In other words, trying to specify the "real" meaning is inescapably linked to the language we use. of concepts only leads to a quagmire: It mistakes a In some forms of qualitative research, the clari- construct for a real entity. fication of concepts is a key element in the collec- The specification of concepts in scientific in- tion of data. Suppose you were conducting inter- quiry depends instead on nominal and operational views and observations in a radical political group definitions. A nornu?al defin~tionis one that is sim- devoted to combating oppression in U.S. society. ply assigned to a term without any claim that the Imagine how the meaning of oppression would definition represents a "real" entity. Nominal defini- shift as you delved more and more deeply into the tions are arbitrary-I could define compassion as members' experiences and worldviews. For ex- "plucking feathers off helpless birds" if I wanted ample, you might start out thinking of oppression to-but they can be more or less useful. For most in physical and perhaps economic terms. The more purposes, especially communication, that last defi- you learned about the group, however, the more nition of compassion would be pretty useless. Most you might appreciate the possibility of psychologi- nominal definitions represent some consensus, cal oppression. or convention, about how a particular term is to he same point applies even to contexts where be used. meanings might seem more fixed. In the analysis of An operatlonu1 definition, as you may recall from textual materials, for example, social researchers Chapter 2, specifies precisely how a concept will be sometimes speak of the "hermeneutic circle," a measured-that is, the operations we choose to cyclical process of ever deeper understanding. perform. An operational definition is nominal rather than real, but it has the advantage of achiev- The understanding of a text takes place through ing maximum clarity about what a concept means a process in which the meaning of the separate in the context of a given study. In the midst of dis- parts is determined by the global meaning of the agreement and confusion over what a term "really" text as it is anticipated. The closer determination means, we can specify a working definition for the of the meaning of the separate parts may even- purposes of an inquiry Wishing to examine socio- tually change the originally anticipated meaning economic status (SES) in a study, for example, we of the totality, which again influences the mean- may simply specify that we are going to treat SES ing of the separate parts, and so on. - (KVALE as a combination of income and educational at- 1996.47) tainment. In this decision, we rule out other pos- Consider the concept "prejudice." Suppose you sible aspects of SES: occupational status, money in needed to write a definition of the term. You might the bank, property, lineage, lifestyle, and so forth. start out thinking about racial/ethnic prejudice. At Our findings will then be interesting to the extent some point you would realize you should probably that our definition of SES is useful for our purpose. allow for gender prejudice, religious prejudice, antigay prejudice, and the like in your definition. Examining each of these specific types of prejudice Creating Conceptual Order would affect your overall understanding of the The clarification of concepts is a continuing pro- general concept. As your general understanding cess in social research. Catherine Marshall and changed, however, you would likely see each of the Gretchen Rossman (1995:18) speak of a "concep- individual forms somewhat differently. tual funnel" through which a researcher's interest The continual refinement of concepts occurs becomes increasingly focused. Thus, a general in- in all social research methods. Often you'll find 130 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMENT yourself refining the meaning of important con- In this way we would create a working and cepts even as you write up your final report. workable definition of SES. Although others might Although conceptualization is a continuing pro- disagree with our conceptualization and opera- cess, it's vital to address it specifically at the be- tionalization, the definition would have one essen- ginning of any study design, especially rigorously tial scientific virtue It would be absolutely specific structured research designs such as surveys and and unambiguous. Even if someone disagreed experiments. In a survey, for example, operational- with our definition, that person would have a good ization results in a commitment to a specific set of idea how to interpret our research results, because questionnaire items that will represent the con- what we meant by SES-reflected in our analyses cepts under study. Without that commitment, the and conclusions-would be precise and clear study could not proceed further. Here is a diagram showing the progression of Even in less-structured research methods, how- measurement steps from our vague sense of what ever, it's important to begin with an initial set of an- a term means to specific measurements in a fully ticipated meanings that can be refined during data structured scientific study: collection and interpretation. No one seriously be- lieves we can observe life with no preconceptions; Conceptualization for this reason, scientific observers must be con- 1 scious of and explicit about these conceptual start- Nominal Definition ing points. 1 Let's explore initial conceptualization as it ap- Operational Definition plies to structured inquiries such as surveys and ex- 1 periments. Though specifying nominal definitions Measurements in the Real World focuses our observational strategy, it does not al- low us to observe. As a next step we must specify An Example of Conceptualization: exactly what we are going to observe, how we will The Concept of Anomie observe it, and how we will interpret various pos- sible observations. All these further specifications To look at the overall process of conceptualization make up the operational definition of the concept. in research, let's look briefly at the history of a spe- In the example of socioeconomic status, we cific social scientific concept. Researchers studying might decide to measure SES in terms of income urban riots are often interested in the part played and educational attainment. We might then ask by feelings of powerlessness. Social researchers survey respondents two questions: sometimes use the word nr~omiein this context. This term was first introduced into social science 1. What was your total family income during the by Emile Durkheim, the great French sociologist, in past 12 months? his classic 1897 study, Suicide. 2. What is the highest level of school you Using only government publications on suicide completed? rates in different regions and countries, Durkheim To organize our data, we would probably want produced a work of analytical genius. To determine to specify a system for categorizing the answers the effects of religion on suicide, he compared the people give us. For income, we might use cate- suicide rates of predominantly Protestant countries gories such as "under $5,000," "$5,000 to $1 0,000," with those of predominantly Catholic ones, Protes- and so on. Educational attainment might be simi- tant regions of Catholic countries with Catholic re- larly grouped in categories: less than high school, gions of Protestant countries, and so forth. To de- high school, college, graduate degree. Finally, we termine the possible effects of the weather, he would specify the way a person's responses to compared suicide rates in northern and southern these two questions would be combined to create countries and regions, and he examined the differ- a measure of SES. ent suicide rates across the months and seasons CONCEPTUALIZATION 131 of the year. Thus, he could draw conclusions about When the ends of action become contradictory, a supremely individualistic and personal act with- inaccessible or insignificant, a condition of out having any data about the individuals engaging anomie arises. Characterized by a general loss in it. of orientation and accompanied by feelings of At a more general level, Durkheim suggested "emptiness" and apathy, anomie can be simply that suicide also reflects the extent to which a soci- conceived as meaninglessness. - (1958: 132) ety's agreements are clear and stable. Noting that times of social upheaval and change often present Powell went on to suggest there were two dis- individuals with grave uncertainties about what is tinct kinds of anomia and to examine how the two expected of them, Durkheim suggested that such rose out of different occupational experiences to re- uncertainties cause confusion, anxiety, and even sult at times in suicide. In his study, however, Pow- self-destruction. To describe this societal condition ell did not measure anomia per se; he studied the of normlessness, Durkheim chose the term anomie. relationship between suicide and occupation, mak- Durkheim did not make this word up. Used in both ing inferences about the two kinds of anomia. Thus, German and French, it literally meant "without the study did not provide an operational definition law." The English term anomy had been used for at of anomia, only a further conceptualization. least three centuries before Durkheim to mean dis- Although many researchers have offered oper- regard for divine law. However, Durkheim created ational definitions of anomia, one name stands the social scientific concept of anomie. out over all. Two years before Powell's article ap- In the years that have followed the publication peared, Leo Srole (1956) published a set of ques- of Suicide, social scientists have found anomie a tionnaire items that he said provided a good mea- useful concept, and many have expanded on sure of anomia as experienced by individuals. It Durkheim's use. Robert Merton, in a classic article consists of five statements that subjects were entitled "Social Structure and Anomie" (1938), con- asked to agree or disagree with: cluded that anomie results from a disparity be- 1. In spite of what some people say, the lot of the tween the goals and means prescribed by a society. average man is getting worse. Monetary success, for example, is a widely shared 2. It's hardly fair to bring children into the goal in our society, yet not all individuals have the world with the way things look for the future. resources to achieve it through acceptable means. 3. Nowadays a person has to live pretty much An emphasis on the goal itself, Merton suggested, for today and let tomorrow take care of itself. produces normlessness, because those denied the 4. These days a person doesn't really know traditional avenues to wealth go about getting it who he can count on. through illegitimate means. Merton's discussion, 5. There's little use writing to public officials then, could be considered a further conceptualiza- because they aren't really interested in the tion of the concept of anomie. problems of the average man. - (1956: 7 13) Although Durkheim originally used the concept of anomie as a characteristic of societies, as did In the decades following its publication, the Merton after him, other social scientists have used Srole scale has become a research staple for social it to describe individuals. To clarify this distinction, scientists. You'll likely find this particular opera- some scholars have chosen to use anomie in refer- tionalization of anomia used in many of the re- ence to its original, societal meaning and to use the search projects reported in academic journals. term anomia in reference to the individual charac- Srole touches on this in the following box, "The teristic. In a given society, then, some individuals Origins of Anomia," which he prepared for this experience anomia, and others do not. Elwin Pow- book before his death. ell, writing 20 years after Merton, provided the fol- This abbreviated history of anomie and anomia lowing conceptualization of anomia (though using as social scientific concepts illustrates several the term anomie) as a characteristic of individuals: points. First, it is a good example of the process 132 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMENT

by Leo SroIe moral force of the interpersonal ties that bind us to our time, place, and past, and also his insights y career-long fixation on anomie began about the lethal consequences that can follow M with reading Durkheim's Le Suicide as a from shrinkage and decay in those ties. Harvard undergraduate. Later, as a graduate My interest in anomie received an eyewitness student at Chicago, I studied under two Durk- jolt at the finale of World War 11, when I served heimian anthropologists: William Lloyd Warner with the United Nations Relief and Rehabilitation and Alfred Radcliffe-Brown. Radcliffe-Brown Administration, helping to rebuild a war-torn Eu- had carried on a lively correspondence with rope. At the Nazi concentration camp of Dachau, Durkheim, making me a collateral "descendant" I saw first-hand the depths of dehumanization of the great French sociologist. that macro-social forces, such as those that en- For me, the early impact of Durkheim's work gaged Durkheim, could produce in individuals on suicide was mixed but permanent. On the one like Hitler, Eichmann, and the others serving their hand, I had serious reservations about his stren- dictates at all levels in the Nazi death factories. uous, ingenious, and often awkward efforts to Returning from my UNRRA post, I felt most ur- force the crude, bureaucratic records on suicide gently that the time was long overdue to come to rates to fit with his unidirectional sociological de- an understanding of the dynamics underlying terminism. On the other hand, I was moved by disintegrated social bonds. We needed to work Durkheim's unswerving preoccupation with the expeditiously, deemphasizing proliferation of

through which general concepts become opera- nomic measures developed by the U.S. Census Bu- tionalized measurements. This is not to say that the reau. Using such established measures has two ad- issue of how to operationalize anomie/anomia vantages: They have been extensively pretested has been resolved once and for all. Scholars will and debugged, and studies using the same scales surely continue to reconceptualize and reopera- can be compared. If you and I do separate studies of tionalize these concepts for years to come, contin- two different groups and use the Srole scale, we can ually seeking more useful measures. compare our two groups on the basis of anomia. The Srole scale illustrates another important Social scientists, then, can measure anything point. Letting conceptualization and operational- that's real; through conceptualization and opera- ization be open-ended does not necessarily pro- tionalization, they can even do a pretty good job of duce anarchy and chaos, as you might expect. Or- measuring things that aren't. Granting that such der often emerges. For one thing, although we concepts as socioeconomic status, prejudice, com- could define nnomia any way we chose-in terms passion, and anomia aren't ultimately real, social of, say, shoe size-we would likely define it in ways scientists can create order in handling them. This not too different from other people's mental im- order is based on utility, however, not on ultimate ages. If you were to use a really offbeat definition, truth. people would probably ignore you. A second source of order is that, as researchers DEFINITIONS IN DESCRIPTIVE discover the utility of a particular conceptualization AND EXPLANATORY STUDIES and operationalization of a concept, they're likely to adopt it, which leads to standardized definitions of concepts. Besides the Srole scale, examples in- As described in Chapter 4, two general purposes of clude IQ tests and a host of demographic and eco- research are description and explanation. The dis- DEFINITIONS IN DESCRIPTIVE AND EXPLANATORY STUDIES 133

macro-level theory in favor of a direct exploratory social meaning and to sharply segregate it from encounter with individuals, using newly devel- its individual manifestations. For the latter pur- oped state-of-the-art survey research method- pose, the cognate but hitherto obsolete Greek ology. Such research, I also felt, should focus on a term, anomia, readily suggested itself. broader spectrum of behavioral pathologies than I first published the anomia construct in a suicide. 1956 article in the American Sociological Review, * My initial investigations were a diverse effort. describing ways of operationalizing it, and pre- In 1950, for example, 1 was able to interview a senting the results of its initial field application sample of 401 bus riders in Springfield, Massa- research. By 1982, the Science Citation Index chusetts. Four years later, the Midtown Manhat- and Social Science Citation Index had listed tan Mental Health Study provided a much larger some 400 publications in political science, psy- population reach. These and other field projects chology, social work, and sociology journals gave me scope to expand and refine my mea- here and abroad that had cited use of that ar- surements of that quality in individuals which ticle's instruments or findings, warranting the reflected the macro-social quality Durkheim had American Institute for Scientific Information to called anomie. designate it a "citation classic." While I began by using Durkheim's term in "Leo Srole, "Social Integration and Certain Corollaries: my own work, I soon decided that it was neces- An Exploratory Study,"American Sociological Review sary to limit the use of that concept to its macro- 21 (1 956): 709-16.

tinction between them has important implications as unemployed such people as high school stu- for definition and measurement. If it seems that de- dents, the retired, the disabled, and homemakers. scription is simpler than explanation, you may be We might follow the Census convention further by surprised to learn that definitions are more prob- defining the labor force as "all persons 14 years of lematic for descriptive research than for explana- age and over who are employed, looking for work, tory research. Before we turn to other aspects of or waiting to be called back to a job from which measurement, you'li need a basic understanding of they have been laid off or furloughed." If a student, why this is so (we'll discuss this point more fully in homemaker, or retired person is not looking for Part 4). work, such a person would not be included in the It's easy to see the importance of clear and pre- labor force. Unemployed people, then, would be cise definitions for descriptive research. If we want those members of the labor force, as defined, who to describe and report the unemployment rate in a are not employed. city, our definition of "being unemployed" is obvi- But what does "looking for work" mean? Must a ously critical. That definition will depend on our person register with the state employment service definition of another term: the labor force. If it or go from door to door asking for employment? Or seems patently absurd to regard a three-year-old would it be sufficient to want a job or be open to an child as being unemployed, it is because such a offer of employment? Conventionally, "looking for child is not considered a member of the labor force. work" is defined operationally as saying yes in re- Thus, we might follow the U.S. Census Bureau's sponse to an interviewer's asking, "Have you been convention and exclude all people under age 14 looking for a job during the past seven days?" from the labor force. (Seven days is the period most often specified, but This convention alone, however, would not give for some research purposes it might make more us a satisfactory definition, because it would count sense to shorten or lengthen it.) 134 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMENT

by Patricia Fisher ipation in the planning process by citizens is im- Graduate School of Planning, University of Tennessee portant to the success of plan implementation. Citizen participation is an aid to planners' un- perationalization is one of those things derstanding of the real and perceived needs of 0that's easier said than done. It is quite a community, and such involvement by citizens simple to explain to someone the purpose and tends to enhance their cooperation with and importance of operational definitions for vari- support for planning efforts. Although many dif- ables, and even to describe how operationaliza- ferent conceptual definitions might be offered tion typically takes place. However, until you've by different planners, there would be little mis- tried to operationalize a rather complex variable, understanding over what is meant by citizen you may not appreciate some of the subtle participation. The name of the variable seems difficulties involved. Of considerable importance adequate. to the operationalization effort is the particular Howeven; if we asked different planners to name that you have chosen for a variable. Let's provide very simple operational measures for consider an example from the field of Urban citizen participation, we are likely to find a vari- Planning. ety among their responses that does generate A variable of intcrest to planners is citizen confusion. One planner might keep a tally of at- part~cipation.Planners are convinced that partic- tendance by private citizens at city commission

As you can see, the conclusion of a descriptive rate, this percentage would depend directly on the study about the unemployment rate depends di- definition of what is being measured-in this case, rectly on how each issue of definition is resolved. political conservatism. A different definition might Increasing the period during which people are result in the conclusion "Five percent of the student counted as looking for work would add more un- body are politically conservative." employed people to the labor force as defined, Ironically, definitions are less problematic in the thereby increasing the reported unemployment case of explanatory research. Let's suppose we're rate. If we follow another convention and speak of interested in explaining political conservatism. the civilian labor force and the civilian unemploy- Why are some people conservative and others not? ment rate, we are excluding military personnel; More specifically, let's suppose we're interested in that, too, increases the reported unemployment whether conservatism increases with age. What if rate, because military personnel would be em- you and I have 25 different operational definitions ployed-by definition. Thus the descriptive state- of conservative, and we can't agree on which defini- ment that the unemployment rate in a city is 3 per- tion is best? As we saw in the discussion of indi- cent, or 9 percent, or whatever it might be, depends cators, this is not necessarily an insurmountable directly on the operational definitions used. obstacle to our research. Suppose we found old This example is relatively clear because there people to be more conservative than young people are several accepted conventions relating to the la- in terms of all 25 definitions. Clearly, the exact defi- bor force and unemployment. Now, consider how nition would be of small consequence. Suppose we difficult it would be to get agreement about the found old people to be more conservative than definitions you would need in order to say, "Forty- young people by every reasonable definition of five percent of the students at this institution are conservatism we could think of. It wouldn't matter politically conservative." Like the unemployment what our definition was. We would conclude that OPERATIONALIZATION CHOICES 135

and other local government meetings; another The precise and exact naming of variables is might maintain a record of the different topics important in research. It is both essential to and addressed by private citizens at similar meet- a result of good operationalization. Variable ings; while a third might record the number of lo- names quite often evolve from an iterative pro- cal government meeting attendees, letters, and cess of forming a conceptual definition, then phone calls received by the mayor and other an operational definition, then renaming the public officials, and meetings held by special concept to better match what can or will be mea- interest groups during a particular time period. sured. This looping process continues (our ex- As skilled researchers, we can readily see that ample above illustrates only one iteration), re- each planner would be measuring (in a very sim- sulting in a gradual refinement of the variable plistic fashion) a different dimension of citizen name and its measurement until a reasonable fit participation: extent of citizen participation, is- is obtained. Sometimes the concept of the vari- sues prompting citizen participation, and form able that you end up with is a bit different from of citizen participation. Therefore, the original the original one that you started with, but at least naming of our variable, citizen participation, you are measuring what you are talking about, if which was quite satisfactory from a conceptual only because you are talking about what you are point of view, proved inadequate for purposes of measuring! operationalization.

old people are generally more conservative than and operationalization is the development of young people-even though we couldn't agree specific research procedures (operations) that will about exactly what conservative means. result in empirical observations representing those In practice, explanatory research seldom results concepts in the real world. in findings quite as unambiguous as this example As with the methods of data collection, social suggests; nonetheless, the general pattern is quite researchers have a variety of choices when opera- common in actual research. There are consistent tionalizing a concept. Although the several choices patterns of relationships in human social life that are intimately interconnected, I've separated them result in consistent research findings. However, for the sake of discussion. Realize, however, that such consistency does not appear in a descriptive operationalization does not proceed through a sys- situation. Changing definitions almost inevitably tematic checklist. result in different descriptive conclusions. The box "The Importance of Variable Names" explores Range of Variation this issue in connection with the variable citizen pal-ticipation. In operationalizing any concept, researchers must be clear about the range of variation that interests them. The question is, to what extent are we will- OPERATIONALIZATION CHOICES ing to combine attributes in fairly gross categories? Let's suppose you want to measure people's in- In discussing conceptualization, I frequently have comes in a study by collecting the information from referred to operationalization, for the two are inti- either records or interviews. The highest annual in- mately linked. To recap: Conceptualization is the comes people receive run into the millions of dol- refinement and specification of abstract concepts, lars, but not many people earn that much. Unless 136 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMENT you're studying the very rich, keeping track of ex- Similarly, people are not just more or less religious; tremely high categories probably won't add much some are positively antireligious. to your study. Depending on whom you study, you'll The point is not that you must measure the full probably want to establish a highest income cate- range of variation in every case. You should, how- gory with a much lower floor-maybe $1 00,000 or ever, consider whether you need to do so, given more. Although this decision will lead you to throw your particular research purpose. If the difference together people who earn a trillion dollars a year between not religious and antireligious isn't rele- with paupers earning a mere $1 00,000, they'll sur- vant to your research, forget it. Someone has vive it, and that mixing probably won't hurt your re- defined pragmatism as "any difference that makes search any, either. The same decision faces you at no difference is no difference." Be pragmatic. the other end of the income spectrum. In studies of Finally, decisions about the range of variation the general U.S.population, a bottom category of should be governed by the expected distribution $5,000 or less usually works fine. of attributes among the subjects of the study. In In studies of attitudes and orientations, the a study of college professors' attitudes toward question of range of variation has another dimen- the value of higher education, you could probably sion. Unless you're careful, you may end up mea- stop at no value and not worry about those who suring only half an attitude without really meaning might consider higher education dangerous to to. Here's an example of what I mean. students' health. (If you were studying students, Suppose you're interested in people's attitudes however. . . .) toward expanding the use of nuclear power gen- erators. If you reasonably guess or have experi- Variations between the Extremes enced that some people consider nuclear power the greatest thing since the wheel, whereas other Degree of precision is a second consideration in people have absolutely no interest in it, it makes operationalizing variables. What it boils down to is sense to ask people how much they favor expand- how fine you will make the distinctions among the ing the use of nuclear energy and to give them an- various possible attributes composing a given vari- swer categories ranging from "Favor it very much" able. Does it matter for your purposes whether a to "Don't favor it at all." person is 17 or 18 years old, or could you conduct This operationalization, however, conceals half your inquiry by throwing them together in a group the attitudinal spectrum regarding nuclear energy labeled 10 to 19 years old? Don't answer too Many people have feelings that go beyond simply quickly If you wanted to study rates of voter regis- not favoring it: They are, with greater or lesser de- tration and participation, you'd definitely want to grees of intensity, actively opposed to it. In this in- know whether the people you studied were old stance, there is considerable variation on the left enough to vote. In general, if you're going to mea- side of zero. Some oppose it a little, some quite sure age, you must look at the purpose and proce- a bit, and others a great deal. To measure the dures of your study and decide whether fine or full range of variation, then, you'd want to opera- gross differences in age are important to you. In a tionalize attitudes toward nuclear energy with a survey, you'll need to make these decisions in or- range from favoring it very much, through no feel- der to design an appropriate questionnaire. In the ings one way or the other, to opposing it very much. case of in-depth interviews, these decisions will This consideration applies to many of the vari- condition the extent to which you probe for details. ables that social researchers study. Virtually any The same thing applies to other variables. If you public issue involves both support and opposition, measure political affiliation, will it matter to your each in varying degrees. Political orientations inquiry whether a person is a conservative Demo- range from very liberal to very conservative, and crat rather than a liberal Democrat, or will know- depending on the people you're studying, you may ing just the party suffice? In measuring religious want to allow for radicals on one or both ends. affiliation, is it enough to know that a person is a OPERATIONALIZATION CHOICES 137

Protestant, or do you need to know the denomina- How certain are they that they would be will tion? Do you simply need to know whether or not ing to do what they say they would do? a person is married, or will it make a difference to know if he or she has never married or is sepa- The list could go on and on-how people feel rated, widowed, or divorced? about corruption in government has many dimen- There is, of course, no general answer to such sions. It's essential to be clear about which ones questions. The answers come out of the purpose are important in our inquiiy; otherwise, you may of a given study, or why we are making a partic- measure how people feel about corruption when ular measurement. I can give you a useful guide- you really wanted to know how much they think line, though. Whenever you're not sure how much there is, or vice versa. detail to pursue in a measurement, go after too Once you've determined how you're going to much rather than too little. When a subject in an collect your data (for example, survey, field re- in-depth interview volunteers that she is 37 years search) and decided on the relevant range of vari- old, record "37" in your notes, not "in her thirties." ation, the degree of precision needed between the When you're analyzing the data, you can always extremes of variation, and the specific dimensions combine precise attributes into more general cate- of the variables that interest you, you may have an- gories, but you can never separate any variations other choice: a mathematical-logical one. That is, you lumped together during observation and you may need to decide what level of measure- measurement. ment to use. To discuss this point, we need to take another look at attributes and their relationship to variables. A Note on Dimensions

We've already discussed dimensions as a charac- Defining Variables and Attributes teristic of concepts. When researchers get down to the business of creating operational measures of An attribute, you'll recall, is a characteristic or variables, they often discover-or worse, never quality of something. "Female" is an example. So is notice-that they haven't been exactly clear about "old" or "student." Variables, on the other hand, are which dimensions of a variable they're really inter- logical sets of attributes. Thus, gender is a variable ested in. Here's an example. composed of the attributes female and male. Let's suppose you're studying people's attitudes The conceptualization and operationalization toward government, and you want to include an processes can be seen as the specification of vari- examination of how people feel about corruption. ables and the attributes composing them. Thus, in Here are just a few of the dimensions you might the context of a study of unemployment, employ- examine: ment status is a variable having the attributes em- ployed and unemployed; the list of attributes could Do people think there is corruption in also be expanded to include the other possibilities government? discussed earlier, such as homemaker. How much corruption do they think there is? Every variable must have two important quali- How certain are they in their judgment of how ties. First, the attributes composing it should be much corruption there is? exhaustive. For the variable to have any utility in re- How do they feel about corruption in govern- search, we must be able to classify every observa- ment as a problem in society? tion in terms of one of the attributes composing the What do they think causes it? variable. We'll run into trouble if we conceptualize Do they think it's inevitable? the variable politicnl party njiliation in terms of What do they feel should be done about it? the attributes Republican and Democrat, because What are they willing to do personally to elimi- some of the people we set out to study will identify nate corruption in government? with the Green Party, the Reform Party, or some 138 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMENT other organization, and some (often a large per- cal party aflliation, birthplace, college majoc and centage) will tell us they have no party affiliation. hair color: Although the attributes composing each We could make the list of attributes exhaustive by of these variables-as male and female compose adding "other" and "no affiliation." Whatever we the variable gender-are distinct from one another do, we must be able to classify every observation. (and exhaust the possibilities of gender among At the same time, attributes composing a vari- people), they have no additional structures. Nomi- able must be mutually exclusive. That is, we must nal measures merely offer names or labels for be able to classify every observation in terms of characteristics. one and only one attribute. For example, we need Imagine a group of people characterized in to define "employed" and "unemployed" in such a terms of one such variable and physically grouped way that nobody can be both at the same time. by the applicable attributes. For example, say That means being able to classify the person who we've asked a large gathering of people to stand is working at a job but is also looking for work. (We together in groups according to the states in which might run across a fully employed mud wrestler they were born: all those born in Vermont in one who is looking for the glamour and excitement of group, those born in California in another, and so being a social researcher.) In this case, we might forth. The variable is place of birth; the attributes define the attributes so that employed takes prece- are born in Calfornia, born in Vermont, and so on. dence over unemployed, and anyone working at a All the people standing in a given group have at job is employed regardless of whether he or she is least one thing in common and differ from the looking for something better. people in all other groups in that same regard. Where the individual groups form, how close they are to one another, or how the groups are arranged Levels of Measurement in the room is irrelevant. All that matters is that all Attributes operationalized as mutually exclusive the members of a given group share the same state and exhaustive may be related in other ways as of birth and that each group has a different shared well. For example, the attributes composing vari- state of birth. All we can say about two people in ables may represent different levels of measure- terms of a nominal variable is that they are either ment. In this section, we'll examine four levels of the same or different. measurement: nominal, ordinal, interval, and ratio. Ordinal Measures Variables with attributes we Nominal Measures Variables whose attributes can logically rank-order are ordinal measures. have only the characteristics of exhaustiveness and The different attributes of ordinal variables repre- mutual exclusiveness are nominal measures. sent relatively more or less of the variable. Vari- Examples include gender, religious aBliation, politi- ables of this type include social class, conservatism, alienation, prejudice, and intellectual sophistication. In addition to saying whether two people are the same or different in terms of an ordinal variable, nominal measure A variable whose attr~buteshave only we can also say one is "more" than the other-that the characteristics of exhaustiveness and mutual exclusive- is, more conservative, more religious, older, and so ness. In other words, a level of measurement describing a forth. variable that has attributes that are merely different, as In the physical sciences, hardness is the most distinguished from ordinal, interval, or ratio measures. frequently cited example of an ordinal measure. Gender would be an example of a nominal measure. We can say that one material (for example, dia- ordinal measure A level of measurement describing a variable with attributes we can rank-order along some di- mond) is harder than another (say, glass) if the for- mension. An example would be socioeconomlcstatus as mer can scratch the latter and not vice versa. By composed of the attr~buteshigh, medium, low. attempting to scratch various materials with other OPERATIONALIZATION CHOICES 139 materials, we might eventually be able to arrange cause in both the Fahrenheit and Celsius scales, several materials in a row, ranging from the softest "zero" is arbitrary; that is, zero degrees does to the hardest. We could never say how hard a not really mean total lack of heat. Similarly, minus given material was in absolute terms; we could 30 degrees on either scale doesn't represent 30 de- only say how hard in relative terms-which mate- grees less than no heat. (In contrast, the Kelvin rials it is harder than and which softer than. scale is based on an absolute zero, which does Let's pursue the earlier example of grouping the mean a complete lack of heat.) people at a social gathering. This time imagine that About the only interval measures commonly we ask all the people who have graduated from used in social scientific research are constructed college to stand in one group, all those with only a measures such as standardized intelligence tests high school diploma to stand in another group, and that have been more or less accepted. The interval all those who have not graduated from high school separating IQ scores of 100 and 1 10 may be re- to stand in a third group. This manner of grouping garded as the same as the interval separating people satisfies the requirements for exhaustive- scores of 110 and 120, by virtue of the distribution ness and mutual exclusiveness discussed earlier. In of observed scores obtained by many thousands of addition, however, we might logically arrange the people who have taken the tests over the years. But three groups in terms of the relative amount of for- it would be incorrect to infer that someone with an mal education (the shared attribute) each had. 1Ye IQ of 150 is 50 percent more intelligent than some- might arrange the three groups in a row! ranging one with an IQ of 100. (A person who received a from most to least formal education. This arrange- score of 0 on a standard IQ test could not be re- ment would provide a physical representation of garded, strictly speaking, as having no intelligence, an ordinal measure. If we knew which groups two although we might feel he or she was unsuited to individuals were in, we could determine that one be a college professor or even a college student. had more, less, or the same formal education as But perhaps a dean. . . ?) the other. When comparing two people in terms of an in- Notice in this example that it doesn't matter how terval variable, we can say they are different from close or far apart the educational groups are from one another (nominal),and that one is more than one another. The college and high school groups another (ordinal). In addition, we can say "how might be 5 feet apart, and the less-than-high- much" more. school group 500 feet farther down the line. These actual distances don't have any meaning. The high Ratio Measures Most of the social scientific vari- school group, however, should be between the ables meeting the minimum requirements for in- less-than-high-school group and the college group, terval measures also meet the requirements for ra- or else the rank order will be incorrect. tio measures. In ratio measures, the attributes

Interval Measures For the attributes composing some variables, the actual distance separating those attributes does have meaning. Such vari- interval measure A level of measurement describing a ables are interval measures. For these, the logi- varlable whose attributes are rank-ordered and have equal cal distance between attributes can be expressed distances between adjacent attributes. The Fahrenheit tem- in meaningful standard intervals perature scale is an example of this, because the distance For example, in the Fahrenheit temperature between 17 and 18 is the same as that between 89 and 90. scale, the difference, or distance, between 80 de- ratio measure A level of measurement describ~nga vari- grees and 90 degrees is the same as that between able with attributes that have all the qualities of nominal, 40 degrees and 50 degrees. However, 80 degrees ordinal, and interval measures and in addition are based on Fahrenheit is not twice as hot as 40 degrees, be- a "true zero" point. Age is an example of a ratio measure. 140 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMENT composing a variable, besides having all the struc- you won't be invited to many parties), I should tural characteristics mentioned previously, are draw your attention to some of the practical impli- based on a true zero point. The Kelvin temperature cations of the differences that have been distin- scale is one such measure. Examples from social guished. These implications appear primarily in the research include age, length of residence in a given analysis of data (discussed in Part 4),but you need place, number of organizations belonged to, num- to anticipate such implications when you're struc- ber of times attending religious services during a turing any research project. particular period, number of times married, and Certain quantitative analysis techniques require number of Arab friends. variables that meet certain minimum levels of Returning to the illustration of methodological measurement. To the extent that the variables to party games, we might ask a gathering of people to be examined in a research project are limited to a group themselves by age. All the one-year-olds particular level of measurement-say, ordinal- would stand (or sit or lie) together, the two-year- you should plan your analytical techniques accord- olds together, the three-year-olds, and so forth. ingly. More precisely, you should anticipate draw- The fact that members of a single group share the ing research conclusions appropriate to the levels same age and that each different group has a dif- of measurement used in your variables. For ex- ferent shared age satisfies the minimum require- ample, you might reasonably plan to determine ments for a nominal measure. Arranging the sev- and report the mean age of a population under eral groups in a line from youngest to oldest meets study (add up all the individual ages and divide by the additional requirements of an ordinal measure the number of people), but you should not plan to and lets us determine if one person is older than, report the mean religious affiliation, because that is younger than, or the same age as another. If we a nominal variable, and the mean requires ratio- space the groups equally far apart, we satisfy the level data. (You could report the modal-the most additional requirements of an interval measure common-religious affiliation.) and will be able to say how much older one person At the same time, you can treat some variables is than another. Finally, because one of the attri- as representing different levels of measurement. butes included in age represents a true zero (ba- Ratio measures are the highest level, descend- bies carried by women about to give birth), the ing through interval and ordinal to nominal, the phalanx of hapless party goers also meets the re- lowest level of measurement. A variable repre- quirements of a ratio measure, permitting us to say senting a higher level of measurement-say, that one person is twice as old as another. (Re- ratio-can also be treated as representing a lower member this in case you're asked about it in a level of measurement-say, ordinal. Recall, for ex- workbook assignment ) Another example of a ratio ample, that age is a ratio measure. If you wished to measure is income, which extends from an ab- examine only the relationship between age and solute zero to approximately infinity, if you happen some ordinal-level variable-say, self-perceived to be the founder of Microsoft. religiosity: high, medium, and low-you might Comparing two people in terms of a ratio vari- choose to treat age as an ordinal-level variable able, then, allows us to determine (I) that they are as well. You might characterize the subjects of different (or the same),(2) that one is more than the your study as being young, middle-aged, and old, other, (3)how much they differ, and (4) the ratio of specifying what age range determined each of one to another. Figure 5- 1 summarizes this discus- these groupings. Finally, age might be used as a sion by presenting a graphic illustration of the four nominal-level variable for certain research pur- levels of measurement. poses. People might be grouped as being born dur- ing the depression of the 1930s or not. Another Implications of Levels of Measurement Be- nominal measurement, based on birth date rather cause it's unlikely that you'll undertake the physical than just age, would be the grouping of people by grouping of people just described (try it once, and astrological signs. OPERATIONALIZATION CHOICES 141

Nominal Measure Example: Gender

Female Male

Ordinal Measure Example: Religiosity "How important is religion to you?"

Low High

Interval Measure Example: IQ

Ratio Measure Example: Income

FIGURE 5-1 Levels of Measurement

The level of measurement you'll seek, then, is You don't necessarily need to measure variables determined by the analytical uses you've planned at their highest level of measurement, however. If for a given variable, as you keep in mind that some you're sure to have no need for ages of people at variables are inherently limited to a certain level. If higher than the ordinal level of measurement, you a variable is to be used in a variety of ways, requir- may simply ask people to indicate their age range, ing different levels of measurement, the study such as 20 to 29, 30 to 39, and so forth. In a study should be designed to achieve the highest level re- of the wealth of corporations, rather than seek quired. For example, if the subjects in a study are more precise information, you may use Dun & asked their exact ages, they can later be organized Bradstreet ratings to rank corporations. Whenever into ordinal or nominal groupings. your research purposes are not altogether clear, 142 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMENT however, seek the highest level of measurement Single or Multiple Indicators possible. Again, although ratio measures can later be reduced to ordinal ones, you cannot convert With so many alternatives for operationalizing so- an ordinal measure to a ratio one. More generally, cial research variables, you may find yourself wor- you cannot convert a lower-level measure to a rying about making the right choices. To counter higher-level one. That is a one-way street worth this feeling, let me add a momentary dash of cer- remembering. tainty and stability. Typically a research project will tap variables at Many social research variables have fairly obvi- different levels of measurement. For example, ous, straightforward measures. No matter how you William and Denise Bielby (1999) set out to exam- cut it, gender usually turns out to be a matter of ine the world of film and television, using a nomo- male or female: a nominal-level variable that can thetic, longitudinal approach (take a moment to re- be measured by a single observation-either look- mind yourself what that means). In what they ing (well, not always) or asking a question (usu- referred to as the "culture industry," the authors ally). In a study involving the size of families, you'll found that reputution (an ordinal variable) is the want to think about adopted and foster children as best predictor of screenwriters' future productivity. well as blended families, but it's usually pretty easy More interestingly, they found that screenwriters to find out how many children a family has. For who were represented by "core" (or elite) agencies most research purposes, the resident population of were far more likely not only to find jobs (a nomi- a country is the resident population of that coun- nal variable) but also to find jobs that paid more (a try-you can find the number in an almanac. A ratio variable). In other words, the researchers great many variables, then, have obvious single in- found that an agency's reputation (ordinal) was a dicators. If you can get one piece of information, key independent variable for predicting a screen- you have what you need. writer's success. The researchers also found that Sometimes, however, there is no single indica- being older (ratio), being female (nominal), be- tor that will give you the measure of a chosen vari- longing to an ethnic minority (nominal), and hav- able. As discussed earlier in this chapter, many ing more years of experience (ratio) were disad- concepts are subject to varying interpretations- vantageous for a screenwriter. On the other hand, each with several possible indicators. In these higher earnjngs from previous years (measured in cases, you'll want to make several observations for ordinal categories) led to more success in the fu- a given variable. You can then combine the several ture. In the researchers' terms, "success breeds pieces of information you've collected to create a success" (Bielby and Bielbjr 1999:80). composite measurement of the variable in ques- tion. Chapter 6 is devoted to ways of doing that, so here let's look at just one simple illustration. APPLYING THE RESULTS Consider the concept "college performance." All of us have noticed that some students per- form well in college courses and others do not. In ay j~ouwant to be a Holl~woodscreen- studying these differences, we might ask what S writer How might you use the results of characteristics and experiences are related to high the Bielby and Bielby (1999) study to en- levels of performance (many researchers have hance your career? Say you didn't do so tvell done just that). How should we measure overall and instead started a school for screenwrit- performance? Each grade in any single course is a ers. How could the results of the study be potential indicator of college performance, but it used to plan courses? Finally, how might the also may not typify the student's general perfor- results be useful to you if you were a social mance. The solution to this problem is so firmly es- activist committed to fighting discrimination tablished that it is, of course, obvious: the grade in the "culture industry"? point average (GPA). We assign numerical scores to each letter grade, total the points earned by OPERATIONALIZATION CHOICES 143 a given student, and divide by the number of calculate the percentage of men and the courses taken to obtain a composite measure. (If percentage of women who help. the courses vary in number of credits, we adjust the c. Select a sample of people and do a sur- point values accordingly.) It's often appropriate vey in which you ask them what organi- to create such composite measures in social zations they belong to. Calculate whether research. women or men are more likely to belong to those that seem to reflect compassion- ate feelings. To take account of men who Some Illustrations of belong to more organizations than do Operationalization Choices women in general-or vice versa-do To bring together all the operationalization choices this: For each person you study, calculate available to the social researcher and to show the the percentage of his or her organiza- potential in those possibilities, let's look at some tional memberships that reflect compas- of the distinct ways you might address various sion. See if men or women have a higher research problems. The alternative ways of op- average percentage. erationalizing the variables in each case should 2. Are sociology students or accounting demonstrate the opportunities that social research students better informed about world can present to our ingenuity and imaginations. affairs? To simplify matters, I have not attempted to de- a. Prepare a short quiz on world affairs and scribe all the research conditions that would make arrange to administer it to the students in one alternative superior to the others, though in a sociology class and in an accounting a given situation they would not all be equally class at a comparable level. If you want appropriate. to compare sociology and accounting majors, be sure to ask students what they 1. Are women more compassionate than men? are majoring in. a. Select a group of subjects for study, with b. Get the instructor of a course in world equal numbers of men and women. Pre- affairs to give you the average grades of sent them with hypothetical situations sociology and accounting students in the that involve someone's being in trouble. course. Ask them what they would do if they c. Take a petition to sociology and account- were confronted with that situation. What ing classes that urges that "the United would they do, for example, if they came Nations headquarters be moved to New across a small child who was lost and York City." Keep a count of how many crying for his or her parents? Consider in each class sign the petition and how any answer that involves helping or com- many inform you that the UN headquar- forting the child as an indicator of com- ters is already located in New York City. passion. See whether men or women are 3. Do people consider New York or California more likely to indicate they would be the better place to live? compassionate. a. Consulting the Statistical Abstroct of'the b. Set up an experiment in which you pay United States or a similar publication, a small child to pretend that he or she check the migration rates into and out is lost. Put the child to work on a busy of each state. See if you can find the sidewalk and observe whether men or numbers moving directly from New York women are more likely to offer assis- to California and vice versa. tance. Also be sure to count the total b. The national polling companies-Gallup, number of men and women who walk Harris, Roper, and so forth-often ask by, because there may be more of one people what they consider the best than the other. If that's the case, simply state to live in. Look up some recent 144 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMENT

results in the library or through your CRITERIA OF local newspaper. MEASUREMENT QUALITY c. Compare suicide rates in the two states. 4. Who are the most popular instructors on This chapter has come some distance. It began your campus-those in the social sciences, with the bald assertion that social scientists can the natural sciences, or the humanities? measure anything that exists. Then we discovered a. If your school has formal student evalua- that most of the things we might want to measure tions of instructors, review some recent and study don't really exist. Next we learned that results and compute the average ratings it's possible to measure them anyway. Now we of each group. conclude the chapter with a discussion of some of b. Begin visiting the introductory courses the yardsticks against which we judge our relative given in each group of disciplines and success or failure in measuring things-even measure the attendance rate of each things that don't exist. class. c. In December, select a group of faculty in each of the three divisions and ask them Precision and Accuracy to keep a record of the numbers of holi- day greeting cards and presents they To begin, measurements can be made with varying receive from admiring students. See degrees of precision. As we saw in the discussion who wins. of operationalization, precision concerns the fine- ness of distinctions made between the attributes The point of these examples is not neces- that compose a variable. The description of a sarily to suggest respectable research projects woman as "43 years old" is more precise than but to illustrate the many ways variables can be "in her forties." Saylng a street-corner gang was operationalized. formed in the summer of 1996 is more precise than saying "during the 1990s." As a general rule, precise measurements are su- Operationalization perior to imprecise ones, as common sense sug- Goes On and On gests. There are no conditions under which impre- Although I've discussed conceptualization and cise measurements are intrinsically superior to operationalization as activities that precede data precise ones. Even so, exact precision is not always collection and analysis-for example, you must necessary or desirable. If knowing that a woman design questionnaire items before you send out is in her forties satisfies your research require- a questionnaire-these two processes continue ments, then any additional effort invested in learn- throughout any research project, even if the data ing her precise age is wasted. The operationaliza- have been collected in a structured mass survey. As tion of concepts, then, must be guided partly by an we've seen, in less-structured methods such as understanding of the degree of precision required. field research, the identification and specification of If your needs are not clear, be more precise rather relevant concepts is inseparable from the ongoing than less. process of observation. Don't confuse precision with accuracy, how- As a researcher, always be open to reexamining ever. Describing someone as "born in New En- your concepts and definitions. The ultimate pur- gland" is less precise than "born in Stowe, Ver- pose of social research is to clarify the nature of so- mont"-but suppose the person in question was cial life. The validity and utility of what you learn actually born in Boston. The less-precise descrip- in this regard doesn't depend on when you first tion, in this instance, is more accurate, a better figured out how to look at things any more than it reflection of the real world. matters whether you got the idea from a learned Precision and accuracy are obviously important textbook, a dream, or your brother-in-law. qualities in research measurement, and they prob- CRITERIA OF MEASUREMENT QUALITY 145 ably need no further explanation. When social re- they would prefer their current arrangement or the searchers construct and evaluate measurements, other one being studied. By comparing what we however, they pay special attention to two techni- observed in the different factories, we might reach cal considerations: reliability and validity. a conclusion about which assembly process pro- duces the higher morale. Now let's look at some reliability problems in- Reliability herent in this method. First, how you and I are feel- In the abstract, reliability is a matter of whether a ing when we do the observing will likely color what particular technique, applied repeatedly to the we see. We may misinterpret what we observe. We same object, yields the same result each time. Let's may see workers kidding each other but think say you want to know how much I weigh. (No, I they're having an argument. We may catch them don't know why.) As one technique, say you ask on an off day. If we were to observe the same group two different people to estimate my weight. If the of workers several days in a row, we might arrive at first person estimates 150 pounds and the other different evaluations on each day. If several ob- estimates 300, we have to conclude the technique servers evaluated the same behavior, on the other of having people estimate my weight isn't very hand, they similarly might arrive at different con- reliable. clusions about the workers' morale. Suppose, as an alternative, that you use a bath- Here's another strategy for assessing morale. room scale as your measurement technique. I step Suppose we check the company records to see on the scale twice, and you note the result each how many grievances have been filed with the time. The scale has presumably reported the same union during some fixed period. Presumably this weight both times, indicating that the scale pro- would be an indicator of morale: the more griev- vides a more reliable technique for measuring a ances, the lower the morale. This measurement person's weight than asking people to estimate strategy would appear to be more reliable: Count- it does. ing up the grievances over and over, we should Reliability, however, does not ensure accuracy keep arriving at the same number. any more than precision does. Suppose I've set my If you find yourself thinking that the number of bathroom scale to shave five pounds off my weight grievances doesn't necessarily measure morale, just to make me feel better. Although you would you're worrying about validity, not reliability. We'll (reliably)report the same weight for me each time, discuss validity in a moment. The point for now is you would always be wrong. This new element, that the last method is more like my bathroom called bias, is discussed in Chapter 7. For now, just scale-it gives consistent results. be warned that reliability does not ensure accuracy. In social research, reliability problems crop up Let's suppose we're interested in studying mo- in many forms. Reliability is a concern every time a rale among factory workers in two different kinds single observer is the source of data, because we of factories. In one set of factories, workers have have no certain guard against the impact of that specialized jobs, reflecting an extreme division of observer's subjectivity. We can't tell for sure how labor. Each worker contributes a tiny part to the much of what's reported originated in the situation overall process performed on a long assembly line. observed and how much came from the observer. In the other set of factories, each worker performs many tasks, and small teams of workers complete the whole process. reliability That qual~tyof measurement method that sug- How should we measure morale? Following gests that the same data would have been collected each one strategy, we could observe the workers in each time In repeated observations of the same phenomenon. In factory, noticing such things as whether they joke the context of a survey, we would expect that the question "Did you attend church last week?" would have higher reli- with one another, whether they smile and laugh a ability than the question "About how many times have you lot, and so forth. We could ask them how they like attended church in your life?" This is not to be confused their work and even ask them whether they think with validity. 146 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMENT

Subjectivity is a problem not only with single then you should expect the same response both observers, however. Survey researchers have times. If answers vary, the measurement method known for a long time that different interviewers, may, to the extent of that variation, be unreliable. because of their own attitudes and demeanors, Here's an illustration. get different answers from respondents. Or, if we In their research on Health Hazard Appraisal were to conduct a study of newspapers' editorial (HHA),a part of preventive medicine, Jeffrey Sacks, positions on some public issue, we might create a W. Mark Krushat, and Jeffrey Newman (1980) team of coders to take on the job of reading hun- wanted to determine the risks associated with var- dreds of editorials and classifying them in terms of ious background and lifestyle factors, making it their position on the issue. Unfortunately, different possible for physicians to counsel their patients ap- coders will code the same editorial differently. Or propriately. By knowing patients' life situations, we might want to classify a few hundred specific physicians could advise them on their potential for occupations in terms of some standard coding survival and on how to improve it. This purpose, of scheme, say a set of categories created by the De- course, depended heavily on the accuracy of the in- partment of Labor or by the Census Bureau. You formation gathered about each subject in the study. and I would not place all those occupations in the To test the reliability of their information, Sacks same categories. and his colleagues had all 207 subjects complete a Each of these examples illustrates problems of baseline questionnaire that asked about their char- reliability. Similar problems arise whenever cve ask acteristics and behavior. Three months later, a fol- people to give us information about themselves. low-up questionnaire asked the same subjects for Sometimes we ask questions that people don't the same information, and the results of the two know the answers to: How many times, if any, have surveys were compared. Overall, only 15 percent you been to religious services this year? Some- of the subjects reported the same information in times we ask people about things they consider both studies. totally irrelevant: Are you satisfied with China's Sacks and his colleagues reported the current relationship with Albania? In such cases, following: people will answer differently at different times be- Almost 10 percent of subjects reported a differ- cause they're making up answers as they go. ent height at follow-up examination. Parental Sometimes we explore issues so complicated that age was changed by over one in three subjects. a person who had a clear opinion in the matter One parent reportedly aged 20 chronologic might arrive at a different interpretation of the years in three months. One in five ex-smokers question when asked a second time. and ex-drinkers have apparent difficulty in reli- So how do you create reliable measures? If your ably recalling their previous consumption pat- research design calls for asking people for infor- tern. - (1980: 730) mation, you can be careful to ask only about things the respondents are likely to know the answer to. Some subjects had erased all traces of previ- Ask about things relevant to them, and be clear in ously reported heart murmurs, diabetes, emphy- what you're asking. Of course, these techniques sema, arrest records, and thoughts of suicide. One don't solve every possible reliability problem. For- subject's mother, deceased in the first question- tunately, social researchers have developed several naire, was apparently alive and well in time for the techniques for cross-checking the reliability of the second. One subject had one ovary missing in the measures they devise. first study but present in the second. In another case, an ovary present in the first study was miss- Test-Retest Method Sometimes it's appropriate ing in the second study-and had been for ten to make the same measurement more than once, a years! One subject was reportedly 55 years old technique called the test-retest method. If you don't in the first study and 50 years old three months expect the information being sought to change, later. (You have to wonder whether the physician- CRITERIA OF MEASUREMENT QUALITY 147 counselors could ever have nearly the impact on Replication works in other situations also. If their patients that their patients' memories had.) you're worried that newspaper editorials or occu- Thus, test-retest revealed that this data-collection pations may not be classified reliably, you could method was not especially reliable. have each independently coded by several coders. Those cases that are classified inconsistently can Split-Half Method As a general rule, it's always then be evaluated more carefully and resolved. good to make more than one measurement of any subtle or complex social concept, such as prej- udice, alienation, or social class. This procedure APPLYING THE RESULTS lays the groundwork for another check on reliabil- ity. Let's say you've created a questionnaire that eplication in measurement is exactly the contains ten items you believe measure prejudice Rissue that was raised in Florida during against women. Using the split-half technique, you the 2000 Presidential election. Specifically, would randomly assign those ten items to two sets given the thousands of ballots rejected by of five. As we saw in the discussion of interchange- the vote-counting machines, many people ability of indicators, each set should provide a good questioned the reliability of this method of measure of prejudice against women, and the two measuring votes. Had the election been a sets should classify respondents the same way. If survey of voting intentions, the researchers the two sets of items classify people differently, you would have reviewed the ballots rejected by most likely have a problem of reliability in your the machine and sought to make judgments measure of the variable. regarding those intentions. Notice that deci- sions about hanging and pregnant "chads" Using Established Measures Another way to would have concerned measurement proce- help ensure reliability in getting information from dures. Much of the debate hinged on how people is to use measures that have proven their much each type of Chad reflected voter in- reliability in previous research. If you want to mea- tent: What does a bump in a chad "really" sure anomia, for example, you might want to fol- mean? lliithout agreement on what really low Srole's lead. constituted a vote, researchers would have The heavy use of measures, though, does simply scored ballots in terms which candi- not guarantee their reliability. For example, the date was apparently selected and why-that Scholastic Assessment Tests (SATs) and the Min- is, the basis on which the researcher's deci- nesota Multiphasic Personality Inventory (MhIPI) sion was made (e.g., "pregnant chad"). Of have been accepted as established standards in course, there would first have to be clear op- their respective domains for decades. In recent erational definitions of what "swinging," years, though, they've needed fundamental over- "pregnant," and other sorts of chads were. It hauling to reflect changes in society, eliminating would certainly have been possible to deter- outdated topics and gender bias in wording. mine the results in terms of the standards used-that is, how many of each type of Reliability of Research Workers As we've seen, vote for each candidate. measurement unreliability can also be generated by research workers: interviewers and coders, for example. There are several ways to check on relia- bility in such cases. To guard against interviewer Finally, clarity, specificity, training, and practice unreliability, it is common practice in surveys to can prevent a great deal of unreliability and grief. have a supervisor call a subsample of the respon- If you and I spent some time reaching a clear dents on the telephone and verify selected pieces agreement on how to evaluate editorial posi- of information. tions on an issue-discussing various positions 148 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMENT and reading through several together-we could with our common agreements and our individual probably do a good job of classifyjng them in the mental images concerning a particular concept. same way independently. For example, you and I might quarrel about the ad- The reliability of measurements is a funda- equacy of measuring worker morale by counting mental issue in social research, and we'll return the number of grievances filed with the union. Still, to it more than once in the chapters ahead. For we'd surely agree that the number of grievances now, however, let's recall that even total reliability has something to do with morale. That is, the mea- doesn't ensure that our measures measure what sure is valid "on its face," whether or not it's ade- we think they measure. Now let's plunge into the quate. If I were to suggest that we measure morale question of validity. by finding out how many books the workers took out of the library during their off-duty hours, you'd undoubtedly raise a more serious objection: That Validity measure wouldn't have much face validity. In conventional usage, validity refers to the extent Second, I've already pointed to many of the to which an empirical measure adequately reflects more formally established agreements that define the real meaning of the concept under considera- some concepts. The Census Bureau, for example, tion. Whoops! I've already committed us to the has created operational definitions of such con- view that concepts don't have real meanings. How cepts as family, household, and employment status can we ever say whether a particular measure that seem to have a workable validity in most stud- adequately reflects the concept's meaning, then? ies using these concepts. Ultimately, of course, we can't. At the same time, as Three additional types of validity also specify we've already seen, all of social life, including particular ways of testing the validity of measures. social research, operates on agreements about The first, criterion-related validity, sometimes the terms we use and the concepts they repre- called predictive validity, is based on some external sent. There are several criteria of success in mak- criterion. For example, the validity of College Board ing measurements that are appropriate to these exams is shown in their ability to predict students' agreed-on meanings of concepts. success in college. The validity of a written driver's First, there's something called face validity. test is determined, in this sense, by the relationship Particular empirical measures may or may not jibe between the scores people get on the test and their subsequent driving records In these examples, college success and driving ability are the criteria. validity A term describing a measure that accurately To test your understanding of criterion-related reflects the concept it is intended to measure. For ex- validity, see whether you can think of behaviors ample, your 1Q would seem a more valid measure of your that mjght be used to validate each of the followjng ~ntelligencethan would the number of hours you spend in attitudes: the library. Though the ultimate validity of a measure can never be proven, we may agree to its relative validity on Is very religious the basis of face validity, criterion-related validity, content Supports equality of men and women validity, construct validity, internal validation, and external Supports far-right militia groups validation. This must not be confused with reliability. Is concerned about the environment face validity That quality of an indicator that makes it seem a reasonable measure of some var~able.That the fre- Some possible validators would be, respec- quency of church attendance is some indication of a per- tively, attends religious services, votes for son's religiosity seems to make sense without a lot of ex- planation. It has face validity. candidates, belongs to the NRA, and belongs to the Sierra Club. Sometimes it's difficult to find behav- criterion-related validity The degree to which a mea- sure relates to some external criterion, Also called Dredic. ioral criteria that can be taken to validate measures tive validity. as directly as in such examples. In those instances, CRITERIA OF MEASUREMENT QUALITY 149

I I Reliable but not valid Valid but not reliable Valid and reliable

FIGURE 5-2 An Analogy to Validity and Reliability however, we can often approximate such criteria maritalfdelity) represents in a given situation. It's by applying a different test. We can consider how less important to distinguish the two types of va- the variable in question ought, theoretically, to re- lidity tests than to understand the logic of valida- late to other variables. Construct validity is based tion that they have in common: If we have been on the logical relationships among variables. successful in measuring some variable, then our Suppose, for example, that you want to study measures should relate in some logical way to the sources and consequences of marital satisfac- other measures. tion. As part of your research, you develop a mea- Finally, content validity refers to how much a sure of marital satisfaction, and you want to assess measure covers the range of meanings included its validity. within a concept. For example, a test of mathemat- In addition to developing your measure, you'll ical ability cannot be limited to addition alone but have developed certain theoretical expectations also needs to cover subtraction, multiplication, di- about the way the variable rnarital satisfaction re- vision, and so forth. Or, if we're measuring preju- lates to other variables. For example,you might rea- dice, do our measurements reflect all types of prej- sonably conclude that satisfied husbands and wives udice, including prejudice against racial and ethnic will be less likely than dissatisfied ones to cheat on groups, religious minorities, women, the elderly, their spouses. If your measure relates to marital and so on? fidelity in the expected fashion, that constitutes evi- Figure 5-2 presents a graphic portrayal of the dence of your measure's construct validity If difference between validity and reliability. If you satisfied marriage partners are as likely to cheat on think of measurement as analogous to repeatedly their spouses as are the dissatisfied ones, however, shooting at the bull's-eye on a target, you'll see that that would challenge the validity of your measure. reliability looks like a "tight pattern," regardless of Tests of construct validity, then, can offer a where the shots hit, because reliability is a function weight of evidence that your measure either does or does not tap the quality you want it to measure, but this evidence is not definitive proof. Although 1 construct validity The degree to whch a measure relates have suggested that tests of construct validity are to other var~ablesas expected wlthln a system of theoretical less compelling than tests of criterion validitj~,there relatlonsh~ps. is room for disagreement about which kind of test content validity The degree to whlch a measure covers a particular comparison variable (driving record, the range of meanlngs ~ncludedwlthln a concept 150 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMENT

domly distributed around the target; the failure of an social scientists measure religiosity validity is systematically off the mark. Notice that Cand determine its causes? As you've neither an unreliable nor an invalid measure is seen, making descriptive statements about likely to be very useful. religiosity is harder than making explanatory ones. Any assertion about who is or is not re- Who Decides What's Valid? ligious depends directly on the definitions used. By one definition, nearly all of the pop- Our discussion of validity began with a reminder ulation would be deemed religious; by an- that we depend on agreements to determine other definition, few would be so designated. what's real, and we've just seen some of the ways As the discussion of the interchangeabil- social scientists can agree among themselves that ity of indicators suggested, however, we can they have made valid measurements. There is yet be more confident and definitive in speaking another way of looking at validity. about the causes of religiosity. For example, Social researchers sometimes criticize them- it is often reported that U.S. women are more selves and one another for implicitly assuming religious than U.S. men. This assertion is they are somewhat superior to those they study. based on the observation that women are For example, researchers often seek to uncover more religious than men on virtually all indi- motivations that the social actors themselves are cators of religiosity: attendance at religious unaware of. You think you bought that new Burpo- services, prayer, beliefs, and so forth. So, Blasto because of its high performance and good even though we may likely disagree on looks, but we know you're really trying to achieve which of these indicators is the best or truest a higher social status. measure of what we mean by the term rell- This implicit sense of superiority would fit com- giosity, women would be more religious than fortably with a totally positivistic approach (the bi- men regardless of the indicator chosen. ologist feels superior to the frog on the lab table), but it clashes with the more humanistic and typi- cally qualitative approach taken by many social scientists. We'll explore this issue more deeply in Tension between Chapter 10. Reliability and Validity In seeking to understand the way ordinary people make sense of their worlds, ethnomethod- Clearly, we want our measures to be both reliable ologists have urged all social scientists to pay more and valid. Often, however, a tension arises be- respect to these natural social processes of con- tween the criteria of reliability and validity, forcing ceptualization and shared meaning. At the very a trade-off between the two. least, behavior that may seem irrational from the Recall the example of measuring morale in dif- scientist's paradigm may make logical sense when ferent factories. The strategy of immersing yourself viewed from the actor's point of view. in the day-to-day routine of the assembly line, ob- Ultimately, social researchers should look to serving what goes on, and talking to the workers both colleagues and subjects as sources of agree- would seem to provide a more valid measure of mo- ment on the most useful meanings and measure- rale than would counting grievances. It just seems ments of the concepts they study. Sometimes one obvious that we'd get a clearer sense ofwhether the source will be more useful, sometimes the other. morale was high or low using this first method. But neither should be dismissed. As I pointed out earlier, however, the counting MAIN POINTS 151 strategy would be more reliable. This situation To some extent, this dilemma explains the per- reflects a more general strain in research measure- sistence of two quite different approaches to so- ment. Most of the really interesting concepts we cial research: quantitative, nomothetic, structured want to study have many subtle nuances, and it's techniques such as surveys and experiments on hard to specify precisely what we mean by them. the one hand and qualitative, idiographic methods Researchers sometimes speak of such concepts as such as field research and historical studies on the having a "richness of meaning." Although scores of other. In the simplest generalization, the former books and articles have been written on the topic methods tend to be more reliable, the latter more of anomie/anomia, for example, they still haven't valid. exhausted its meaning. By being forewarned, you'll be effectively fore- Very often, then, specifying reliable operational armed against this persistent and inevitable di- definitions and measurements seems to rob con- lemma. If there is no clear agreement on how to cepts of their richness of meaning. Positive morale measure a concept, measure it several different is much more than a lack of grievances filed with ways. If the concept has several dimensions, mea- the union; anomia is much more than what is mea- sure them all. Above all, know that the concept sured by the five items created by Leo Srole. Yet, does not have any meaning other than what you the more variation and richness we allow for a and I give it. The only justification for giving any concept, the more opportunity there is for dis- concept a particular meaning is utility. Measure agreement on how it applies to a particular situa- concepts in ways that help us understand the world tion, thus reducing reliability. around us.

Main Points degree of precision needed varies with the type and purpose of a study. u Conceptions are mental images we use as a Operationalization is an extension of concep- summary devices for bringing together obser- tualization that specifies the exact procedures vations and experiences that seem to have that will be used to measure the attributes of something in common. We use terms or labels variables. to reference these conceptions. u Operationalization involves a series of interre- J Concepts are constructs; they represent the lated choices: specifying the range of variation agreed-on meanings we assign to terms. Our that is appropriate for the purposes of a study, concepts don't exist in the real world, so they determining how precisely to measure vari- can't be measured directly, but we can mea- ables, accounting for relevant dimensions of sure the things that our concepts summarize. variables, clearly defining the attributes of vari-

_I Conceptualization is the process of specifying ables and their relationships, and deciding on observations and measurements that give an appropriate level of measurement.

concepts definite meaning for the purposes of 3 Researchers must choose from four types of a research study. measures that capture increasing amounts of o Conceptualization includes specifying the indi- information: nominal, ordinal, interval, and ra- cators of a concept and describing its dimen- tio. The most appropriate level depends on the sions. Operational definitions specify how vari- purpose of the measurement. ables relevant to a concept will be measured. o A given variable can sometimes be measured o Precise definitions are even more important in at different levels. When in doubt, researchers descriptive than in explanatory studies. The should use the highest level of measurement 152 CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMENT

appropriate to that variable so they can cap- 2. If you wanted to compare two societies on ture the greatest amount of information. anomie, or normlessness, what indicators

7 Operationalization begins in the design phase of might you look at? For example, what statisti- a study and continues through all phases of the cal indicators might you examine? What non- research project, including the analysis of data. statistical, qualitative indicators? 3. What level of measurement-nominal, ordinal, J Criteria of the quality of measures include pre- cision, accuracy, reliability, and validity. interval, or ratio-describes each of the follow- ing variables? i Whereas reliability means getting consistent a. Race (white, African American, Asian, and results from the same measure, validity refers so on) to getting results that accurately reflect the b. Order of finish in a race (first, second, third, concept being measured. and so on) i Researchers can test or improve the reliability c. Number of children in families of measures through the test-retest method, d. Populations of nations the split-half method, the use of established e. Attitudes toward nuclear energy (strongly measures, and the examination of work per- approve, approve, disapprove, strongly formed by research workers disapprove) 7 The yardsticks for assessing a measure's valid- f. Region of birth (Northeast, Midwest, and ity include face validity, criterion-related valid- so on) ity, construct validity, and content validity. g. Political orientation (very liberal, some- what liberal, somewhat conservative, very 7 Creating specific, reliable measures often seems to diminish the richness of meaning conservative) our general concepts have. This problem is 4. Suppose you've been contracted by Wads- inevitable. The best solution is to use several worth Publishing Company to interview a different measures, tapping the various as- group of students and evaluate their level of pects of a concept. satisfaction with this textbook. How would you measure satisfaction in this case? Kev Terms Additional Readings conceptualization reliability indicator validity Bohrnstedt, George W. 1983. "Measurement." Pp. 70-12 1 dimension face validity in Handbook ofsurvey Research, edited by Peter H. nominal measures criterion-related validity Rossi, James D. Wright, and Andy B. Anderson. New ord~nalmeasures construct validity York: Academic Press. This essay offers the logical interval measures content validity and statistical grounding of reliability and validity in ratio measures measurement. Carmines, Edward G., and Richard A. Zeller. 1979. Relia- Review Ouestions bility and Validity Assessment. Thousand Oaks, CA: Sage. In this chapter, we've examined the basic logic of validity and reliability in social science measure- 1. Pick a social science concept such as liberalism ment. Carmines and Zeller explore these issues in or alienation, then specify that concept so that more detail and examine some of the ways to calcu- it could be studied in a research project. How late reliability mathematically. would you specify the indicators you would use Gould, Julius, and William Kolb. 1964.A Dictionary oJ as well as the dimensions you wish to include the Social Sciences. New York: Free Press. A primary in and exclude from your conceptualization? reference to the social scientific agreements on MEDIA RESOURCES 153

various concepts. Although the terms used by so- conceptualization to observation on the one hand cial scientists do not have ultimatelj~"true" inean- and to generalization on the other. ings, this reference book lays out the meanings social scientists have in mind when they use those terms. Multimedia Resources Grimes, blichael D. 199 1. Class in Tlzfentieih-Century Ana-icon Sociology: An Arluiysis of Tlieories and Mea- The Companion Web Site for The Basics sul-emerlt Strategies. New York: Praeger. This book of Social Research, Third Edition provides an excellent, long-term view of conceptual- http://sociology.wadsworth.com/ ization as the author examines a variety of theoreti- babbie/basics3e/ cal views of social class and the measurement tech- Supplement your review of this chapter by going to the niques appropriate to those theories. companion Web site to take one of the Tutorial Quizzes, Lazarsfeld, Paul F., and Morris Rosenberg, eds. 1955. use the flash cards to master key terms, and check out the The Language of Social Research. Section 1. New many other study aids you'll find there. You'll also find spe- York: Free Press. An excellent and diverse classic cial features such as primers for various data analysis collection of descriptions of specific measurements software (e.g.,SPSS and NVivo), GSS Data, Census 2000 in past social research. These 14 articles present information, up-to-date URLs, and other data and re- useful and readable accounts of actual measure- sources at your fingertips to help you with that special ment operations performed by social researchers project or do some research on your own. as well as more conceptual discussions of measure- ment in general. InfoTrac College Edition Silverman, David. 1993. h~ierpretingQualitative Data: http://www.infotrac-college.com hlethodsjbr Anniyring Talk, Text, and hi teloction, Access the latest news and journal articles Chapter 7. Thousand Oaks, CA: Sage. This deals with with InfoTrac College Edition, an easy-to-use online library Conduct an electronic search using the the issues of validity and specific all^! in' re- gard to qualitative research. following search terms selected by the author: U.S. Department of Health and Human Services. 1992. Anomie Sui-vej,Lfeasuren7ent qFDrug Use. Washington, DC: Attributes AND variables U.S. Government Printing Office. An extensive review Cohort study of techniques devised and used for measuring vari- Conceptualization ous kinds of drug use. Construct validity Wallace, Walter. 1971. The Logic ($Science in Sociolo~, Operationalization Chapter 3. Chicago: Aldine-Atherton. A brief and lu- Panel study cid presentation of concept formation within the con- Trend study text of other research steps. This discussion relates Validity AND reliability