AN UNPUBLISHED QUANTITATIVE RESEARCH METHODS BOOK Thomas R. Knapp © 2016 I have put together in this book a number of slightly-revised unpublished papers I wrote during the last several years. Some were submitted for possible publication and were rejected. Most were never submitted. They range in length from 2 pages to 91 pages, and in complexity from easy to fairly technical. The papers are included in an order in which I think the topics should be presented (design first, then instrumentation, then analysis). You might find some things repeated two or three times. That's because I wrote the papers at different times; the repetition was not intentional. There's something in here for everybody. Feel free to download anything you find to be of interest. Enjoy! Table of contents Chapter 1: How many kinds of quantitative research studies are there?...3 Chapter 2: Should we give up on causality?...8 Chapter 3: Should we give up on experiments?...15 Chapter 4: Random...19 Chapter 5: What a pilot study is…and isn’t...57 Chapter 6: Womb mates...64 Chapter 7: Validity? Reliability? Different terminology altogether?...68 Chapter 8: Seven: A commentary regarding Cronbach's Coefficient Alpha...73 Chapter 9: Assessing the validity and reliability of Likert scales and Visual Analog(ue) scales...79 Chapter 10: Rating, ranking, or both?...90 Chapter 11: Polls...96 Chapter 12: Minus vs. divided by...106 Chapter 13: Change...114 Chapter 14: Separate variables vs. composites...123 Chapter 15: Percentages: The most useful statistics ever invented...130 1 Chapter 16: The unit justifies the mean...221 Chapter 17: The median should be the message...226 Chapter 18: Medians for ordinal scales should be letters, not numbers...236 Chapter 19: Investigating the relationship between two variables...242 Chapter 20: Specify, hypothesize, assume, obtain, test, or prove?...257 Chapter 21: The independence of observations...261 Chapter 22: Significance test, confidence interval, both, or neither?...267 Chapter 23: N (or n) vs. N-1 (or n-1) revisited...277 Chapter 24: Standard errors...283 Chapter 25: In (partial) support of null hypothesis significance testing...286 Chapter 26: p-values...294 Chapter 27: p, n, and t: Ten things you need to know...297 Chapter 28: The all-purpose Kolmogorov-Smirnov test for two independent samples...301 Chapter 29: To pool or not to pool: That is the confusion...307 Chapter 30: Learning statistics through baseball...313 Chapter 31: Learning statistics through finite populations and sampling without replacement...350 Chapter 32: Bias...358 Chapter 33: Numbers of things...363 Chapter 34: Three...381 Chapter 35: Alphabeta soup...384 Chapter 36: Verbal 2x2 tables...389 Chapter 37: Statistics without the normal distribution: A fable...392 2 CHAPTER 1: HOW MANY KINDS OF QUANTITATIVE RESEARCH STUDIES ARE THERE? You wouldn't believe how many different ways authors of quantitative research methods books and articles "divide the pie" into various approaches to the advancement of scientific knowledge. In what follows I would like to present my own personal taxonomy, while at the same time pointing out some other ways of classifying research studies. I will also make a few comments regarding some ethical problems with certain types of research. Experiments, surveys, and correlational studies That's it (in my opinion). Three basic types, with a few sub-types. 1. Experiments If causality is of concern, there is no better way to try to get at it than to carry out an experiment. But the experiment should be a "true" experiment (called a randomized clinical trial, or randomized controlled trial, in the health sciences), with random assignment to the various treatment conditions. Random assignment provides the best and simplest control of possibly confounding variables that could affect the dependent (outcome) variable instead of, or in addition to, the independent ("manipulated") variable of primary interest. Experiments are often not generalizable, for two reasons: (1) they are usually carried out on "convenience" non-random samples; and (2) control is usually regarded as more important in experiments than generalizability, since causality is their ultimate goal. Generalizability can be obtained by replication. Small but carefully designed experiments are within the resources of individual investigators. Large experiments involving a large number of sites require large research grants. An experiment in which some people would be randomly assigned to smoke cigarettes and others would be randomly assigned to not smoke cigarettes is patently unethical. Fortunately, such a study has never been carried out (as far as I know). 2. Surveys Control is almost never of interest in survey research. An entire population or a sample (hopefully random) of a population is contacted and the members of that population or sample are asked questions, usually via questionnaires, to which the researcher would like answers. Surveys based upon probability samples (usually multi-stage) are the most generalizable of the three types. If the survey research is carried out on an 3 entire well-defined population, better yet; but no generalizability beyond that particular population is warranted. Surveys are rarely regarded as unethical, because potential respondents are free to refuse to participate wholly (e.g., by throwing away the questionnaire) or partially (by omitting some of the questions). 3. Correlational studies Correlational studies come in various sizes and shapes. (N.B.: The word "correlational" applies to the type of research, not to the type of analysis, e.g., the use of correlation coefficients such as the Pearson product-moment measure. Correlation coefficients can be as important in experimental research as in non-experimental research for analyzing the data.) Some of the sub-types of correlational research are: (1) measurement studies in which the reliability and/or validity of measuring instruments are assessed; (2) predictive studies in which the relationship between one or more independent (predictor) variables and one or more dependent (criterion) variables are explored; and (3) theoretical studies that try to determine the "underlying" dimensions of a set of variables. This third sub-type includes factor analysis (both exploratory and confirmatory) and structural equation modeling (the analysis of covariance structures). The generalizability of a correlational research study depends upon the method of sampling the units of analysis (usually individual people) and the properties of the measurements employed. Correlational studies are likely to be more subject to ethical violations than either experiments or surveys, because they are often based upon existing records, the access to which might not have the participants' explicit consents. (But I don't think that a study of a set of anonymous heights and weights for a large sample of males and females would be regarded as unethical; do you?) Combination studies The terms "experiment", "survey", and "correlational study" are not mutually exclusive. For example, a study in which people are randomly assigned to different questionnaire formats could be considered to be both an experiment and a survey. But that might better come under the heading of "methodological research" (research on the tools of research) as opposed to "substantive research" (research designed to study matters such as the effect of teaching method on pupil achievement or the effect of drug dosage on pain relief). Pilot studies Experiments, surveys, or correlational studies are often preceded by feasibility studies whose purpose is to "get the bugs out" before the main studies are undertaken. Such studies are called "pilot studies", although some researchers 4 use that term to refer to small studies for which larger studies are not even contemplated. Whether or not the substantive findings of a pilot study should be published is a matter of considerable controversy. Two other taxonomies Epidemiology In epidemiology the principal distinction is made between experimental studies and "observational" studies. The basis of the distinction is that experimental studies involve the active manipulation (researcher intervention) of the independent variable(s) whereas observational studies do not. An observational epidemiological study usually does not involve any actual visualization of participants (as the word implies in ordinary parlance), whereas a study in psychology or the other social sciences occasionally does (see next section). There are many sub-types of epidemiological research, e.g., analytic(al) vs. descriptive, and cohort vs. case-control. Psychology In social science disciplines such as psychology, sociology, and education, the preferred taxonomies are similar to mine, but with correlational studies usually sub-divided into cross-sectional vs. longitudinal, and with the addition of quantitative case studies of individual people or groups of people (where observation in the visual sense of the word might be employed). Laboratory animals Much research in medicine and in psychology is carried out on infrahuman animals rather than human beings, for a variety of reasons; for example: (1) using mice, monkeys, dogs, etc. is generally regarded as less unethical than using people; (2) certain diseases such as cancer develop more rapidly in some animal species and the benefits of animal studies can be realized sooner; and (3) informed consent of the animal itself is not required (nor can be obtained). The necessity for animal research