Methods to Analyze Likert-Type Data in Educational Technology Research
Total Page:16
File Type:pdf, Size:1020Kb
Journal of Educational Technology Development and Exchange (JETDE) Volume 13 Issue 2 3-25-2021 Methods to Analyze Likert-Type Data in Educational Technology Research Li-Ting Chen University of Nevada, Reno, [email protected] Leping Liu University of Nevada, Reno, [email protected] Follow this and additional works at: https://aquila.usm.edu/jetde Part of the Educational Technology Commons Recommended Citation Chen, Li-Ting and Liu, Leping (2021) "Methods to Analyze Likert-Type Data in Educational Technology Research," Journal of Educational Technology Development and Exchange (JETDE): Vol. 13 : Iss. 2 , Article 3. DOI: 10.18785/jetde.1302.04 Available at: https://aquila.usm.edu/jetde/vol13/iss2/3 This Article is brought to you for free and open access by The Aquila Digital Community. It has been accepted for inclusion in Journal of Educational Technology Development and Exchange (JETDE) by an authorized editor of The Aquila Digital Community. For more information, please contact [email protected]. Chen, L.T., & Liu, L.(2020). Methods to Analyze Likert-Type Data in Educational Technology Research. Journal of Educational Technology Development and Exchange, 13(2), 39-60 Methods to Analyze Likert-Type Data in Educational Technology Research Li-Ting Chen University of Nevada, Reno Leping Liu University of Nevada, Reno Abstract: Likert-type items are commonly used in education and related fields to measure attitudes and opinions. Yet there is no consensus on how to analyze data collected from these items. In this paper, we first provided a synthesis of the existing literature on methods to analyze Likert-type data and computing tools for these methods. Secondly, to examine the use and analysis of Likert-type data in the field of educational technology, we reviewed 424 articles that were published in the journal Educational Technology Research and Development between 2016 and 2020. Our review showed that about 50% of the articles reported Likert-type data. A total of 139 articles used Likert-type data as a dependent variable, among which 86% employed parametric methods to analyze the data. In addition, less than 3% of the 139 articles used an ordered probit/logit model, transformation, or strategy for rescaling Likert-type data to interval data to perform statistical analysis. Finally, to empower educational technology researchers to handle Likert-type data effectively, we concluded the paper with our suggestions and insight regarding alternative strategies and methods. Keywords: Likert, ordinal data, research method, self-report measures, rescaling, nonparametric, ordered probit, ordered logit, robust Volume 13, No. 2, December, 2020 39 Journal of Educational Technology Development and Exchange 1. Introduction permissible and impermissible transformations for numbers yielded from the four types of Questionnaires that ask individuals (e.g., measurements. Permissible transformations students, parents, or teachers) to rate their are transformations that maintain the same attitudes and opinions about statements using meanings of the numerals assigned to Likert-type items (Likert, 1932) are common observations. Permissible transformations in education and related fields (Antonialli et for ordinal data, such as Likert-type data, al., 2017; Carifio & Perla, 2007; Edmondson, are monotonic transformations, or positive 2005; Harwell & Gatti, 2001; Liddell & linear transformations, but not one-to-one Kruschke, 2018; Potvin & Hasni, 2014; substitutions. Permissible transformations for Tsui, 1997). For example, a teacher may be interval data are positive linear transformations asked to respond to the statement “I can learn but not monotonic transformations or one-to- technology easily” using the five response one substitutions. Following the prescription, options: 0 = strongly disagree, 1 = disagree, Stevens (1946, 1955, 1968) urged researchers 2 = neither agree nor disagree, 3 = agree, 4 to attend to the type of measurement. Stevens = strongly agree. Although the options are stated labelled using numbers, the numerals only The ordinal scale arises from the operation indicate orders. They do not necessarily imply of rank-ordering. … most of the scales used that distances between two adjacent options widely and effectively by psychologists are equal. That is, the distances between 0 are ordinal scales. In the strictest propriety and 1, 1 and 2, 2 and 3, and 3 and 4 may be the ordinary statistics involving means and different. standard deviations ought not to be used with these scales, for these statistics imply According to Stevens (1946), there a knowledge of something more than the are four types of measurements (nominal, relative rank-order of data (Stevens, 1946, ordinal, interval, and ratio) and the types p. 679). of measurements are determined by their basic empirical operations. Nominal In fact, contrary to Stevens’s suggestion, measurement consists of category labels analyzing Likert-type data using statistical (e.g., numbers or symbols) that can be methods that require interval or ratio data is assigned to observations (or individuals) a common practice. These statistical methods so that those with different labels are not include but are not limited to the t test, F equivalent. With an ordinal measurement, test, Pearson’s correlation, and ordinary least category labels are assigned to observations squares regression. These methods are also to rank and order them with respect to one called parametric methods. In a commentary another. Using an interval measurement, article that published in Medical Education, numbers are assigned to observations. The Carifio and Perla (2008) pointed out “How numbers have the property of order, and Likert type measurement scales should be equal differences between any two adjacent appropriately used and analysed has been numbers reflect equal magnitude. All the debated for over 50 years” (p. 1150). Just properties of an interval measurement apply what are the discussions related to the use and to a ratio measurement, and in addition, there analysis of Likert-type data? What analysis is a true zero point for a ratio measurement strategies have been proposed in the literature to reflect the absence of the measured for dealing with Likert-type data? How characteristic. Stevens emphasized the prevalent is Likert-type data used in studies 40 Volume 13, No. 2, December, 2020 Methods to Analyze Likert-Type Data in Educational Technology Research in the field of education technology? What the 47-item Pre-service Teacher Technological are the analysis strategies used by educational Pedagogical Content Knowledge Instrument technology researchers to handle Likert-type with five response options (Schmidt et al., data? In this paper, we address these questions. 2009). Specifically, there are three aims of this paper: One potential problem with using 1. to summarize the strategies and methods to parametric methods for Likert-type data use and analyze Likert-type data from the is about the normality assumption, which literature; requires continuous/interval data. Scores 2. to investigate the use and analysis of yielded from Likert-type items are discrete Likert-type data in educational technology and ordinal in nature. For example, when research; and five items are used to measure students’ 3. to provide suggestions for educational perceived competence in information and technology researchers to handle Likert- communication technology and all five type data. items are rated using four response options (Areepattamannil & Santos, 2019), ranging 2. Literature Review: Strategies and from 1 (strongly disagree) to 4 (strongly Methods to Analyze Likert-Type Data agree), there are 16 possible total scores with the minimum score of 5 and maximum score Likert summative attitude scales were of 20. Furthermore, when computing the total first introduced by Rensis Likert in the 1930s. scores with weighting each item equally, it Likert (1932) suggested that response options ignores the unique characteristics of each item to several statements on Likert summative (Harwell & Gatti, 2001). attitude scales can be written as 1 = strongly disapprove to 5 = strongly approve. Such While some researchers think it is crucial response options are widely used to measure to consider the type of measurement when attitudes and opinions. In this paper, we used using a statistical method (Jamieson, 2004; the term Likert-type data to refer to data Kuzon et al., 1996; Siegel, 1956; Sprinthall, collected from such response format, and the 2012), others care less about the type of term Likert-type item to refer to a statement measurement (Carifio & Perla, 2007; Howell, with such response format to measure 2013; Lord, 1953; Norman, 2010; Velleman individuals’ attitudes and opinions. In the & Wilkinson, 1993; Zimmerman, 2011). field of educational technology, researchers Kuzon et al. (1996) referred using parametric may ask online students to respond to methods for ordinal data as the first sin of statements on the 34-item Community of the seven deadly sins of statistical analysis Inquiry framework survey with five response and suggested “to avoid committing Sin options to understand students’ perception 1, for nominal or ordinal scaled data, use of cognitive presence, social presence, and nonparametric statistical analysis” (p. 266). teaching presence (Arbaugh et al., 2008). In contrast, Carifio and Perla (2007) presented To understand pre-service teachers’ self- the top ten myths about “Likert scales” and assessment of Technological Pedagogical wrote