Best Practice Suggestions for Cognitive Psychology and Neuroscience

How to run behavioural experiments online: best practice suggestions for cognitive psychology and neuroscience Nathan Gagné1,* & Léon Franzen1,2,* 1 Department of Psychology, Concordia University, Montreal, Quebec, Canada 2 Department of Psychology, University of Lübeck, Schleswig-Holstein, Germany * These authors contributed equally ORCID iD: 0000-0003-2232-9408 (NG), 0000-0003-2277-5408 (LF) Keywords: online research, best practice suggestions, cognition, neuroscience, dyslexia, working memory Abstract The combination of a replication crisis, global COVID-19 pandemic, and recent technological advances have accelerated the on-going transition of research in cognitive psychology and behavioural neuroscience to the online realm. When participants cannot be tested in-person in the laboratory, data of acceptable quality can be collected online still. While conducting research online has many advantages and generic advice on their infrastructure and implementation exists, numerous pitfalls can hinder researchers addressing their research question appropriately or even result in unusable data. Here, we present detailed best practice suggestions that span the range from initial study design to the final interpretation of the data. These suggestions take a critical look at issues regarding the recruitment of typical and (sub)clinical samples, their comparison, and the importance of context- dependency for each part of a study. We illustrate our suggestions by means of a recent online study investigating cognitive working memory skills in adult dyslexia. Gagné & Franzen, 2021 Suggestions for online research version 1 page 1 Introduction In the midst of a global pandemic, experimental research has exceeded the realm of physical space through online experimentation. In a change that started years ago, its transition at large scale was long overdue. Most researchers were forced to transition their research to an online setting almost overnight. This increasing shift to online experiments in cognitive psychology and behavioural neuroscience was enabled by increasing technological innovation and put a new and more intense focus on both its advantages and disadvantages. Until the start of the COVID-19 pandemic, data validity and reliability had only been investigated for selected leading platforms, such as Amazon’s Mechanical Turk (MTurk; e.g., Berinsky et al., 2012; Chandler et al., 2014; Crump et al., 2013), and groups of experiments (e.g., psychophysics: de Leeuw & Motz, 2016; perception: Woods et al., 2015). However, the sudden shift encouraged publications addressing the generic implementation of online experiments in accordance with the current technological possibilities (Grootswagers, 2020; Sauter et al., 2020). These constitute a great starting point but a more nuanced take on many, often practical and experimental, aspects of online research remains absent from the literature. Here, we aim to provide best practice suggestions spanning the entire study cycle from the design to interpreting the results, with a specific focus on the sampling of scarce populations. To this end, we use a recent online investigation of working memory skills in adults with dyslexia as a practical and tangible illustration, as the recruitment of dyslexia and other specific populations has a long history of proving difficult, especially in a university context. Online research has gained further importance in recent years as some fields including psychology and neuroscience have found themselves in a dire replication crisis recently (Baker & Penny, 2016; Ioannidis, 2005; Ioannidis et al., 2014; Makel et al., 2012). Their failure to replicate findings of previous work has been a growing trend (Sharpe & Poets, 2020) and reproducibility rates of published findings in the field of psychology are estimated to only range between 35% and 75% (Artner et al., 2020; Camerer et al., 2018; Etz & Vandekerckhove, 2016). Many well-known psychological findings have later failed to replicate, such as the “marshmallow test” and the “ego depletion” theory (Hagger et al., 2016; Open Science Collaboration, 2015; Watts et al., 2018). Out of 1,576 researchers surveyed, 52% of them believed Gagné & Franzen, 2021 Suggestions for online research version 1 page 2 there was a significant reproducibility crisis (Baker & Penny, 2016). In fact, more than half of researchers have tried and failed to reproduce their own studies. These numbers do not build trust in reported findings. At the same time, between 1900- 2012, only 1.2% of articles in major psychology journals were replication studies, indicating its low value in the community until then (Makel et al., 2012). Causes for low replication rates had been attributed to the complex nature of reproducing experimental methodology including, problems with statistical power, selective and small convenience sampling, engaging in questionable research practices, publication bias, and high costs associated with running a study repeatedly including a sufficient sample size among others (Button et al., 2013; Holcombe, 2020; Ioannidis et al., 2014; John et al., 2012; Munafò et al., 2017; Nosek et al., 2012; Rosenthal, 1979; Simmons et al., 2011). Online research provides solutions for these issues that can be implemented with relative ease (see advantages listed in Table 1). These solutions include the important ability to efficiently collect large samples from a wider socio-economic background for less monetary and time investments on the researchers’ end, which would otherwise be difficult to obtain in-person (for discussions, see Grootswagers, 2020; Mathôt & March, 2021; Sauter et al., 2020), and all-in-one platforms offering experiment building, presentation, recruitment and distribution capabilities. Hence, online research could become advantageous for both single studies and research fields in the context of this replication crisis. Increased possibilities for recruitment are the most frequently mentioned argument for online studies. Differences in recruitment strategies can increase an experiment’s ecological validity. For example, researchers examining the beliefs of registered voters by calling numbers listed in a telephone book would fail to capture registered voters not listed in the directory, resulting in a form of under-coverage bias (Schaeffer et al., 2011). Many of these sampling biases in experimental research are a product of convenience sampling, which involves drawing a sample from a population that is close and easy to reach (Saunders et al., 2019). This type of sampling method is often useful for pilot studies, but psychological research has become increasingly reliant on convenience samples of undergraduate students from western universities, resulting in a misrepresentation of the true population (Masuda et al., 2020). Henrich et al. (2010) grouped these issues with the growing reliance on such samples in psychological research in the descriptive term WEIRD Gagné & Franzen, 2021 Suggestions for online research version 1 page 3 (Westernized, Educated, Industrialized, Rich, and Democratic). These individuals tend to be undergraduate students who are taking courses in psychology or neuroscience and have previously been exposed to psychological research. WEIRD samples in university participant pools allow researchers to conduct their experiments at a low cost and with limited recruitment effort. Students are already familiar with the experimental process and only receive course credit as a compensation, which results in an easy and low-cost form of recruitment (Masuda et al., 2020). The evident convenience associated with WEIRD samples has often left researchers reluctant to explore new ways to expand their sampling efforts. One example illustrating the importance of appropriate sampling comes from the field of dyslexia research. Researchers have failed to reach a consensus on the perceptual (non-linguistic) nature of developmental dyslexia. A point of contention in the field is the conflicting evidence for a visual, perceptual magnocellular deficit in dyslexia (Skottun, 2000; Stein, 2001, 2018; Stein et al., 2000; Stein & Walsh, 1997). The foundations of this theory may be subject to sampling bias themselves, as researchers reporting supporting evidence have predominantly sampled from populations at ophthalmology hospitals: a convenience sample (Blythe et al., 2018). In other words, whether dyslexia, a specific learning difficulty defined by selective deficits in the linguistic realm, also affects non-linguistic tasks or aspects of tasks involving higher cognitive functions such as working memory capacity remains elusive and is addressed in the exemplary study. Online research provides an alternative method that is well-suited to bridge this gap in knowledge by allowing for easier replication of research while not sacrificing the true representation of a population. In fact, Berinsky and colleagues (2012) found online samples on MTurk to be more representative of the US population than standard samples recruited for laboratory studies in university settings. Nevertheless, the suitability of a sample depends heavily on the context and posed research questions. For instance, if the research question is not US- centric or the common fine-tuning of an experiment’s difficulty level is only performed generically, recruitment through this channel may not provide meaningful data for every participant from the general population. Study-specific method considerations are key to properly utilizing the advantages of a remote

Load more