The Following Full Text Is a Publisher's Version
Total Page:16
File Type:pdf, Size:1020Kb
PDF hosted at the Radboud Repository of the Radboud University Nijmegen The following full text is a publisher's version. For additional information about this publication click this link. http://hdl.handle.net/2066/199928 Please be advised that this information was generated on 2021-09-27 and may be subject to change. SOCIAL MOVEMENT STUDIES 2018, VOL. 17, NO. 6, 736–748 https://doi.org/10.1080/14742837.2018.1499511 Articles reporting research on Latin American social movements are only rarely transparent Sven Da Silvaa, Peter A. Tamás b and Jarl K. Kampen b,c aSociology of Development and Change, Wageningen University, Wageningen, Netherlands; bBiometris, Wageningen University, Wageningen, Netherlands; cStatua, Dept Epidemiology and Medical Statistics, Antwerp University, Antwerpen, Belgium ABSTRACT ARTICLE HISTORY Social movement scholars often want their research to make a dif- Received 5 December 2017 ference beyond the academy. Readers will either read reports directly Accepted 9 July 2018 fi or they will read reviews that aggregate ndings across a number of KEYWORDS fi reports. In either case, readers must nd reports to be credible before Systematic review; social they will take their findings seriously. While it is not possible to movements; Latin America; predict the indicators of credibility used by individual, direct readers, reporting standards; formal systems of review do explicate indicators that determine methodological whether a report will be recognized as credible for review. One transparency such indicator, also relevant to pre-publication peer review, is meth- odological transparency: the extent to which readers are able to detect how research was done and why that made sense. This paper tests published primary research articles on and for social movements in Latin America for compliance with a generous inter- pretation of methodological transparency. We find that, for the most part, articles are not methodologically transparent. If transparency matters to social movement scholars, the research community may wish to formalize discussions of what aspects of research should be reported and how those reports should be structured. Academics have engaged in research on and with social movements, often to support those movements’ objectives, for at least a generation (e.g. Gutierrez & Lipman, 2016; Smith, 1990). These scholars have discussed a number of methodological challenges such as those attending the use of memoirs as data (Marche, 2015); the effect of engagement on critical reflexivity (e.g. Petray, 2012); and the impact of analytic con- veniences, such as the naturalization of the individual as the unit of analysis, on researchers’ ability to detect and discuss mechanisms of oppression (Fine, 1989). The implications of deliberate academic activism for research ethics, as manifest for example in community-based participatory research, are well discussed (e.g. Cordner, Ciplet, Brown, & Morello-Frosch, 2012). Stepping back a bit, the universities we constitute have been argued, themselves, to be functional to the structural inequalities that attract engaged scholarship (Meyerhoff & Thompsett, 2017), and all representations made by outsiders may be found to be inescapably violent as such truth making is necessarily CONTACT Peter A. Tamás [email protected] Biometris, Wageningen University, Postbus 8130, Wageningen 6700EW Netherlands © 2018 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way. SOCIAL MOVEMENT STUDIES 737 extractive (Luchies, 2015). These discussions have produced a number of recommenda- tions for practice such as those found in militant ethnography (Apoifis, 2017). While a great deal has been contributed to discussions surrounding primary research, we have found far less on the review of studies of social movements. Review, which determines both publication and reuse, treats the reports of studies as primary data. Serious readers will look beyond conclusions and check if a report is transparent: does it contain a discussion adequate to allow a reader to understand how research was undertaken and why it makes sense? For this essay we decided to borrow expectations of transparency that are used within systematic review as adapted for low- consensus qualitative inquiry. While systematic review sits at the pinnacle of the well- critiqued hierarchy of knowledge, and its adoption in the social sciences carries traces of an unfortunate scientism (Bannister, 1987; Hayek, 1942), the features it looks for in reports are relevant as they are consistent with the requirements for meaningful review by peers and for careful integration into the plans of donors, of non-governmental organizations of governments and other scholars. In addition, adaptations of systematic review are increasingly common in the social sciences and its use is encouraged in the study of social movements (e.g. McAdam, Tarrow, & Tilly, 2001) so the expectations of systematic review may increasingly determine whose voices are recognized. Transparency, as operationalized in this essay, is not the same thing as methodolo- gical quality. The formal quality assessment tools used to examine qualitative research, some analogue of which we hope informs both peer review and use of findings in supporting decision making, tend to mix transparency with tests of internal coherence and quality (e.g. those suggested by Carroll, Booth, & Lloyd-Jones, 2012; Dixon-Woods et al., 2006; Dixon-Woods, Shaw, Agarwal, & Smith, 2004; Fossey, Harvey, McDermott, & Davidson, 2002; Kmet, Lee, & Cook, 2004; Pawson, Boaz, Grayson, Long, & Barnes, 2003; Spencer, Ritchie, Lewis, & Dillon, 2003; Tong, Flemming, McInnes, Oliver, & Craig, 2012). For our study we only asked if published reports are transparent. Consistent with the expectations of reviewers in the social sciences, in our study transparency begins with the ability to find in a report of primary research ‘...sufficient detail of the research question, design, and methods to allow an assessment’ (Popay & Williams, 1998:35). Carlsen and Glenton (2011)statethat‘transparency and account- ability are key elements in any research report, not least in qualitative studies. Thorough reporting of methods allows readers to assess the quality and relevance of research findings’ (p. 1). Seale (1999) called this auditing and, expressing a sentiment that acknowledges the complexities of qualitative research, notes that researchers should provide ‘a methodologically self-critical account of how the research was done’ (p. 468). Similarly acknowledging the reality of qualitative field work, Tracy (2010) states that ‘transparent research is marked by disclosure of the study’s challenges and unexpected twists and turns and revelation of the ways research foci transformed over time’ (p. 842). In the next sections, we present methods and results of a systematic review of transparency in a corpus of articles reporting on empirical inquiries with respect to social movements in Latin America. We provide a concise discussion of our assessment of transparency of these research reports, followed by some conclusions and recom- mendations for future reporting. The full study with its underlying data are, of course, available for review. 738 S. DA SILVA ET AL. Methods Sampling Our study examined research conducted in Latin America. This thematic focus was determined entirely by the interest of the first author: he wished to understand how research is done so that he could do better research. While we have no reason to anticipate significant differences, we did not test if the transparency of reports of research on social movements in Latin America differs from those undertaken in other regions. We retrieved accessible English and Spanish articles reporting primary research on, and at times with, Latin American social movements through a search executed in ISI Web of Science. Table 1 presents the search syntax as it was inserted, with no time limit set, on 16 January 2013, in ISI Web of Science.1 This search string returned all articles in English or Spanish in the social sciences on social movements in Latin America from at least 1975 to the present. We chose to search ISI indexed journals as our expectation was this would bias our sample towards more transparent reporting of methodological details. The search syntax identified 549 related articles of which 510 were English and 39 were Spanish. We then examined the titles, abstracts and keywords of the articles identified in our search. We retained articles for further study when they met all of the following conditions: 1. Articles had to be available through the Wageningen University Library. 2. Articles needed to be on social movement(s) located in Latin America, yet not exclusively as articles that made a comparison of a movement in a Latin American country with one in another continent where also included. A definition of ‘social movement’ was not pre-defined. An article could, for example, be included when ‘urban movement’, ‘peasant movement’ or ‘student movement’ appeared in the keywords, abstract or title. 3. Articles needed to report primary social science research.2 4. Articles needed to be written