Publication, Publication Gary King, Harvard University Introduction The broader scientific community Some students ask: “Why begin an both collectively and in many other in- original research paper by replicating I show herein how to write a publish- dividual fields is also moving strongly some old work?” A paper that is publish- able paper by beginning with the replica- in the direction of participating in or able is one that by definition advances tion of a published article. This strategy requiring some form of data sharing. knowledge. If you start by replicating an seems to work well for class projects in Recipients of grants from the National existing work, then you are right at the producing papers that ultimately get pub- Science Foundation and the National cutting edge of the field. If you can then lished, helping to professionalize students Institutes of Health now are required to improve any one aspect of the research into the discipline, and teaching the sci- make data available to other scholars that makes a substantive difference and entific norms of the free exchange of upon publication or within a year of the is defensible, you have a publishable academic information. I begin by briefly termination of their grant. Replicating, paper. If instead you begin a project revisiting the prominent debate on repli- and thus collectively and publicly vali- from scratch without replication, you cation our discipline had a decade ago dating, the integrity of our published need to defend every coding decision, and some of the progress made in data work is often still more difficult than it every hypothesis, every data source, sharing since. should be, and some still oppose the every method—everything. In contrast, if A decade ago this journal published a whole idea, but our discipline has made you start with replication, you only need symposium on replication policies in po- substantial progress.1 to defend the one area you are improv- litical science. The symposium began The original replication debate in PS ing, and you can stipulate to the rest. If a with an article I wrote entitled “Replica- included discussions about student in- critic doesn’t like something else in the tion, Replication,” and was followed by volvement, and indeed some departments original article other than that which you opposing and supporting comments by now require students writing disserta- are improving, you need not defend that 19 others ~King, 1995!. The debate over tions and senior theses to submit a repli- point since it is already part of the pub- proper policies continued for a few years cation data set that, after an optional lished record and is the recognized state in subsequent issues of the journal and a embargo period, gets made public and of the art. After all, this strategy was not variety of other public fora. Since then, is permanently archived. In the decade originally designed for students; it is ex- many journals in political science have since “Replication, Replication,” and actly the procedure followed by many adopted some form of a data sharing or also in the decade leading up to it, I faculty in political science and most replication policy. Some strongly recom- have tried other ways to help students other scientific fields. It is one of the mend or expect data sharing and some benefit from this trend. Chief among reasons that the process of providing ac- require it as a condition of publication. these has been an effort to professional- cess to the raw materials of research with The editors of the major international ize my students by, among other things, sufficient precision necessary to repli- relations journals have collectively writ- giving them first-hand experience repli- cate, and of accessing that from other ten and committed themselves to a cating published work and publishing scholars, has become a deeply estab- strong standard minimum replication pol- their own. In particular, I require my lished part of the scientific process.3 icy ~Gleditsch et al. 2003!. Most impor- students to write a “publishable” empiri- What follows is some of the advice I tant, numerous individual scholars now cal paper for their class project based on give my students. regularly share their data, produce repli- their replication of an existing published cation data sets, put these data sets on article. Indeed, most of this paper is Elements of the Paper their web sites, send them to the ICPSR taken from a handout I have edited and and other archives, or distribute them on re-edited over 20 years to maximize the 1. Your paper should address a sub- request to other scholars. Scholars some- chance that students are able to publish stantive problem in your field of interest times worry about being “scooped,” the paper they write for a methods class and contain one or a few clear points; about maintaining the confidentiality of I teach.2 Students are told that successful one point with several supporting points their respondents, or about being proven projects need not actually be published is better than a lot of unrelated points. wrong, but since authors who make their or even submitted for publication. How- Your point should unambiguously answer data available are more than twice as ever, although writing a publishable the question: Whose mind are you going cited and influential as those who do not paper may sound hard at first, revised to change about what? If that question ~Gleditsch, Metelits, and Strand 2003!, versions of a large proportion of student isn’t answered, then you’re not making a the strong trend toward data sharing in papers every year eventually result in contribution and there’s little reason for the discipline should not come as a published journal articles, and many the paper to be published. surprise. have also appeared as convention papers, 2. Begin by locating an article in your dissertations, or senior theses, and they field, acquiring the data used in the arti- have won many awards. Almost all of cle, and replicating the specific numeri- those who closely follow the suggestions cal results in the tables and0or figures in Gary King is the David Florence Profes- ~ sor of Government at Harvard University, below wind up with published articles. that analysis. You may start with the where he also serves as director of the The advice offered here is not the only original article and find the data used, Institute for Quantitative Social Science. way to conduct high quality research, but or work backwards from the data, such His homepage can be found at http:// it is one relatively high probability path as stored in the ICPSR’s Publication gking.harvard.edu. to success. Related Archive, or one of the other PSOnline www.apsanet.org 119 archives of datasets constructed for the way depending on what you find or diffi- less they help you demonstrate this one purpose of facilitating replication, and culties in replication and do it all again. key point. Resist the temptation to in- find the scholarly article.! This article Perhaps this is why they call it research, clude all this just to demonstrate how should have been published in a peer- rather than merely search! ~If you change much work you did; that’s not the crite- reviewed scholarly journal, preferably articles, please bring the new article to rion on which you will be judged in this within the last 4–5 years, the more re- me as well.! You may wish to follow the class ~or afterwards!. This task is a cru- cent and prominent the better. The better procedure that many of us follow by cial aspect of your socialization into the the article, journal, and author you starting several projects at once and then profession, and your success requires that choose, and the more often the article following up those that seem most you learn it at some point. It might as has been cited, the more likely your productive. well be now. paper will be publishable. Checking cita- 4. If you decide that the conclusions 7. After replicating the article, follow tion indices ~ISI or Google Scholar, for of the original article are incorrect, then the logic of King, Tomz, and Wittenberg example! is often a good idea, but be show why you think that but also what ~2000! and try to improve the presenta- mindful of the selection problem that led the authors of the original article to tion of the original results. See whether occurs because more recent articles will think otherwise. You should never dis- you can find useful, additional, or even have had less time to be cited. cuss it in the paper—directly or indi- contradictory information not discussed Please beware: replicating an article, rectly—but you should assume, unless in the article without changing any as- even if you secure access to the original you have overwhelming evidence to the sumptions in the original paper. If you data, is normally a highly uncertain and contrary and maybe even then, that the are able to do this, then you need not difficult process. Analyses that look neat authors were well-intentioned, smart, defend anything other than your method and clean in published articles often honest, and hard-working. Your article is of presentation, which would put you on prove to be far from that in reality. Most about the author’s findings, not about the very strong grounds in your claim for students find that prominent articles by author. journal space. You may find Zelig ~Imai, leading scholars in the field contain er- 5. Clarify with precision the extent to King, and Lau 2004! or Clarify ~Tomz, rors, confusions, lack of essential infor- which you were able to replicate the Wittenberg, and King 2003! software mation about how the analysis was author’s results.