Publication, Publication

Gary King, Harvard University

Introduction The broader scientific community Some students ask: “Why begin an both collectively and in many other in- original research paper by replicating I show herein how to write a publish- dividual fields is also moving strongly some old work?” A paper that is publish- able paper by beginning with the replica- in the direction of participating in or able is one that by definition advances tion of a published article. This strategy requiring some form of data sharing. knowledge. If you start by replicating an seems to work well for class projects in Recipients of grants from the National existing work, then you are right at the producing papers that ultimately get pub- Science Foundation and the National cutting edge of the field. If you can then lished, helping to professionalize students Institutes of Health now are required to improve any one aspect of the research into the discipline, and teaching the sci- make data available to other scholars that makes a substantive difference and entific norms of the free exchange of upon publication or within a year of the is defensible, you have a publishable academic information. I begin by briefly termination of their grant. Replicating, paper. If instead you begin a project revisiting the prominent debate on repli- and thus collectively and publicly vali- from scratch without replication, you cation our discipline had a decade ago dating, the integrity of our published need to defend every coding decision, and some of the progress made in data work is often still more difficult than it every hypothesis, every data source, sharing since. should be, and some still oppose the every method—everything. In contrast, if A decade ago this journal published a whole idea, but our discipline has made you start with replication, you only need symposium on replication policies in po- substantial progress.1 to defend the one area you are improv- litical science. The symposium began The original replication debate in PS ing, and you can stipulate to the rest. If a with an article I wrote entitled “Replica- included discussions about student in- critic doesn’t like something else in the tion, Replication,” and was followed by volvement, and indeed some departments original article other than that which you opposing and supporting comments by now require students writing disserta- are improving, you need not defend that 19 others ~King, 1995!. The debate over tions and senior theses to submit a repli- point since it is already part of the pub- proper policies continued for a few years cation data set that, after an optional lished record and is the recognized state in subsequent issues of the journal and a embargo period, gets made public and of the art. After all, this strategy was not variety of other public fora. Since then, is permanently archived. In the decade originally designed for students; it is ex- many journals in political science have since “Replication, Replication,” and actly the procedure followed by many adopted some form of a data sharing or also in the decade leading up to it, I faculty in political science and most replication policy. Some strongly recom- have tried other ways to help students other scientific fields. It is one of the mend or expect data sharing and some benefit from this trend. Chief among reasons that the process of providing ac- require it as a condition of publication. these has been an effort to professional- cess to the raw materials of research with The editors of the major international ize my students by, among other things, sufficient precision necessary to repli- relations journals have collectively writ- giving them first-hand experience repli- cate, and of accessing that from other ten and committed themselves to a cating published work and publishing scholars, has become a deeply estab- strong standard minimum replication pol- their own. In particular, I require my lished part of the scientific process.3 icy ~Gleditsch et al. 2003!. Most impor- students to write a “publishable” empiri- What follows is some of the advice I tant, numerous individual scholars now cal paper for their class project based on give my students. regularly share their data, produce repli- their replication of an existing published cation data sets, put these data sets on article. Indeed, most of this paper is Elements of the Paper their web sites, send them to the ICPSR taken from a handout I have edited and and other archives, or distribute them on re-edited over 20 years to maximize the 1. Your paper should address a sub- request to other scholars. Scholars some- chance that students are able to publish stantive problem in your field of interest times worry about being “scooped,” the paper they write for a methods class and contain one or a few clear points; about maintaining the confidentiality of I teach.2 Students are told that successful one point with several supporting points their respondents, or about being proven projects need not actually be published is better than a lot of unrelated points. wrong, but since authors who make their or even submitted for publication. How- Your point should unambiguously answer data available are more than twice as ever, although writing a publishable the question: Whose mind are you going cited and influential as those who do not paper may sound hard at first, revised to change about what? If that question ~Gleditsch, Metelits, and Strand 2003!, versions of a large proportion of student isn’t answered, then you’re not making a the strong trend toward data sharing in papers every year eventually result in contribution and there’s little reason for the discipline should not come as a published journal articles, and many the paper to be published. surprise. have also appeared as convention papers, 2. Begin by locating an article in your dissertations, or senior theses, and they field, acquiring the data used in the arti- have won many awards. Almost all of cle, and replicating the specific numeri- those who closely follow the suggestions cal results in the tables and0or figures in Gary King is the David Florence Profes- ~ sor of Government at Harvard University, below wind up with published articles. that analysis. You may start with the where he also serves as director of the The advice offered here is not the only original article and find the data used, Institute for Quantitative Social Science. way to conduct high quality research, but or work backwards from the data, such His homepage can be found at http:// it is one relatively high probability path as stored in the ICPSR’s Publication gking.harvard.edu. to success. Related Archive, or one of the other

PSOnline www.apsanet.org 119 archives of datasets constructed for the way depending on what you find or diffi- less they help you demonstrate this one purpose of facilitating replication, and culties in replication and do it all again. key point. Resist the temptation to in- find the scholarly article.! This article Perhaps this is why they call it research, clude all this just to demonstrate how should have been published in a peer- rather than merely search! ~If you change much work you did; that’s not the crite- reviewed scholarly journal, preferably articles, please bring the new article to rion on which you will be judged in this within the last 4–5 years, the more re- me as well.! You may wish to follow the class ~or afterwards!. This task is a cru- cent and prominent the better. The better procedure that many of us follow by cial aspect of your socialization into the the article, journal, and author you starting several projects at once and then profession, and your success requires that choose, and the more often the article following up those that seem most you learn it at some point. It might as has been cited, the more likely your productive. well be now. paper will be publishable. Checking cita- 4. If you decide that the conclusions 7. After replicating the article, follow tion indices ~ISI or Google Scholar, for of the original article are incorrect, then the logic of King, Tomz, and Wittenberg example! is often a good idea, but be show why you think that but also what ~2000! and try to improve the presenta- mindful of the selection problem that led the authors of the original article to tion of the original results. See whether occurs because more recent articles will think otherwise. You should never dis- you can find useful, additional, or even have had less time to be cited. cuss it in the paper—directly or indi- contradictory information not discussed Please beware: replicating an article, rectly—but you should assume, unless in the article without changing any as- even if you secure access to the original you have overwhelming evidence to the sumptions in the original paper. If you data, is normally a highly uncertain and contrary and maybe even then, that the are able to do this, then you need not difficult process. Analyses that look neat authors were well-intentioned, smart, defend anything other than your method and clean in published articles often honest, and hard-working. Your article is of presentation, which would put you on prove to be far from that in reality. Most about the author’s findings, not about the very strong grounds in your claim for students find that prominent articles by author. journal space. You may find Zelig ~Imai, leading scholars in the field contain er- 5. Clarify with precision the extent to King, and Lau 2004! or Clarify ~Tomz, rors, confusions, lack of essential infor- which you were able to replicate the Wittenberg, and King 2003! software mation about how the analysis was author’s results. If you can’t replicate the helpful in calculating new quantities of conducted, and other problems. Some of author’s results even with the help of the interest from the same model. these issues do not matter to substantive author that is important information that 8. Next, you should run some con- conclusions, and some do, but all make needs to be on the public record, but it trolled methodological experiments de- replication more difficult. As such, com- also means you can’t build on this work signed to advance the state of knowledge pleting the replication will likely be to make further progress. And if you about the substantive project. That is, more troublesome and time consuming can’t find out what the problem is, it make one improvement, or the smallest than you anticipate ~even after you adjust might mean that you do not have a pub- number of improvements possible to pro- for the information in this sentence!!. lishable paper and so might need to start duce new results, and show the results so After you have done everything you can with a different article. So try hard, and that we can attribute specific changes in do on your own, you may need to con- you may have to try very hard, to substantive conclusions to particular tact the author of the article ~please do replicate. methodological changes. ~Improvements so respectfully and diplomatically!. 6. Unlike almost all previous papers can include changing the way the author ~The remarkable difficulties students you may have written, do not allocate dealt with missing data, selection bias, have in replicating published articles space in your paper in proportion to how omitted variable bias, the model specifi- teaches more about the state of the litera- much work you put in accomplishing cation, differential item functioning, the ture, and conveys more about the some- each task. The point of this paper is to functional form, etc., adding control vari- times shaky foundations of academic make your scholarly point, not to show ables or better measures, extending the knowledge, than reading all the pub- how smart you are. This paper should time series and conducting out-of-sample lished literature one person could possi- not be about you or a report of what you tests, applying a better statistical model, bly consume on his or her own. Every did; it should be about what you contrib- etc.! If you are able to produce an inter- year some students are incredulous or ute to our collective knowledge about the esting substantive result that is different stunned by what they find; the experi- world. For example, a large fraction of from the original article, with only one ence is in part disheartening, but it also your effort will probably go into replicat- completely justifiable methodological seems to empower students who ~cor- ing a prior result ~and thus getting up to change, then you only need to defend rectly! conclude that they can do better. I the cutting edge of the field!, but only in this change fully and carefully. typically devote some time during class rare cases will that take more than a 9. If you are able to improve or to share these experiences.! page or two of your paper. Space in your change the author’s results in some im- 3. Please bring me a copy of the arti- paper should be allocated in proportion portant way with the minimal change cle you choose and ask for my views to how much of a contribution it makes necessary ~and that is maximally justifi- before proceeding. This will generate to changing the minds of someone in the able!, write that up separately. Then, in a advice on what is unlikely to work, and literature about something important. separate section, go ahead and make all might be useful for other reasons, but to Thus, for example, if at the end of week the changes you think are desirable and be clear it is no guarantee that you will of data analysis you make one important see what difference that makes to your be able to replicate the work chosen and finding on last day that would add to or results. But make sure the minimal successfully complete the assignment. change the conventional wisdom about a changes necessary to produce the new Your assignment is to pick an article ac- subject, then you should change the title, conclusions are described and justified cording to the criteria above and to repli- subject, abstract, introduction, and orga- first with results fully presented. Once cate it. The choice of the article is part nization of your paper to focus on this you’ve done that, then you’re home free of the assignment and so, just as happens finding. All your other efforts that, de- in your quest for journal space. The rest to faculty researchers, you may need to spite your best efforts, led to dead ends are further improvements that you will change your choice of topic along the should be excluded from your paper un- have much more free reign to explore as

120 PS January 2006 you see fit. But if it turns out that all will have the side benefit for the rest of pose of that memo is not to destroy these other changes don’t change any us of making you a lot more fun to hang anyone’s work, but to improve it. substantive conclusions, then leave them out with. It’s also not fair that some 6. Do not ask the author of the pub- out or report on them very briefly. came to this class with better math lished article whose work you are repli- 10. Provide evidence that your model skills, or get to ski more often than I do, cating for comments on your paper, and fits the data in and out of sample ~or but such is life. A normative standard do not share it with him or her, or any- perhaps that it fits better than the model that is much more in your career interest one outside of this class, until I have in the original article!. For example, are is to ask yourself instead only: Is your read it and you have revised it accord- the probabilistic assumptions implied by coauthor making a positive contribution ingly. Why? In all likelihood this experi- your model correct? Do 95% of the data to your paper? If it’s a positive contribu- ence will be your first interaction with points fall within the 95% confidence the outside world as a professional aca- tion, then you’re getting something out intervals? Are you able to predict a set- demic and, like all academics, as a cer- of your collaboration. Be thankful. aside test set from the rest of the data tain kind of public figure. The academic with the predicted level of uncertainty? world has highly specific, and often un- You can test this for regression models, 2. The authors on your paper should stated, expectations about a whole host logistic models, and all other models. be listed alphabetically, which ~in most of matters you may not now perceive, The basic idea is the same as any scien- social sciences! conveys equal contribu- and authors tend to be very sensitive tific analysis: look for as many observ- tions, or that everyone was a full-fledged about what you write and how you write able implications of your theory as member of the research team. Customs it, especially if you find something even possible and to check those ~King, Keo- in public health and medicine, and in slightly wrong with what they did. You hane, and Verba 1994!. some other areas, usually give most can avoid a lot of trouble with a quick 11. Understand your raw data prior to credit to the first-named author, but it is reality check. So please come by first. statistical modeling, and help your read- possible even in those fields to convey to 7. After the paper is revised ~for sub- ers do so. Include graphics or descriptive readers that contributions were equal, stance and style! to my satisfaction and statistics to help in this goal. Giving such as via explanatory footnotes ~e.g., yours, it will be much safer for you to some concise sense of the data while you “authors were listed alphabetically”!. Get go public, and going public then is es- are describing the variables is a useful these issues out of the way quickly so sential. The procedure is, before you space saving device. they don’t affect your work. show it to anyone else outside this class, 3. Papers should be no longer than send a copy to the author of the work Ground Rules about 20 pages ~double-spaced, one-inch you’re replicating or critiquing and re- margins, 12pt, including figures, tables, spectively request comments. When you 1. The paper must be coauthored with and references!. Think in terms of a receive a response, you should revise, another member of this class. short research note, not a full-length arti- being as generous as possible, but only cle. Journal space is scarce and so the as you think is appropriate. Only at that ~a! Why? Although papers are rarely longer the paper you write, the harder it point should you post the paper on your coauthored in school, almost half of all will be to publish. If you can do it in 10 web site and make it fully public, which political science articles are, which is a pages, so much the better. you certainly should do. If your contri- seven-fold increase since the 1950s 4. In addition to coauthoring your bution still stands, in your view, after ~Fisher et al. 1998!. This class is about paper, get others to give you written receiving comments from a wider audi- research as it is actually practiced in the comments on a draft version. Why? The ence, you should then consider submit- field. reason academics hang out together in ting the paper to a scholarly journal or ~b! What if your coauthor doesn’t carry universities is not ~necessarily! because presenting it at a conference. For infor- we like each other; it’s because our work mation about where to submit your paper his or her weight? Deal with it some- gets better in the process of interacting. and how to do it, come by and we’ll talk how, and make your best individual ef- When you graduate, you will need to about it. fort even if it is asymmetric. You will build a network of people who will care- have to deal with this when you gradu- fully read your work before you distrib- Style ate too. Your goal ~and given task! is to ute it widely; students in this class often make your paper as good as possible, form the start of that network for each 1. Your paper should be rigorously and you have at your disposal your ef- other. But it is an implicit quid pro quo: structured and organized into sections fort and whatever effort you can marshal If you want others to read your work, and subsections. ~Heading titles should from your coauthor. In most of the so- make sure to give them detailed com- be clear, contain no acronyms, and cial sciences, credit is not divided ments too. should summarize the key point you are among the coauthors: each coauthor gets 5. We provide a formal way to provide making in the section. They may be almost full credit for the entire paper.4 you some advice along the way: In class, numbered, but the numbers should be in As long as you’re getting credit for what you will turn in a very early draft of your addition to a substantive title.! The best you’re doing, it doesn’t hurt you for paper with the tables and figures in near way to understand how to organize a someone else to have more credit than final form but relatively little text. You’ll paper is to imagine that your readers will he or she deserves. also turn in a replication data set, just as randomly fall asleep at any time for five faculty routinely do. We will then give minutes and yet keep turning pages; ~c! But, some might scream, “it’s not this to another student, who will try to when they wake up, they should know fair to share credit equally!” With all the replicate your results ~without talking exactly where they are from your sub- time and mental energy you could de- with you!. That student will then write a headings alone. Something like this will vote to developing normative standards memo to you about your paper, with surely happen: Think about what you do to apply to your collaborators, you could copy to me. In science, we compete to when you read long, boring textbooks. write another article. That would be a lot advance knowledge about the world, not You are writing for anonymous refer- better for you and your career, and it to tear each other down. Thus, the pur- ees. Referees are busy people looking

PSOnline www.apsanet.org 121 for a way to finish the thankless ~anony- 5. Front matter Why? Quality may be everything, but it mous!! task of reviewing your paper as is hard to measure and so style provides ~ ! your quickly as possible. Since you’re not a Your title should convey key important signals. For example, as a likely to have as much time from them point by summarizing clearly your argu- purely predictive matter, papers format- as you think, you need to make reading ment or angle. An appropriate title is not ted with LaTeX are much less likely to your paper as easy as possible. And in a list of topics or “the effect of A on B.” contain egregious methodological flaws. ~ this game a tie doesn’t go to the runner: Quite like haikus which I recommend Similarly, you should run the paper ! If a referee didn’t read carefully, pay reading and writing for practice , writing through a ssppeelllliinngg checker. Use attention, or understand you, or missed titles takes considerable time, effort, at- the appearance of your paper to your or misunderstood something important tention, and thought. advantage. in your paper, it is your fault. And since it is your paper and not you that mat- ~b! Include a footnote on the title page to ~b! Follow the same rules used for pre- ters, anonymous referees will not ~and the title, and put the text of the footnote paring convention papers: Use 12pt for the sake of the literature normally at the bottom of the first page. In it, put double-spaced black text on white 8.5 ϫ should not! give an anonymous paper your contact information, where others 11 inch paper with a staple in the upper writer the benefit of the doubt. Anony- can get your replication data set, and ac- left corner—no polywhatever colored mous referees are not normally prone to knowledgments to everyone who helped plastic covers. The paper should have a spontaneous generosity and do not gen- you with this paper. There is no cost to title, your name, affiliation, and the class erally impute favorable motives to au- being generous with acknowledgments. number. For examples, see the preprints thors who are not clear or impute Be sure to thank everyone who read the at http:00GKing.Harvard.edu0preprints. appropriate assumptions when you leave paper for you, including students who shtml. them unstated. ~And this business is no read it for a class assignment, anyone worse than any other: Human beings who you discussed it with, those who ~c! Follow this style for references acting anonymously in other circum- helped you solve computer or method- “~Beck 1985!,” etc., of the American stances tend not to be especially nice ological problems, or anyone who pro- Political Science Review. ~Why? This is either. You may have noticed that cars, vided you data. If you had any contact the most common style in the discipline which have drivers’ identities mostly with the authors of the article you’re rep- and is quite common in other disciplines obscured, cut each other off all the time, licating, be sure to acknowledge them as well.! If you want details, see the in- but this almost never happens walking too. structions to contributors in the journal down the same streets.! you are writing for, but most of these 2. The overall structure of the paper, ~c! Include a one-paragraph abstract, no instructions should be ignored until ac- and all the key points you want to make, longer than 150 words, on the page fol- ceptance since they are designed to should make sense in terms of accom- lowing the title page. The purpose of the plishing your goal. If you include many make things easier for copyeditors, not title is to convey your entire point in one reviewers. section breaks it is easier for your read- phrase, and to convince people to read ers to skip over things they are not inter- the abstract. The point of the abstract is ~d! Avoid gratuitous citations to your ested in while still getting your point; if to drive home to readers your main con- professors ~or anyone else!. You are wel- you include too few, they will get lost, tribution and “who’s mind you’re going come to tell others how wonderful they and so will your chances at publication. to change about what.” It should contain are in person, but keep this out of your 3. While writing, keep revising the list papers. Cite only those whose research of section headings until it looks like a all relevant information about the impor- you use or build on in some way. ~Do table of contents that conveys your key tance of your work and who should read not leave out those likely to be your re- point well even if one does not read the it, and not much else. Reading it should paper. make you want to read the paper. By at viewers either; no one likes to be 4. Do not try to hide weaknesses in least three weeks before the paper is ignored.! your paper. If you know of a problem due, send a copy of the title and this ~e! The paper should be a formal presen- with your analysis that you have not abstract to the class mailing list to get tation, not a personal letter. Occasional solved, clearly delineate the problem. If comments from me and others. You may humor is fine, but inside jokes or ques- you think the problem is not that bad, send more than one version, and I may tions are best left out. Raw computer explain why, but do so honestly. If you ask for revisions. This step improves the output should not be included. have an idea of how to solve it, but paper substantially by helping you to haven’t done so, offer it as a suggestion clarify the paper’s primary contribution 7. In the text, identify the specific em- for future researchers. If you don’t know and to focus the paper on it. Once the pirical question you are interested in im- how to solve it, suggest that future re- title and the abstract are set, the entire mediately and get to it. Beginnings such searchers try to tackle it. rest of the paper is likely to be affected. as “In this paper, we demonstrate that Why do you need to be so forthright Be sure to read all the abstracts and . . .” are favored. If it is not clear to a with potential problems? If a reader sees comments; this process is often ex- reader what you are planning to accom- a problem you didn’t mention, you’re tremely informative about how to write plish with some specificity ~including making it possible for them to say “not what your dependent variable is! after a papers, about what is important, and only didn’t the author correct this prob- few pages, something is wrong. As about how to make the findings in your lem, but he or she didn’t even realize it someone once said, if the first bite of an was a problem!” If all you do is to note paper important. apple tastes bad, you don’t keep taking the problem, you can take the edge off 6. Appearance bites to see whether some other part of it the criticism. This is of course as it might be better. should be, since your paper will also be ~a! Prepare this paper as if it were to be 8. In almost all cases, do not include a a more appropriate scientific statement of submitted for formal review at a profes- section titled “literature review,” and any the problem. sional journal. literature review you include should be

122 PS January 2006 short and directed toward your point a suggestion, I recommend LaTeX, draw readers’ attention to these points. only. Other people don’t deserve to be which is a standard in most mathemati- They are not included to demonstrate cited in your paper unless they help you cally oriented fields, is used by many that you did something. They are not make your point; they already have their publishers, is free, and is available for obligatory every time you run a regres- own papers. If prior literature doesn’t most operating systems. Startup costs sion, for example. Readers will interpret help you make your key point, omit it. are higher and so it is not for everyone, your including a table or figure about a 9. If you have long technical lists of but once you know how to use it is a lot point as your judging it important. So coding rules, or anything else that seems faster than MS Word or other WYSI- choose carefully what you want to dis- essential but distracts from your point, WYG programs. Introductory material play in this way. put it in an appendix. can be found at my homepage. 10. Be nice ~b! All tables and figures should be sep- ~a! Treat authors you are replicating as ~b! Equations included in papers should arately and fully documented. Someone you would want to be treated. Your goal not be treated like figures. They are part reading only them, without the paper, is to stand on the shoulders of the schol- of the sentence structure and so should should be able to understand what is ars whose article you are replicating, not include punctuation, etc. The equal sign going on. Adding an explanatory para- to step on their faces. In all likelihood, is the verb in math, and so we say that graph at the bottom of each figure or 2 ϩ 2 ϭ 2 there is a good reason why these people a b c ~and read it as “a squared table is usually necessary to accomplish did whatever they did. Remember that if plus b squared equals c squared.”!.Or this. Similarly, someone who reads the all goes well, you will one day be in we explain that 5 is the result of evaluat- paper and ignores the table or figure their shoes. ing 7-2. ~Note also the periods at the should also be able to follow it all. The end of the previous two sentences and point of the text is to walk the reader by ~b! Talk about the article you are repli- the punctuation at the end of the hand through the table or figure so it cating, not about the authors.Soyou Equation 1.! is easy to understand. Picking out one should write “Jones and Smith ~2003! is number in the table and explaining it in mistaken” rather than “Jones and Smith ~c! For numbers, use only as many deci- detail at the outset as an example is are mistaken.” mal places as you have precision. Nor- mally one or two digits to the right of often a good strategy. ~c! Do not be personal and be careful of the decimal point is enough, but the the language you use to describe your ~c! Do not add lines between every col- right number depends on the context. work. Remember that it doesn’t matter umn and row in your tables ~as is the Why? Think about the advice you would whether you “agree” with the author you usual default in programs like Excel and give to the local weather forecaster who are replicating; no one cares what you Word!, and do not include superfluous predicts that tomorrow’s temperature will “think”; no one is interested in your shading of various sorts. If you have a be 37.828280019277647381 degrees “opinion”; and readers don’t want to column of percentages or dollars, only Fahrenheit. Your standard errors provide know what you “believe.” In your paper, the item in the first row should have a a guide: If you present more digits after you do not matter; the scientific commu- % or $ sign. In columns of numbers, the your decimal point than your standard nity only cares what you can decimal points should align vertically, errors indicate you can measure accu- demonstrate. and do not use proportionally spaced rately, then you are filling your paper fonts. 11. Use active ~“We ran a least with numbers created by rounding error. squares regression.”! rather than passive ~This silliness is not uncommon: if stan- ~d! Make tables and figures only as large ~“A least squares regression was run.”! dard errors indicate that coefficients are as they need to be; in most cases, small tense. Why? This is not only good gram- accurate to 2 digits and 4 digits of accu- is beautiful so long as they can be read mar and easier to understand; it is a mat- racy are presented, then a majority of ~including by people over 50!. Remem- ter of standing behind what you’re doing the numerals printed in the table are to- ber that journal space is valuable. Try to and sounding like you mean it. tally irrelevant.! 12. Math keep tables and figures oriented as the text is ~profile rather than landscape! so ~a! Don’t let your word processor con- ~d! A sophisticated reader must be able to write down your statistical model and the reader doesn’t have to keep turning trol the look, style, or content of your the paper around. It is much easier for paper. For example, mathematics in En- likelihood function ~or other method of estimation and analysis! from reading the reader if the tables and figures are glish text is, by scholarly convention, ~ your text. You can convey this by writ- run into the text floated to the top of always in italics, and Greek letters ! ing down the precise form of your the page rather than collected at the end should be in Greek. Math should not be of the paper or on separate pages or bro- ϭ ϩ * model, but do not derive or reproduce written as yhat beta-b gamma-w X ken between pages. b w equations that are well known unless ϩe, but rather y[ i ϭ b ϩ g Xi ϩ ei. Larger equations should be set with you cannot otherwise explain precisely ~e! In most cases, if you can present the equation numbers, and be referred to as what you did. For example, saying that same information either way, a figure is with this example of Bayes Theorem, you used even something as simple as a better than a table. negative binomial regression model P~ y6u!P~u! would require clarification since the ~f! As in the journals, number the fig- P~u6y! ϭ , ~1! P~ y! variance function can differ across soft- ures consecutively, and separately num- ware implementations. ber the tables consecutively. Refer to where u is an unknown parameter and y 13. Tables and Figures each in the text by number ~e.g., see is a data vector. Figure 4!. The total number of tables Many word processors can do this ~al- ~a! Tables and figures should be in- and figures in your paper should proba- though the defaults often look lame!; for cluded to make specific points, and to bly be a single digit number.

PSOnline www.apsanet.org 123 ~g! Follow a no dumping rule. That is, 3. Quantities of Interest of inference. You talk about your statisti- each table and figure should be pre- ~a! If you run some analysis, don’t cal model, and then say you used ML to estimate it ~if you did!. Regression is sented and discussed in the text in turn. present long lists of coefficients that are ML, but it is more commonly understood You should not casually refer to Fig- ~or anything else that is! hard to inter- as regression. ures 1–12 and then go on to the next pret. Instead, compute the precise quan- subject. Explain in detail what you want 5. Don’t include control variables in a tity of interest ~and a measure of model that are consequences of the readers to see in them. Remember that uncertainty! that best helps you make what is obvious to you after looking at causal effect you are trying to estimate your substantive point. ~If you feel you ~which is known as post-treatment bias!. these for weeks will probably not be must add the lists of coefficients to the obvious to anyone else. Similarly, if you This is an important point that is often paper, add them as appendices so they missed. To estimate two causal effects find yourself preparing big tables with can be skipped easily.! No one cares usually requires estimating separate mod- lots of numbers and only talking about a about numbers that even the author els for each; although it may be possible few, then you should rethink your doesn’t want to interpret, and so these to include both variables in the same strategy. should not waste space in your paper. regression, the coefficients cannot be ~h! When explaining the content of a ~b! Don’t write a paper that has a long interpreted as causal effects unless you are careful about this point. See King figure, it is good practice to devote one buildup to an estimation and then have ~1991! and King and Zeng ~2006! on this paragraph to the setup—the horizontal one table and a paragraph that summa- point. and vertical axis measurements, the unit rizes all your work. Spend time carefully 6. Examples: A full-length replication of analysis, etc.—and then to start a new interpreting your results in substantive can be found at King and Laver ~1993!. paragraph that explains your results. terms, in terms that a non-quantitative Other shorter examples can be found in political scientist would understand. The King, Tomz, and Wittenberg ~2000! and goal ought to be to satisfy someone Problems to Avoid and King et al. ~2001!. quantitative ~by doing the statistics right! Other Suggestions 7. Provide sufficient information about and a smart non-quantitative type ~by your analysis so that it is possible for 1. Decisions about what to present fully explaining things in sufficient de- someone who reads your paper to repli- should be made by you, not by computer tail! in the same paper. cate the analysis. This means that you programs. You have been given the tools ~c! Make sure all point estimates come must be very precise about coding rules, in this class to create your own statistical with some measure of their uncertainty, where the data came from, how indices models, perform your own simulations were computed, what the unit of analysis such as confidence intervals, posterior and to calculate a quantity of interest is, etc. A class exercise will include an- distributions, or standard errors. from any model, if need be. The fact that other student replicating a draft of your R, Zelig, Clarify, or some other program ~d! Do not say that quantities are “statis- work, but be sure the final version can does not do what you want is not a rea- tically significant” unless you have a be replicated too. Likely the only way son not to do it. very good substantive reason to do so you will be able to continue to revise the 2. Don’t do things like this: ~hint: you probably don’t!!.Inmost paper for publication after class is over is cases, this is unhelpful information that to prepare a replication data set now, • Hypothesis 1: The effect of . . . distracts from the substantive purpose at while the work is fresh in your mind and • Hypothesis 2: Instead, the effect hand. Calculate your quantity of interest all the data, files, and code you used are of... by giving a posterior density, confidence still available. You now know how hard interval, or point estimate and some it has been to replicate someone else’s It looks very scientific, but you want measure of uncertainty. Once you’ve work; don’t make the same mistakes. your points emphasized, not words in presented all that, you have conveyed all If you reach the stage of publishing everyone else’s paper ~like “hypothesis”!. this paper, be sure to prepare a final ver- that your data have to say about your If you want to emphasize something, sion of a replication data set and make it quantity of interest; what more would emphasize your point this way. Number- publicly available, such as by submitting ing hypotheses and using scientific you want to know? it to the ICPSR’s Publication Related sounding words also doesn’t usually help 4. Don’t claim to be using “the maxi- Archive. you make your point. mum likelihood model.” ML is a method

Notes * My deepest appreciation goes, in addition shtml. The course is taken by undergraduates 3. If you have a topic that has never before to my students, to the numerous scholars who and graduate students from the Government been addressed, it is still best to begin with have cheerfully, and in some cases repeatedly, Department and a variety of other departments the closest article to your area. Similarly, major responded to my students’ queries over many and schools. An important feature of the class new data collections, while highly desirable years. Thanks also to the National Institutes of for undergraduates is that they are treated just generally, are likely to take longer than the Aging ~P01 AG17625-01! and the National Sci- like graduate students. The graduate students time available in a class project and so should ence Foundation ~SES-0318275, IIS-9874747! have more wisdom about the literature and be avoided for the purposes of this paper. for research support. what constitute important questions, but the Even if you ultimately plan a major data 1. See King ~2003! and http:00GKing. undergraduates often have better mathematical collection project, replicating an article at the Harvard.edu0replication.shtml for more informa- backgrounds or other useful skills. In my cutting edge in the literature is usually an tion on the replication and data sharing experience, the two groups often mesh well excellent place to start. You will learn what movement in political science and other fields. together, compete successfully, and can is lacking and what might be fixed by your 2. The class is Government 2001 at Harvard make great coauthor teams, about which more data collection project. You may also be able University. See http:00gking.harvard.edu0class. below. to gather convincing evidence for potential

124 PS January 2006 funding agencies before you invest a great deal come up for tenure, try to coauthor with differ- appropriately without collinearity. You should of time. ent people. That way, if there is any question as probably limit the total number of coauthors on 4. Some partial qualifications: Across all the to your contribution, it will be easy to control each paper to three or four when possible. articles you write after this class and before you

References Fisher, Bonnie S., Craig T. Cobane, Thomas M. gking.harvard.edu0files0abs0truth-abs. King, Gary, and Michael Laver. 1993. “On Vander Ven, and Francis T. Cullen. 1998. shtml. Party Platforms, Mandates, and Government “How Many Authors Does It Take to Publish _. 1995. “Replication, Replication.” PS: Spending.” American Political Science an Article? Trends and Patterns in Political Political Science and Politics 28~Septem- Review 87~September!: 744–750. http:00 Science.” PS: Political Science and Politics ber!: 443–499. http:00gking.harvard.edu0 gking.harvard.edu0files0abs0hoff-abs. 31~4!: 847–856. files0abs0replication-abs.shtml. shtml. Gleditsch, Nils Petter, Claire Metelits, and Ha- _. 2003. “The Future of Replication.” In- King, Gary, Michael Tomz, and Jason Witten- vard Strand. 2003. “Posting Your Data: Will ternational Studies Perspectives 4~February!: berg. 2000. “Making the Most of Statistical You be Scooped or Will You be Famous?” 443–499. http:00gking.harvard.edu0files0abs0 Analyses: Improving Interpretation and Pre- International Studies Perspectives 4: 89–97. replvdc-abs.shtml. sentation.” American Journal of Political Gleditsch, Nils Petter, Patrick James, James Lee King, Gary, James Honaker, Anne Joseph, and Science 44~April!: 341–355. http:00 Ray, and Bruce Russett. 2003. “Editors’ Joint Kenneth Scheve. 2001. “Analyzing Incom- gking.harvard.edu0files0abs0making- Statement: Minimum Replication Standards plete Political Science Data: An Alternative abs.shtml. for International Relations Journals.” Inter- Algorithm for Multiple Imputation.” Ameri- King, Gary, Robert O. Keohane, and Sidney national Studies Perspectives 4: 105. can Political Science Review 95~March!: Verba. 1994. Designing Social Inquiry: Sci- Imai, Kosuke, Gary King, and Olivia Lau. 2004. 49–69. http:00gking.harvard.edu0files0abs0 entific Inference in Qualitative Research. “Zelig: Everyone’s Statistical Software.” evil-abs.shtml. Princeton: Princeton University Press. http:00gking.harvard.edu0zelig. King, Gary, and Langche Zeng. 2006. “When Tomz, Michael, Jason Wittenberg, and Gary King, Gary. 1991. “ ‘Truth’ is Stranger than Pre- Can History Be Our Guide? The Pitfalls of King. 2003. “CLARIFY: Software for Inter- diction, More Questionable Than Causal In- Counterfactual Inference.” International preting and Presenting Statistical Results.” ference.” American Journal of Political Studies Quarterly. http:00gking.harvard.edu0 Journal of Statistical Software 8~1!. http:00 Science 35~November!: 1047–1053. http:00 files0counterf.pdf. gking.harvard.edu0stats.shtml.

PSOnline www.apsanet.org 125