Comparison of Review Articles Published in Peer-Reviewed and Throwaway Journals

REVIEW ARTICLES IN PEER-REVIEWED AND THROWAWAY JOURNALS

tongue and in English might have been statistical advice and Arnstein Finset, PhD, for advice Rundsch. 2000;40:50-54. on sex and age aspects of memory. 4. Delamothe T, Smith R. PubMed Central: creating even larger in a representative sample an Aladdin’s cave of ideas. BMJ. 2001;322:1-2. 5. Ingebrigtsen T, Romner B, Kock-Jensen C. Scan- of family physicians. REFERENCES dinavian guidelines for initial management of mini- 1. Sousa Escando´ n MA, Gonzalez Guitia´ n C, Gonza- mal, mild, and moderate head injuries. J Trauma. 2000; Author Contributions: Study concept and design, lez Ferna´ndez MM. Which language will Medline speak 48:760-766. analysis and interpretation of data, and critical revi- in the next millenium [in Spanish]? Arch Esp Urol. 2000; 6. Richardson JTE. Gender differences in imagery, cog- sion of the manuscript for important intellectual con- 53:93-99. nition, and memory. In: Logie RH, Denis M, eds. Men- tent: Gulbrandsen, Schroeder, Milerad, Nylenna. 2. Nylenna M, Hagve T-A. Small journals and non- tal Images in Human Cognition. Amsterdam, the Neth- Acquisition of data: Gulbrandsen, Schroeder, Milerad. English language journals. In: Godlee F, Jefferson T, erlands: Elsevier; 1991:271-303. Drafting of the manuscript: Gulbrandsen. eds. Peer Review in the Health Sciences. London, En- 7. Hyde JS, McKinley NM. Gender differences in cog- Statistical expertise: Gulbrandsen, Schroeder. gland: BMJ Books; 1999. nition: results from meta-analyses. In: Caplan PJ. Gen- Obtained funding: Schroeder, Milerad, Nylenna. 3. Beller FK. The future of German language in der Differences in Human Cognition. New York, NY: Acknowledgment: We thank Eva Skovlund, PhD, for science [in German]. Gynakol Geburtshilfliche Oxford University Press; 1997:30-51.

Comparison of Review Articles Published in Peer-Reviewed and Throwaway Journals

Paula A. Rochon, MD, MPH Context To compare the quality, presentation, readability, and clinical relevance of Lisa A. Bero, PhD review articles published in peer-reviewed and “throwaway” journals. Ari M. Bay, BSc Methods We reviewed articles that focused on the diagnosis or treatment of a medical condition published between January 1 and December 31, 1998, in the 5 leading Jennifer L. Gold, BSc peer-reviewed general medical journals and high-circulation throwaway journals. Re- Julie M. Dergal, MSc viewers independently assessed the methodologic and reporting quality, and evaluated each article’s presentation and readability. Clinical relevance was evaluated in- Malcolm A. Binns, MSc dependently by 6 physicians. David L. Streiner, PhD Results Of the 394 articles in our sample, 16 (4.1%) were peer-reviewed system- Jerry H. Gurwitz, MD atic reviews, 135 (34.3%) were peer-reviewed nonsystematic reviews, and 243 (61.7%) were nonsystematic reviews published in throwaway journals. The mean (SD) quality HROWAWAY” JOURNALS ARE scores were highest for peer-reviewed articles (0.94 [0.09] for systematic reviews and characterized as journals that 0.30 [0.19] for nonsystematic reviews) compared with throwaway journal articles (0.23 Ͻ contain no original investiga- [0.03], F2,391=280.8, P .001). Throwaway journal articles used more tables (P=.02), tions, are provided free of figures (P=.01), photographs (PϽ.001), color (PϽ.001), and larger font sizes (PϽ.001) charge,T have a high advertisement-to- compared with peer-reviewed articles. Readability scores were more often in the col- text ratio, and are nonsociety publica- lege or higher range for peer-reviewed journals compared with the throwaway jour- 1 1 nal articles (104 [77.0%] vs 156 [64.2%]; P=.01). Peer-reviewed article titles were tions. Large circulations and reader- judged less relevant to clinical practice than throwaway journal article titles (PϽ.001). ship polls2 suggest that throwaway journals are more widely read than some Conclusions Although lower in methodologic and reporting quality, review articles published in throwaway journals have characteristics that appeal to physician readers. peer-reviewed journals in the same sub- JAMA. 2002;287:2853-2856 ject areas. Despite their popularity, www.jama.com throwaway journals are judged dispar- agingly as a source of “instant cook- in a sample of peer-reviewed journals medical conditions published in 5 lead- book medicine”3 and journals that are compared with those published in a ing peer-reviewed general medical jour- given away.4 Indeed, throwaway jour- sample of throwaway journals. nals (Annals of Internal Medicine, BMJ, nal articles1 are seldom peer reviewed and JAMA, The Lancet, and New England are almost never cited in the medical lit- METHODS Journal of Medicine) and the throw- erature. They are considered to be of poor We identified all review articles that fo- away journals with the highest circu- quality compared with peer-reviewed cused on the diagnosis or treatment of lation5 (Consultant, Hospital Practice, journal articles, despite the lack of for- 1 Author Affiliations: Kunin-Lunenfeld Applied Re- Bero); Department of Psychiatry, University of Toronto, mal quality comparisons. Given the suc- search Unit (Drs Rochon and Streiner, Mr Bay, and Mss Toronto, Ontario (Dr Streiner); and Meyers Primary Care cess of throwaway publications, we Gold and Dergal) and Rotman Research Institute (Mr Institute, Fallon Healthcare System, and University of Binns), Baycrest Centre for Geriatric Care, Department Massachusetts Medical School, Worcester (Dr Gurwitz). sought to understand why so many phy- of Medicine, University of Toronto and Institute for Clini- Corresponding Author and Reprints: Paula A. Ro- sicians read them. We assessed the qual- cal Evaluative Sciences (Dr Rochon), Toronto, Ontario; chon, MD, MPH, Kunin-Lunenfeld Applied Research Department of Clinical Pharmacy, School of Phar- Unit, Baycrest Centre for Geriatric Care, 3560 Bathurst ity, presentation, readability, and clini- macy, and Institute for Health Policy Studies, School of St, Toronto, Ontario, Canada M6A 2E1 (e-mail: paula cal relevance of review articles published Medicine, University of California, San Francisco (Dr [email protected]).

Downloaded From: https://jamanetwork.com/ on 09/30/2021 REVIEW ARTICLES IN PEER-REVIEWED AND THROWAWAY JOURNALS

Patient Care, and Postgraduate Medi- index generates scores from 0 to 100 RESULTS cine) between January 1 and Decem- (higher scores indicate easier read- Of the 394 articles in our sample, 16 ber 31, 1998. A 3-stage process was ing); a score of 30 or lower was asso- (4.1%) were classified as peer- used to identify clinically relevant re- ciated with a college-level reading abil- reviewed journal systematic review ar- view articles for inclusion in our sample. ity. The Gunning FOG index scores also ticles, 135 (34.3%) as peer-reviewed First, we identified sections of each reflect reading difficulty (lower scores journal nonsystematic review articles, peer-reviewed journal that published re- indicate easier reading); a score of 17 and 243 (61.7%) as throwaway journal view articles. Throwaway journals have or more was considered too difficult for review articles. Most peer-reviewed ar- no designated review sections; there- medical writing.11 ticles (n=126, 83.4%) were classified by fore, we identified sections most likely Six physicians who were recent gradu- MEDLINE as tutorial reviews. System- to contain review articles. Second, we ates in full-time clinical practice (see “Re- atic reviews were published exclu- excluded all review article subsec- view Article Study Group”) indepen- sively in the peer-reviewed journals. tions where the primary focus was not dently rated the clinical relevance of all Quality scores were highest for the on the clinical diagnosis or treatment 394 articles in 2 ways. First, physicians 16 systematic review articles. The mean of a specific medical condition or sec- blinded to the journal name read a com- (SD) quality score was 0.94 (0.09) for tions that published only case studies. puter-generated random list of all ar- the peer-reviewed systematic review ar- Two authors (A.M.B. and P.A.R.) ex- ticle titles and indicated their agree- ticles compared with 0.30 (0.19) for the cluded 68 peer-reviewed and 72 throw- ment (1=strongly disagree; 5=strongly peer-reviewed nonsystematic review ar- away journal articles that did not meet agree) to 2 statements: (1) this article ticles and 0.23 (0.03) for nonsystem- our inclusion criteria. Our cohort in- may provide useful information for my atic review articles published in throw- Ͻ cluded 394 review articles. practice, and (2) I would consider read- away journals (F2,391=280.8, P .001). Each article was classified as either ing this article. Second, the reviewers Peer-reviewed journal articles pro- a systematic or a nonsystematic re- evaluated the clinical relevance of all 30 vided significantly more references than view. To identify systematic reviews, we heart disease articles (heart disease was throwaway journal articles (53.6 [36.8] used an approach based on the com- one of the most frequent topics). The vs 14.4 [11.6]; PϽ.001). prehensive search strategy outlined by physicians independently read each ar- As outlined in TABLE 1, throwaway Hunt and McKibbon.6 Two trained re- ticle and used the scale to respond to the journal articles were more likely to use viewers (J.L.G. and Y.C.K.) indepen- following statements: (1) the article ad- tables, figures, color, and larger font size dently evaluated methodologic and re- dresses an important issue; (2) the topic compared with review articles pub- porting quality using the Barnes and is of interest to me; (3) the topic is rel- lished in peer-reviewed journals. Bero7 quality scoring assessment tool. evant to my practice; (4) the article pro- Among the 378 nonsystematic review This instrument is a modification of the vides practical strategies for physicians articles, 228 throwaway journal ar- Oxman et al8,9 and Mulrow10 instru- such as myself; and (5) I will use the in- ticles (93.8%) used color compared ments. The quality score was based on formation to help care for patients. with only 77 (57.0%) of the peer- 12 questions that evaluated the pur- Tables were also evaluated. reviewed journal articles. All of the pose of the review, review strategy, in- The quality scores obtained by the 2 throwaway journal articles and none of clusion and exclusion criteria, quality reviewers were very consistent; hence, the peer-reviewed journal articles used assessment, combining of study re- the quality score assigned was the mean a large font size. Articles published in sults, summarizing of study findings, score. To evaluate the clinical rel- throwaway journals were judged to be limitations, and support provided for evance of all of the 394 review article easiest to read. Among the 378 non- conclusions. Each question was scored titles and the subset of the 30 heart dis- systematic review articles, the mean as 0 (no), 1 (partial), or 2 (yes). The ease articles, we calculated the mean Flesch score was significantly higher in final score was a percentage, in which score obtained from the 6 physician re- throwaway journal articles than in the higher scores indicate better quality. As viewers. Differences in continuous vari- peer-reviewed journal articles (23.7 an additional measure of quality, we ables among the 3 types of articles (ie, [15.4] vs 15.8 [17.7]), indicating that counted the references cited. peer-reviewed systematic review, peer- throwaway journal articles were easier Presentation was evaluated using the reviewed nonsystematic review, and to read (PϽ.001). More scores were in article’s font size (ie, small or large), use nonsystematic review articles pub- the college level or higher range for the of color, and numbers of tables and fig- lished in the throwaway journals) were peer-reviewed journal articles com- ures. To quantify readability, we used compared using analysis of variance. We pared with the throwaway journal ar- 2 validated readability formulas11: the used ␹2 tests to assess differences in cat- ticles (104 [77.0%] vs 156 [64.2%]; Flesch reading ease index12 and the egorical variables. Analyses were per- P=.01). Using the Gunning FOG in- Gunning Frequency of Gobbledygook formed using SPSS version 10 (SPSS Inc, dex, mean (SD) scores were signifi- (FOG) index.13 Scores were based on Chicago, Ill) and for all tests PϽ.05 was cantly lower in the throwaway journal sentence and word length. The Flesch considered significant. articles compared with the peer-

Downloaded From: https://jamanetwork.com/ on 09/30/2021 REVIEW ARTICLES IN PEER-REVIEWED AND THROWAWAY JOURNALS

reviewed journal articles (17.2 [2.9] vs Table 1. Presentation and Readability of Review Articles (N = 394) Published in 19.2 [3.3]) indicating that throwaway Peer-Reviewed and Throwaway Journals in 1998* journal articles were easier to read Peer-Reviewed Articles (PϽ.001). Peer-reviewed journal ar- Systematic Nonsystematic Throwaway ticles were significantly more likely than Reviews Reviews Journal Articles P throwaway journal articles to score in (n = 16) (n = 135) (n = 243) F2,391 Value the range judged too difficult even for Presentation medical writing (67.4% vs 53.5%; No. of tables 2.4 (1.7) 2.1 (1.6) 2.6 (1.6) 3.8 .02 P=.009). No. of figures 1.4 (1.5) 2.5 (2.1) 3.2 (3.2) 4.6 .01 Peer-reviewed journal article titles No. of photographs 0 0.98 (1.5) 1.9 (3.1) 8.6 Ͻ.001 were judged to be significantly less rel- Color used, No. (%) 0 77.0 (57.0) 228 (93.8) 124.33† Ͻ.001 Readability evant to clinical practice than throw- Flesch index score12 11.7 (19.0) 15.8 (17.7) 23.7 (15.4) 12.5 Ͻ.001 away journal article titles. When the Gunning FOG index score13 19.1 (2.4) 19.2 (3.3) 17.2 (2.9) 19.0 Ͻ.001 physicians reviewed the article titles and *Data are presented as mean (SD) unless otherwise indicated. For the Flesch index, a lower score indicates more were asked whether the article pro- difficult reading. For the Gunning Frequency of Gobbledygook (FOG) index, a higher score indicates more difficult reading. 2 vided useful information for their clini- †The ␹ 2 test was conducted. cal practice, throwaway journal articles were rated more relevant (mean Table 2. Clinical Relevance of Review Articles on Heart Disease (N = 30) Published in [SD], 3.89 [0.55]) compared with peer- Peer-Reviewed and Throwaway Journals in 1998* reviewed nonsystematic (3.50 [0.67]) Peer-Reviewed Articles or systematic (3.41 [0.73]) review articles (F =20.7, PϽ.001). Simi- Systematic Nonsystematic Throwaway 2,391 Reviews Reviews Journal Articles P larly, when the physicians reviewed ar- General Content Question (n=4) (n=9) (n = 17) F2,27 Value ticle titles and were asked whether they The article addresses an 3.85 (1.01) 4.40 (0.33) 4.66 (0.31) 5.4 .01 would consider reading the article, important issue throwaway journal articles were rated The topic is of interest to me 3.63 (0.81) 3.83 (0.71) 4.43 (0.39) 5.3 .01 as an article they were more likely to The topic is relevant to my practice 3.21 (1.29) 3.46 (1.04) 4.21 (0.59) 3.6 .04 read (3.74 [0.59]) compared with peer- The article provides practical 3.04 (1.19) 3.30 (0.98) 4.00 (0.56) 4.0 .03 strategies for physicians reviewed nonsystematic (3.34 [0.69]) such as myself or systematic (3.17 [0.84]) review ar- I will use the information 3.04 (1.19) 3.32 (0.90) 3.88 (0.55) 2.9 .07 Ͻ ticles (F2,391=20.2, P .001). to help care for patients TABLE 2 outlines the reviewers’ as- *Data are presented as mean (SD). sessment of the clinical relevance of the subset of 30 heart disease manage- lished in peer-reviewed journals on of the scientific material they publish ment articles. Compared with the peer- virtually all measures of presentation, to their readership. Despite these reviewed journal articles, throwaway readability, and the clinical relevance efforts, our findings suggest that peer- journal articles were judged more likely of the message. As expected, peer- reviewed journal articles lag behind to address important issues and be a reviewed journal articles were of the throwaway journal articles in these topic of interest to the physicians. Fur- superior methodologic and reporting communication techniques. thermore, throwaway journal articles quality relative to articles published in Our study has several limitations. provided tables that were significantly throwaway journals. These findings First, review article quality scoring easier to understand (F2,27=5.5, P=.01), are consistent with the large body of instruments reward articles that are helped to clarify the text (F2,27=9.5, evidence showing that peer-reviewed systematic reviews. Many articles in P=.001), and provided information rel- medical journals produce articles of our sample were not intended to be evant to clinical practice (F2,27=13.5, superior quality compared with those systematic reviews. Nonsystematic PϽ.001). published in non–peer-reviewed reviews can provide valuable informa- journals.14-16 tion. However, systematic review COMMENT The simplest way of writing is not articles are the only type of review We found that review articles pub- always the best.17 Complex messages that has been shown to minimize bias. lished in throwaway journals were may require complex writing to con- Second, our physician reviewers may easier to read than review articles vey accurate information. Through the not be representative of all physicians; published in peer-reviewed medical use of color,18,19 larger font size,18-20 all had a clinical focus and were journals. Review articles published and the incorporation of more graph- recent graduates. Third, titles may not in throwaway journals were rated ics,19 many peer-reviewed journals be the best way to judge clinical rel- consistently better than articles pub- have attempted to improve the appeal evance but play an important role in

Downloaded From: https://jamanetwork.com/ on 09/30/2021 TRANSLATING RESEARCH INTO NEWS

attracting readers’ attention and influ- lar topics. Although lower in methodo- Administrative, technical, or material support: Gold, Dergal. ence the decision of whether to read logic and reporting quality, review ar- Study supervision: Rochon, Gurwitz. an article. ticles published in throwaway journals Review Article Study Group Members: Nancy Byles, MD, Geoffrey Litner, MD, Ranjee Singh, MD, A balance needs to be achieved be- possess characteristics that are appeal- University of Toronto, Toronto, Ontario; Monidipa tween presenting high-quality informa- ing to physician readers. Dasgupta, MD, Sudeep Gill, MD, University of Western Ontario, London, Ontario; Linda tion and communicating the message. Author Contributions: Study concept and design: Devore, MS, MLS, Amy Freedman, MD, Yael Throwaway journals do not serve the Rochon, Bero, Bay, Binns, Streiner, Gurwitz. Karoly, Baycrest Centre for Geriatric Care, Toronto, same markets as peer-reviewed jour- Acquisition of data: Rochon, Bay, Gold. Ontario. Analysis and interpretation of data: Rochon, Bero, Der- Funding/Support: Dr Rochon was supported by an nals and are largely supported by adver- gal, Binns, Streiner, Gurwitz. Investigator Award from the Canadian Institutes of tising; therefore, their editors may choose Drafting of the manuscript: Rochon, Bay, Gold, Streiner, Health Research, Ottawa, Ontario. Mr Bay and Ms Gurwitz. Gold were supported by a Summer Research Student- to publish articles for which there are en- Critical revision of the manuscript for important in- ship from the Baycrest Centre for Geriatric Care, thusiastic sponsors. In contrast, peer- tellectual content: Rochon, Bero, Dergal, Binns, Toronto, Ontario. Streiner, Gurwitz. Acknowledgment: We thank Penelope de Nobrega, reviewed journals may be more likely to Statistical expertise: Binns, Streiner. RN, and Michelle Fischbach, MD, for their assistance tackle difficult and sometimes less popu- Obtained funding: Rochon. with the project.

REFERENCES 1. Rennie D, Bero LA. Throw it away, Sam: the con- health effects of passive smoking reach different con- 14. Rochon PA, Gurwitz JH, Cheung CM, et al. Evalu- trolled circulation journals. CBE Views. 1990;13:31- clusions. JAMA. 1998;279:1566-1570. ating the quality of articles published in journal supple- 35. 8. Oxman A, Guyatt GH. Validation of an index of ments compared with the quality of those published 2. Finklestein D. Oh, the times! tabloids and other non– the quality of review articles. J Clin Epidemiol. 1991; in journal supplements. JAMA. 1994;272:108-113. peer-reviewed publications. Arch Ophthalmol. 1985; 44:1271-1278. 15. Barnes DE, Bero LA. Scientific quality of original 103:1641-1642. 9. Oxman AD, Guyatt GH, Singer J, et al. Agree- research articles on environmental tobacco smoke. Tob 3. Soffer A. What is a practical clinical journal? Arch ment among reviewers of review articles. J Clin Epi- Control. 1997;6:19-26. Intern Med. 1980;140:1419. demiol. 1991;44:91-98. 16. Cho MK, Bero LA. The quality of drug studies pub- 4. Cook D, Meade MO, Fink MP. How to keep up 10. Mulrow CD. The medical review article: state of lished in symposium proceedings. Ann Intern Med. with the critical care literature and avoid being bur- the science. Ann Intern Med. 1987;106:485-488. 1996;124:485-489. ied alive. Crit Care Med. 1996;24:1757-1768. 11. Roberts JC, Fletcher RH, Fletcher SW. Effects of 17. Goodman NW. Too many words? Mozart 1, Em- 5. The Bowker International Serials Database: Urlich’s peer review and editing on the readability of articles peror 0. JAMA. 1995;273:1087-1088. International Periodical Directory [database online]. published in Annals of Internal Medicine. JAMA. 1994; 18. Delamothe T, Smith R. Redesigning the journal: 36th ed. New Providence, RI: RR Bowker; 1998. 272:119-121. having your say. BMJ. 1996;312:232-234. 6. Hunt DL, McKibbon KA. Locating and appraising 12. Flesch RF. A new readability yardstick. J Appl Psy- 19. Flanagin A, Murphy PJ, Lundberg GD. JAMA’s new systematic reviews. Ann Intern Med. 1997;126:532- chol. 1948;32:221-223. look: a New Year’s gift to readers. JAMA. 1999;281:85. 538. 13. Gunning R. The Technique of Clear Writing. New 20. Davidoff F. Annals now and then. Ann Intern Med. 7. Barnes DE, Bero LA. Why review articles on the York, NY: McGraw-Hill International Book Co; 1952. 1996;124:67-68.

Press Releases Translating Research Into News

Steven Woloshin, MD, MS Context While medical journals strive to ensure accuracy and the acknowledgment Lisa M. Schwartz, MD, MS of limitations in articles, press releases may not reflect these efforts.

EDICAL JOURNALS WORK Methods Telephone interviews conducted in January 2001 with press officers at 9 hard to ensure that prominent medical journals and analysis of press releases (n=127) about research articles for the 6 issues of each journal preceding the interviews. articles fairly represent study findings and to Results Seven of the 9 journals routinely issue releases; in each case, the editor acknowledgeM important limitations, with the press office selects articles based on perceived newsworthiness and work that may be undone by the time releases are written by press officers trained in communications. Journals have general guidelines (eg, length) but no standards for acknowledging limitations or for research findings reach the news data presentation. Editorial input varies from none to intense. Of the 127 releases media. Medical journal press releases analyzed, 29 (23%) noted study limitations and 83 (65%) reported main effects are perhaps the most direct way that using numbers; 58 reported differences between study groups and of these, 26 (55%) provided the corresponding base rate, the format least prone to exag- Author Affiliations: VA Outcomes Group, Depart- geration. Industry funding was noted in only 22% of 23 studies receiving such ment of Veterans Affairs Medical Center, White River Junction, Vt; Center for the Evaluative Clinical Sci- funding. ences, Dartmouth Medical School, Hanover, NH; and Conclusions Press releases do not routinely highlight study limitations or the role of Norris Cotton Cancer Center, Lebanon, NH. Corresponding Author and Reprints: Lisa M. Schwartz, industry funding. Data are often presented using formats that may exaggerate the MD, MS, VA Outcomes Group (111B), Department perceived importance of findings. of Veterans Affairs Medical Center, White River Junc- JAMA. 2002;287:2856-2858 www.jama.com tion, VT 05009.

Downloaded From: https://jamanetwork.com/ on 09/30/2021