Does the Medical Literature Contain the Evidence to Answer the Questions of Primary Care Physicians? Preliminary Findings of a Study Paul Gorman, MD, FACP Assistant Director of Medical Education, Providence Medical Center, Portland, Oregon, and Assistant Professor of and Informatics, Biomedical Information Communication Center, Oregon Health Sciences University, Portland, Oregon

ABSTRACT questions are never answered. Extrapolating from these data, as many as 60 questions about patient care Objective: To assess the quality ofinformation arise during one week in a typical prnmary care retrieved in MEDLINE searches performed to answer practice, and some 40 of these questions are never clinical questions ofprimary care physicians. answered. Seting: Ambulatory rural and nonrural practces. To answer those questions that they do pursue, Participants: Active, non-academic primary care physicians make minimal use of the medical literature physicians; professional medical librarians, a [3,4,] relying most heavily on colleagues and printed general internist. sources [5] (Figure 1.) But textbooks available in the Design: Descriptive study in 3 stages: 1) Office office are often outdated [1] and due to delays in Interviews, to record questions that arise in primary publishing, even recent editions of textbooks may not care practice; 2) Online Searches, to locate contain up-to-date information. Furthermore, a recent information that might answer these questions; and 3) study confirmed the shortcomings of this dependence Critical Appraisal to determine the quality ofthe on textbooks and expert colleagues, finding that information contained in the articles retrieved. recommendations in these traditional sources often Main Results: Of 74 articlesjudged by clinicians to lag significantly behind evidence in published be relevant to their questions, 2 were criticat reviews, randomized trials [6.] 14 reported results ofrandomized controlled trials, Much evidence suggests that these information and 4 reported results ofcohort studies. The seeking patterns have a negative impact on patient remainder contained weaker evidence: I reported care. Stross & Harlan found that eighteen months results ofa case-control study, 6 were longitudinal after publication of an authoritative report showing case series, 13 were cross-sectional descriptions of the benefit of photocoagulation for diabetic , 12 were cross-sectional descriptions of retinopathy, two thirds of primary care physicians diagnostic testing, and 22 were review articles which were not aware of and were not referring their did not meet criteriafor a critical review.. patients for this treatment [7.] A national survey of Conclusion: Searching the medical literature to primary care physicians showed that one third to one answer the questions ofprimary care physicians was half of respondents were not aware of or were not successful in retrieving recent, relevant information, and often provided "a clear answer" to the question, Figure 1 but the cost ofusing bibliographic retrieval was high, Sources Used by Primary Care Physicians andfewer than a third ofthe articles retrieved to Answer Their Clinical Questions contained high quality evidence. * unanswered: 216 * textbooks: 22 INTRODUCTION M physician: 25 Questions about optimal patient care arise E drug textbooks: 14 frequently in the course of office practice, yet most of * non MD: 9 these questions are never answered. Interviews with *8 other: 4 internists in urban subspecialty practices [1] and with * reprint file: 4 rural and nonrural primary care physicians [2] reveal 0 computer: 1 that about two questions arise for every three patients I...... I seen in office practice, but that 70% of these from reference [3]

0195-4210/92V5.00 1994 AMIA, Inc. 571 using each of six recent medical advances, such as information that is useful to practitioners [21.] glycohemoglobin measurement to monitor glycemic Similar questions about the usefulness in clinical control in diabetes [4.] More indirect evidence comes practice of much published research have arisen from from the CAST study of antiarrhythmic in within the research community itself [22.] post-MI patients. Although randomized trials and Fourth, there often is no consensus in the meta-analyses of randomized trials have shown since literature about the correct approach to a clinical the early- to mid-1980's that beta-blocking agents problem. The controversy surrounding the issue of reduce mortality in post-MI patients [8,9,] 70% of screening for and treating hypercholesterolemia [23] post-MI patients referred for entry into the CAST is but one example. Practitioners may be reluctant to study were not receiving beta-bloclers at the time of look for answers to their questions when they expect enrollment [10.] to find contradictory recommendations. To remedy this problem, it has been suggested These factors call into question the assumption that physicians in training [11] and those in practice that practitioners who increase their use of the [12,13] should increase their reliance on the medical medical literature will find relevant, valid, and literature. Efforts to increase use of the literature clinically useful answers to their questions about have focused mainly on removing barriers to access, patient care. We designed a study to examine the including the time, cost, training, and difficulty of question, "If barriers to access are removed, can use. Improvements in software, such as Grateful answers to the questions of primary care physicians Med, offer greater ease-of-use and may reduce online be found using the medical literature?" Specifically: time and cost. Training in MEDLINE searching 1) Can relevant answers be found? 2) Can clinically offers experience with the system and can improve useful answers be found? 3) Can evidence-based user performance [14.] Reductions in the cost of answers be found? 4) What would it cost to do this? searching, such as the recent MEDLINE price In the first part of the study, reported elsewhere [24] reduction, remove financial barriers, which have been medical librarians searched the literature for answers shown to impact use of the system [15.] to questions collected from primary care physicians. These efforts to improve access to the literature In the judgment of the clinicians themselves, relevant rest on the assumption that once barriers are removed, information was found for 58% of the questions, and clinicians will be able to find relevant, valid, and "'a clear answer" was found for 46% (Table 1.) The clinically useful information to answer their questions cost of obtaining this information was high, however. about patient care. However, there are reasons to On average, experienced medical librarians spent 43 question this assumption. minutes finding and selecting 2 to 3 articles totaling First, in view of the overwhelming quantity of about 20 pages, at a cost of approximately $21 to $34 biomedical literature published annually and the per question, including the librarian's time and limitations of bibliographic retrieval in even expert charges for copying and forwarding articles to the hands [14] it may be difficult to assure clinicians that clinicians. they will be able to find the answers to their clinical If primary care physicians are to use the medical question, even if answers exists. literature to answer their clinical questions, they must Second, numerous reviews of published research not only search for and retrieve relevant information, have found problems with the methodology [16-19.] but also critically appraise the information they obtain As an example, a comprehensive MEDLINE search to determine whether it is based on evidence that is for papers on diagnosis of iron-deficiency anemia found 1,179 potentially relevant citations, of which Table 1 only 55 were judged adequate for inclusion in their Clinicians' Evaluation of Information Retrieved to review. Of 5 review papers retrieved, none met Answer Their Clinical Questions (n = 48) criteria for a scientific overview [20.] Thus, it may be difficult for clinicians to assume that what they do 11 % find in the literature is based on sound evidence. Third, the usefulness of published research to relevant to the question 28 58 clinical practice has been questioned as well. Based contains "a clear answer" to question 22 46 on interviews with hundreds of nonacademic primary care physicians, Greer has reported the commonly expected impact on the patient 19 40 held view that the literature is primarily a means of expected impact on the physician 24 50 communication among researchers, not a source of data from reference [24]

572 sufficiently sound to be applied in practice. In the Figure 2 second part of this study, preliminary results of which Examples ofPrimary Care Questions are reported here, we examine the quality of the * "At what age is a screening prostate specific information retrieved. antigen test indicated in a low risk patient?" * "What is the exact increase in risk of thrombotic events due to oral contraceptives in a woman METHODS with a family history of myocardial infarction (her grandmother at age 49) and of deep vein Objectives and Design thrombosis?" The objective of the study was to assess the * "Are non acetylated salicylates really safer, and quality of the information retrieved in MEDLINE how much safer, in patients with NSAID- searches performed to answer clinical questions of induced stomach intolerance who benefit from primary care physicians. The study consisted of 1) the anti-inflammatory effect?" * "For diagnosis of deep vein thrombosis, how Office Interviews, to record clinical questions of good is ultrasound: does it obviate the need for primary care physicians; 2) Online Searches to locate venogram?" information that might answer these questions; and 3) * "Is amoxicillin safe for use in a lactating Critical Appraisal of the articles retrieved, to woman?" determine the quality of the information therein. * "What is the sensitivity and specificity of arterial These stages are described in detail below. ultrasound exam of the lower extremities?"

Offi'ce Interviews physician at a remote location looking for an Office Interviews (part of an earlier study [2]) immediate, practical, up-to-date answer to a question took place in ambulatory, primary care practices in about patient care." They searched bibliographic Oregon. Participating physicians were active, non- indices accessible by remote electronic searching but academic primary care physicians, 25 each from rural not texts which, though available in electronic form, and nonrural areas of Oregon. Using a method are primarily used in their printed form and could be modified from Covell et al. [1,] physicians were used in a physician's office without the hardware, interviewed during one half-day of typical office software, and expertise needed to do online searches. practice. After each patient was seen, the physician Most searches were performed on MEDLINE. When was asked, "Do you have any questions about the each search was complete, librarians were asked to diagnosis or management of this patient's problem?" select a small number of articles to copy and send to and any questions were recorded. Physicians were the originating primary care physician, giving asked to state "all the questions which occur to you preference to "high quality evidence" such as meta- during patient care, no matter whether you would analyses and randomized trials, when available. pursue them or not, nor what source you might consult for an answer." These were not questions Critical Appraisal intended to be submitted for MEDLINE searches, nor The goal of the appraisal was not to directed to any other potential source of an answer, comprehensively scrutinize each article for every but rather were "native" clinical questions, as they subtle methodological flaw, but rather to screen for occurred to practicing physicians in the context of a major problems and determine whether the quality of specific patient problem. A sample of these questions evidence was sufficient for application in clinical is shown for illustration in Figure 2. Of 295 questions practice. Appraisal was limited to those articles recorded during 49 half-day interviews, 60 were judged by the clinicians themselves to be relevant to randomly selected and submitted for searches. their question. For each article, the appraiser recorded the year of publication and the type of article, Online Searches according to a classification modified from Fletcher Searches were performed by medical librarians and Fletcher [17] and Bailar et al. [25] and used by familiar with bibliographic retrieval using other groups as the basis for determining the level and MEDLARS and other systems, with an average of quality of evidence for published guidelines [27]. 13.6 (range 5 to 20) years' experience. Librarians The classification scheme is listed in Table 3. were asked to serve, in effect, as the'"gold standard" for what would exist if barriers to access were RESULTS removed. They were instructed to "treat each question as if it had been submitted by a primary care In the Online Search and selection process,

573 Table 2 and thus, according to suggested criteria [26], may Year of Publication of 74 Relevant Articles not merit further examination. Of the remaining articles, only 16, 14 randomized trials and 2 critical year n % reviews, contained high quality evidence upon which to base clinical decisions. 1992 19 26% 1991 14 19% DISCUSSION 1990 15 20% The purpose of this study was to determine 1989 2 3% whether the medical literature can be expected to provide relevant, clinically useful, evidence-based 1988 7 9% answers to the clinical questions of primary care 1987 6 8% physicians, and to measure the cost in time and expense of finding these answers. In the first part of 1986 6 8% the study, we found that, if barriers to access were 1985 1 1% removed, relevant information could be found for 58% of the questions, and "a clear answer" could be pre- 1985 4 5% found for 46%. However the time and effort needed to do this were substantial. In this second part of the reported elsewhere [24] and summarized in Table 1, study the quality of the articles retrieved was 74 articles retrieved to answer 28 questions were examined. The information retrieved was more judged relevant by the clinician who had asked the current than that in most textbooks, with two thirds of question. Appraisal was limited to these 74 articles. the articles having been published within 3 years. As shown in Table 2,45% of the articles retrieved However, nearly one third of the relevant articles had been published in the preceding 2 years; two retrieved were neither critical reviews nor original thirds of the articles were less than 3 years old. research, while only 16 of 74 articles were found to Classification of the 74 relevant articles is shown contain the high level evidence that ideally should in Table 3. As indicated, two thirds of the articles form the basis for clinical recommendations. were reports of original research, while one third were It is important to note that high quality evidence review articles. Of the 24 review articles, only 2 met such as randomized control trials or critical reviews criteria for a critical review. Of 50 reports of original are not always necessary to solve a specific patient research, 14 reported on randomized control trials, 5 management problem. For example, a or reported cohort or case control studies, and the small series may be sufficient to determine whether a remaining 31 reported case series or cross-sectional condition has ever been reported. Similarly, the fact data. Based on these preliminary findings, 30% of the that no information can be found by literature search articles that had been judged relevant by the clinicians may itself be useful information, as in the case of the were neither critical reviews nor original research, existence of a drug interaction. In these and other Table 3. may find the answer to the Table 3. instances, physicians of Relevant Articles question without locating a randomized trial. Classirication A number of limitations and constraints of this Reviews study must be noted; First, this study was not a comprehensive evaluation of the quality of the critical review 2 3% medical literature as a whole. Many previous studies noncritical review 22 30% have identified various methodological shortcomings Reports of Original Research in the medical literature [16] and this study was not intended to duplicate them. Second, this study was randomized controlled trial 14 19% not an attempt to definitively determine whether cohort or trohoc study 4 5% answers to clinical questions could be found through case-control study 1 1% exhaustive literature search, careful critical appraisal, and formal data synthesis, but rather to examine the case series (longitudinal) 6 8% results that might be obtained by practicing cross-sectional, disease description 13 18% physicians looking for immediate, practical answers to about care. searchers were cross-sectional,diagnostic test 12 16% questions patient Third,

574 limited to online bibliographic retrieval only, even literature in search of answers their clinical where another source of information might have been questions,they must have some assurance that they judged more appropriate; and they had no opportunity and their patients are likely to benefit from doing so. to interview the clinicians to refine their search or The results of this study suggest that although selection process. relevant and apparently useful information may often If practitioners are to be encouraged to invest in be found, the time and effort required are substantial, the hardware, software, online costs, and and the information available in the literature may training,necessary for bibliographic retrieval, and frequently not be of adequate quality for application more importantly, if they are to be encouraged to in practice. invest their valuable time searchine the medical References 1. Covell DG, Uman GC, Manning PR. Information Biomed Res. 1990;23:583-593. needs in office practice: Are they being met? Ann Intern 15. Haynes RB, Ramsden MF, McKibbon KA, et al.. Med. 1985;103:596-599. Online access to MEDLINE in clinical settings. Impact of 2. Gorman PN, Ash J, Helfand M, Beck JR. "Assessment user fees. Bull MedLibr Assoc. 1991;79:377-381. of information needs of primary care physicians." in Presentation at the American Medical Informatics 16. Anderson B. Methodological Errors Medical Association Spring Congress; 1992 May 7; Portland (OR). Research. Oxford: Blackwell; 1990. 17. Fletcher Fletcher SW. Clinical research in 3. Gorman PN, Ash J, Helfand M, Beck JR. "Information RH, general needs and information seeking of rural and nonrural medical journals: A 30-year perspective. NEJM. primary care physicians." Poster presentation at Third 1979;301:180-183. Primary Care Research Conference, Agency for Health 18. Freiman JA, Chalmers TC, Smith H, Kuebler RR. The Care Policy and Research; 1993 January 10-12; Atlanta importance of beta, the Type II error, and sample size in the (GA). design and interpretation of the randomized controlled trial. 4. Williamson JW, German PS, Weiss R, Skinner EA, In: Bailar JC, Mosteller F. Medical Uses ofStatistics. 2nd Bowes F. Health science information management and ed. Boston: NEJM Books; 1992:357-373. continuing education of physicians: A survey of US primary 19. Divine GW, Brown JT, Frazier LM. The unit of care practitioners and their opinion leaders. Ann Intern analysis error in studies about physicians' patient care Med. 1989;110:151-160. behavior. J Gen Intern Med 1992;7:623-629. 5. Connelly DP, Rich EC, Curley SP, Kelly JT. 20. Guyatt GH, Oxman AD, Ali M, Willan A, Mcllroy W, Knowledge resource preferences of family physicians. J Patterson C. Laboratory diagnosis of iron-deficiency Fani Pract. 1990;30:353-359. anemia: An overview. J Gen Intern Med 1992;7:145-152. 6. Antman EM, Lau J, Kupelnick B, et al.. A comparison 21 Greer AL. The state of the art versus the state of the of results of meta-analyses of randomized control trials and science: The diffusion of new medical technologies into recommendations of experts. JAMA. 1992;268:240-248. practice. Int J Tech Assess in . 1988;4:5-26. 7. Stross JK, Harlan WR. The dissemination of new 22. Rabenek L, Viscoli CM, Horwitz RI. Problems in the medical information. JAMA. 1979 241:2622-2624. conduct and analysis of randomized clinical trials: Are we 8. Norwegian Multicenter Study Group. Timolol-induced getting the right answers to the wrong questions? Arch Int reduction in mortality and reinfarction in patients surviving Med. 1992;152:507-512. acute myocardial infarction, I: mortality results. N Engl J 23. Muldoon MF. Commentary in ACP Journal Club. Med. 1981;304:801-807. Annals Int Med. 1993; 1 18(Suppl 1):21-23. 9. Yusuf S, Peto R, Lewis J, et al.. Beta blockade during 24. Gorman PN, Wykoff L, Ash J. "Can primary care and after myocardial infarction: an overview of the physicians' questions be answered by using the medical randomized trials. Prog Cardiovas Dis. 1985;27:335-371. literature?" Presentation at Medical Library Association 10. Echt DS, Liebson PR, Mitchell B, Peters RW, et al.. Annual Meeting; 1993 May 15; Chicago, IL. Mortality and morbidity in patients receiving encainide and 25. Bailar JC, Louis TA, Lavori PW, Polansky M. A flecainide, or placebo: The cardiac arrhythmia suppression classification for biomedical research reports. In: Bailar JC, trial. N Engl J Med. 1991;324:781-788. Mosteller F. Medical Uses ofStatistics. 2nd ed. Boston: 11. Kassirer J. Learning Medicine: Too many books, too NEJM Books; 1992:141-156. few journals. N Engl J Med. 1992;326:1427-1428. 26. Sackett DL, Haynes RB, Guyatt GH, Tugwell P. 12. Huth EJ. The underused medical literature. Annals Int Clinical Epidemiology: A Basic Sciencefor Clinical Med. 19849;110:99-100. Medicine. 2nd ed. Boston: Little, Brown; 1991. 27. U.S. Preventive Services Task Force. Guide to 13. Evidence-Based Medicine Working Group. Evidence- Clinical Preventive Services: An Assessment ofthe based medicine: A new approach to teaching the practice of Effectiveness of169 Interventions. Report ofthe U.S. medicine. JAMA. 1992;268:2420-2425. Preventive Services Task Force. Baltimore: Williams & 14. McKibbon KA, Haynes RB, Walker-Dilks CJ, et al.. Wilkins; 1989. How good are clinical MEDLINE searches? A comparative study of clinical end-user and librarian searches. Comp

575