6 6Journal of and Community Health 1994;48:6-15 The lecture J Epidemiol Community Health: first published as 10.1136/jech.48.1.6 on 1 February 1994. Downloaded from

The best and the enemy of the good: randomised controlled trials, uncertainty, and assessing the role of patient choice in medical decision making

Klim McPherson

Abstract clear today as it did then. Nowadays, both the This lecture aimed to create a bridge to problem and solution are very familiar to us span the conceptual and ideological gap all. The problem was that much of the health between randomised controlled trials care provided was unevaluated, and therefore and systematic observational compari- possibly of no benefit, and the solution - sons and to reduce unwanted and unpro- randomised controlled trials - were the only ductive polarisation. The argument, secure way of knowing the truth of the matter. simply put, is that since randomisation Cochrane's essential idea was that the reli- alone eliminates the selection effect of able assessment of effectiveness was the only therapeutic decision making, anything key to scientific health care, but such methods short of randomisation to attribute cause as he advocated had been disparaged by the to consequent outcome is a waste of time. medical establishment - paradoxically because If observational comparison does have they seemed to be less scientific than the more any significant part in evaluating medi- traditional basic methods of scientific evalu- cal outcomes, there is a grave danger of ation. A problem of methodological imperia- "the best", to paraphrase Voltaire, lism which is with us still, but in different becoming "the enemy of the good". The guise? first section aims to emphasise the ad- He argued forcibly that only when the effec- vantages of randomised controlled trials. tiveness of a treatment or preventive measure Then the nature of an essential precondi- had been established and quantified, could the tion - medical uncertainty - is discussed issues of efficiency and equity be discussed

in terms of its extent and effect. Next, the intelligently. But in the absence of reliable http://jech.bmj.com/ role of patient choice in medical decision information about effectiveness, the pursuit of making is considered, both when out- optimal health policies was bound to be per- comes can safely be attributed to treat- manently elusive. ment choice and when they cannot. There Since then Cochrane's arguments have been may be many important situations in largely and impressively accepted by health which choice itself affects outcome and professionals.3 It is probable that without them this could mean that random compari- we would not have all the welcome recent sons give biased estimates of true thera- developments designed specifically to provide on October 3, 2021 by guest. Protected copyright. peutic effects. In the penultimate section, a more effective health care system. I am the implications of this possibility both referring particularly to the NHS Research for randomised controlled trials and for and Development initiative and the systematic outcome research is pursued and lastly concentration on issues of effectiveness and there are some simple recommendations efficiency in the direction of research and for reliable outcome research. health policy. This is why there is a Cochrane lecture at (JT Epidemiol Community Health 1994;48:6-15) the Annual Scientific Meeting of the UK Department of Public Society of Social Medicine and I am honoured Health and Policy, to be giving the lecture this year. I cannot do School of Hygiene and Tropical Archie Cochrane's great contribution to our justice to my predecessors, who have contrib- Medicine, Keppel discipline in 1972 was to describe a most uted significantly and importantly to the work Street, London important problem in health care and to posit a spearheaded by Archie Cochrane, but I hope WC1E 7HT on K McPherson satisfactory solution. In his biography,' he says to make some methodological points the of his work on Effectiveness and Efficiency2: question of evaluation of health care, many of Correspondence to: I want to Professor Klim McPherson. "I decided to concentrate on one simple idea which I have made before. Largely, - the value of the randomised controlled trials try to divert any polarisation of methodologies Presented at the Scientific Meeting of the UK Society in improving the NHS - and to keep the book which may, I believe, lead to an unnecessary for Social Medcicine, of medical uncertainties. The Cambridge, UK in short and simple." prolongation September 1993. He did so in a manner which rings as loud and issues are too important in my view. The best and the enemy of the good 7

(1) The value of randomised controlled provide hard unbiased evidence about the rela- J Epidemiol Community Health: first published as 10.1136/jech.48.1.6 on 1 February 1994. Downloaded from trials tive effectiveness of common, important, and Since the publication of Effectiveness and Ef- expensive treatments. It is absolutely no part ficiency2 there have been several attempts to of my argument to disparage randomised com- minimise or falsify the importance and rele- parisons at all - least of all because they are too vance of Cochrane's message. Each succeeds expensive or complicated. The end justifies the only in accentuating its importance. It is im- means most of the time. To accept these kind possible nowadays to deny the requirements of arguments is really to suggest that some for serious research on outcomes, but the gap alternative is generally better and that some- between true knowledge of outcomes and the how it does not matter, because it is inevitable, need to know and to understand, and hence to that most treatments will not be evaluated by meet these requirements, remains large. randomised comparisons. What was most important in 1972, and still Feinstein and others (Colin Dollery, for is for us now, is how to bridge that gap. Last example, in his Rock Carling lecture7) assert year Iain Chalmers gave a masterful demon- that most treatments will be assessed by a stration of the practical extrapolation from mixture of insight, theory, acumen, and un- Cochrane himself to the Cochrane Centre, controlled observation. Of course large effects clearly a central part of the "outcomes move- can be reliably detected like that, but from now ment". This evolved from the National Peri- on I will be discussing uncertainties over small natal Epidemiology Unit and took on the work but important effect differences in common of Iain, Richard Peto, Rory Collins, and others medical practices which are poorly understood in summing the results of randomised trials of and are simply not amenable to evaluation in treatment. There is no question in my mind this way. that these efforts in understanding the effec- There is a more extreme current idea - tiveness of treatments, and hence helping which is just plain wrong - and this is that the people choose between options, are of enor- use of databases which record everything mous value. For me, the most exciting has about patients can easily, as a consequence of been the work on breast cancer treatment.4 recording everything, be used to adjust for all The increasing use ofrandomised comparisons confounders and then compare treatments as if in judging effectiveness is wholly appropriate there had been no selection.8 This has to be and must go much further. dismissed as ridiculous. Unknown and unsus- pected confounding is mostly important and always elusive. IMPEDIMENTS The great idea of Fisher and Bradford Hill, I want to leave aside, for the moment, the so ably developed by many, including Archie extent of the importance of uncertainty in Cochrane, is one of the few panaceas of our health care decision making, and examine time. In the context of medical interventions, briefly part of the existing resistance to the use the essence of the argument has been repea- of randomised controlled trials. Cochrane dis- tedly put by David Byer.' He says that "Epi- cussed impediments to the wider acceptance of demiology is an essential discipline . . . for these in practice. There are three basic reasons assessing the importance of exposures ... there

for not carrying out randomised controlled is a disassociation between the reason for expo- http://jech.bmj.com/ trials where uncertainty exists: sure and the outcome. On the other hand, in (1) Ethical objection (?) medicine, the doctor chooses the therapy pre- (2) Lack of objective outcome measures cisely to affect the outcome and for no other (3) The resistance of those "threatened" by reason". Archie's view was most succinctly randomised controlled trials. put, and poignant nonetheless: "Observational evidence is clearly better than opinion but it is thoroughly unsatisfactory". THE ARGUMENTS on October 3, 2021 by guest. Protected copyright. We must be clear about the basic arguments, because in advocating greater, more system- UNCERTAINTY, SUPPLIER-INDUCED DEMAND, atic, and rigorous use of observational AND CONSUMER PREFERENCES methods5 in judging efficacy (as I shall be Let me restate the basic premise in case subse- doing later) it is very easy to associate oneself quent remarks should be misunderstood. Ran- with a camp one does not wish to be associated domised comparisons provide the most rigor- with. There have been many exponents of the ous assessment of the therapeutic effect of view, encapsulated by Feinstein,6 that "Cause- particular interventions. The problems they effect can be evaluated observationally" is are set to help solve are, however, formidable, scientific heresy only if two cogent scientific for decision making in medicine is rarely quite realities are ignored - the first, that some as simple as choosing the best treatment for things are not amenable to experimental inter- each objective diagnosis, even when it is vention, and the second, that randomised con- known. In my opinion the dominant issues in trolled trials are too expensive, too difficult, or health care now consist of, firstly, understand- too controversial. Whether or not such notions ing the extent of true uncertainty and, derive from feeling threatened is of secondary secondly, relating this to the quality and costs importance only, but they are common. of care. Then, since medical care is simultan- Of course trials are expensive and difficult, eously and consequently emerging from an era but from a cost effective point of view they can of paternalism and medical domination, all be much more readily justified because they decision making now has explicitly and 8 McPherson

Table I Basic criteria for assessment priorities in but the implications of these other considera- J Epidemiol Community Health: first published as 10.1136/jech.48.1.6 on 1 February 1994. Downloaded from outcome research tions must at least be properly assessed. Provision should be common Provision should be relatively expensive There should be obvious (explicit) uncertainty There should be real treatment choices (2) The extent of true uncertainty in Assessment should have the potential for influencing choices medicine I would like to propose something I have often suggested before which will complement the increasingly to take account of two impc)rtant existing method of setting research priorities - concepts. These are the role of supplier- this is the routine monitoring of practice varia- induced demand and, secondly, how to accom- tion and examination of plausible causes. modate consumer preferences into decitsions. These things were less important in (ioch- rane's day, and it seems to me they bring WlLll..; f4% Proposal 1: Routine monitoring of practice them new emphases and responsibilities, both variation and examination of plausible causes in the method and the mechanism of ujnder- In the determination of health care outcome standing treatment outcomes. Of coursee, the research priorities, several criteria are scientific/biological methods of medicin(e and obviously foremost. These are set out in table its evaluation will always remain domiinant, 1. This is clearly an essential list but of course tends to emphasise the glamorous, the inter- d esting, the pet obsessions, and some general .\vP hierarchy of perceived medical scientific sta- xl:,.,..R tus, as well as the important. The problem may also be that health professionals have wholly e. different assessment priorities from the well 3*00 informed consumers. If so, this would be a . form of supplier-induced demand that is 0

0 0 essentially unstudied in the health services -aa, co 2 00 literature. u

0

0) 0 1 0 50 SMALL AREA VARIATION AND UNCERTAINTY co The true extent of clinical uncertainty can 1 25 00 0 (n) often be measured by the amount of small area c 1 00 0 0 0 variations in hospital admission rates (fig 1), on cn C.' 00 0 the assumption that it is usually safe to assume .E 0 075 0 that important endogenous factors are stochas- co 0 a) tically constant. This is the "small area varia- 0 0 0 tion" argument'0 glibly put. Thus, we can take 0 50 0 away random variation and are left with a a)co 0 crude measure of uncertainty." As an index of C: 0

0 co) uncertainty, this is not a measure we are used http://jech.bmj.com/ to. To measure priorities for outcome research 0 there is a growing tendency to ask people what

0 the major areas are ripe for technology assess- 0 25 ment. Variations can give a crude index of existing Figure 1 Small area variations in standardised uncertainties as measured by practice varia- hospital admission ratios for various disorders. tion. Such a measure may also include genuine preferences for particular outcomes. But what on October 3, 2021 by guest. Protected copyright. Table 2 Magnitude of systematic variation (in ascending order) for selected caruses of it does not include is bias in the assessment of admission among 30 hospital market areas in Maine: 1980-82 priorities based on possibly illegitimate, sub- Variation Medical Surgical jective (and possibly self serving) hierarchies of relative importance. In this way a comple- Low: Inguinal hernia repair 1 5 fold range Hip repair ment to the assessment of priorities for out- Moderate: Acute myocardial infarction Appendectomy come research should, in my view, be utterly 2 5 fold range Gastrointestinal haemorrhage Major bowel surgery routine. If we do this, we discover the by now Cerebrovascular accident Cholecystectomy routine differences between countries in High: Respiratory neoplasms Hysterectomy 3-5 fold range Cardiac arrhythmias Major cardiovascular operations several very common reasons for hospital ad- Angina pectoris Lens operations mission and that the amount of variation Psychosis Major joint operations Depressive neurosis Anal operations between small areas corresponds sensibly with Medical back problems Back and neck operations what we actually know about uncertainties. Digestive malignancy Adult diabetes Examples are shown in table 2.12 Very high: Adult bronchiolitis Knee operations 8 5 + fold range Chest pain Transurethral operations Transient ischemic attacks Extraocular operations Minor skin disorders Breast biopsy ETHICS OF UNCERTAINTY Chronic obstructive lung disease Dilation and curettage From the of view of the appropriate Hypertension Tonsillectomy point Atherosclerosis Tubal interruption place of clinical trials in the evaluation of Chemotherapy treatment we now come across a major obstacle Reprinted with permission from the New England Journal of Medicine 1984;311:295-30/o. of principle. The formal method for assessing The best and the enemy of the good 9

priorities for research rests on experts express- certainly patients don't like it at all. In both J Epidemiol Community Health: first published as 10.1136/jech.48.1.6 on 1 February 1994. Downloaded from ing their own uncertainties. When these cases, the essential distinction between ignor- uncertainties are expressed and agreed (that is, ance and uncertainty is blurred, but for en- are explicit) the ethical basis for randomisation tirely different reasons. Patients are hard put is widely accepted. When uncertainties are to acknowledge, much less recognise, the true observed by epidemiological methods, as the extent of uncertainty when they have been led most plausible explanation for small area var- to believe that medicine is an exact and power- iation, the uncertainties are implied only, and ful science; which of course it mostly is. The may be an inadequate basis for ethical ran- medical profession is hard put to admit to domisation, at least as far as the responsible legitimate uncertainty when patients, on the clinician is concerned. whole, desperately hope for certainties. How- ever, medical advance is ill served in the long run by concealing uncertainties. Increasingly, WIDE VARIATION of course, financial pressure will require evid- The most important finding from small area ence of effectiveness and efficiency, which variation studies is that most hospital admis- brings me to my second proposal. sions are associated with variation as large or larger than for hysterectomy (table 3). Of course some of this variation may also be a Proposal 2: Establish a strong measure of preferences. It is not difficult to multi-disciplinary system of generating imagine systematic differences in preferences appropriate, explicit uncertainties, particularly between small areas with the current state of where unexpected both the knowledge of outcomes and of infor- Such a system will encompass the following mation provision. Hence, most admissions are kinds of work: likely to be associated with more uncertainty Statistical overviews than is commonly expressed or admitted, both Variation studies from the clinician and the consumer, as to Routine databases which is the appropriate treatment. Certainly, Proper audit the most widely varying procedures do come Decision analysis up in explicit hierarchies ofuncertainty but the Reviews of published reports common causes of hospital admission which Surveys of need and preferences vary importantly rarely do. From the point of Cost effectiveness studies view of establishing the appropriate role of Patient outcome research teams (PORT) randomised comparisons this might represent a temporary obstacle only, since the uncertain- ties are implied and not yet explicit. If they are APPROPRIATE CONTEXT legitimate, nonetheless, they must eventually Much is going on along these lines but there is become explicit. uncertainty that the appropriate context is always made clear. Thus, users of databases might well be disparaged by others, just as cost ADDITIONAL REASONS effectiveness studies might be ignored. The

To add to Archie Cochrane's possible reasons role of all of these methods has to be con- http://jech.bmj.com/ for not doing clinical trials there is now an sidered strictly in relation to the nature and the important new fourth, and incidentally, a fifth, extent of true current uncertainties. Cost that is anyway quite well understood, and I effectiveness studies, for example, are useless won't dwell on it. These reasons are as follows: with unreliable information about effective- (4) Uncertainty is not well enough estab- ness, if the plausible range includes zero or lished harmful effects. Likewise database analyses are (5) Reluctance to compare institutions or a waste of time (except in circumstances dis- on October 3, 2021 by guest. Protected copyright. carers. cussed below) when large, unconfounded, ran- domised comparisons are available. That is why organisations like the PORTs in (3) Studying medical uncertainty America"3 have been useful, because they en- As I will discuss later, the nature of the exis- able, in principle, the true research priorities tence of medical uncertainty is complicated for a particular outcome question to be pro- because on the whole it is disparaged. Medical perly evaluated, unconstrained by data or teachers do not like it, for obvious reasons, and disciplinary constraints. Then these objective priorities point to and release these constraints Table 3 Proportion of hospital admissions categorised where possible. At about the same time as by characteristic variation between small areas (data on Effectiveness and Efficiency2 was published, a 400 000 hospital admissions in Maine in the 1980s) seminal experience I had in America produced Amount of variation Typical admission % Admissions a book called Costs, Risk and Benefits of Sur- Low Hernia repair 1 gery,'4 which used many of the kind of, essen- Moderate Appendicectomy 9 tially ancillary, methodologies enumerated High Hysterectomy 42 Very high Disc removal 32 above. Archie Cochrane was asked to review it, Extremely high Tonsillectomy 16 of course, and his comment was: "An excel- 100 lent, courageous, pioneering book. I read it with profit and pleasure." So clearly Archie Reprinted with permission from the New England Journal of was also unambiguously not in favour of let- Medicine 1984;311:295-300. ting the "best become the enemy of the good". 10 McPherson J Epidemiol Community Health: first published as 10.1136/jech.48.1.6 on 1 February 1994. Downloaded from USA suggested by Feinstein et al,6 and as a conse- 300- quence, the explicit uncertainties are low but the implied uncertainties are very high. The criteria vary, the use rates vary, the costs vary, and presumably the outcomes vary. But nowhere did treatment for benign hypertrophy Denmark of the prostate appear on the upper echelons of Canada Norway research priorities - simply because most a) people thought they knew what they were a 200- New Zealand Australia doing. CDE Firstly, variation studies indicated a prob- 0 lem of implied uncertainties.'5 Secondly, basic 0 decision analysis, with comprehensive litera- 0 England and Wales ture reviews, strongly suggested that the cited Co0', liberal reason for intervening surgically in C,a) patients with minimal symptoms - to prevent Ireland the condition worsening and then suffering a : 100-_ Holland 0 less than optimal outcome for surgery - was just not justified.'6 The need for more reliable data was obvious but a clinical trial could not be contemplated, simply because the level of Sweden uncertainty among urologists was not admis- sible. Databases of complete follow up of all operations indicated quite remarkable discre- Figure 2 Crude rates for prostatectomy in various pancies in mortality for surgery than had ever countries in the 1980s. been published from case series in prestigious institutions. An unsuspected finding emerged concerning the new non-invasive operation UNCERTAINTY AND PROSTATECTOMY (the development of which was partly why Now, a much publicised part of this process as randomised comparison was not deemed to be it developed was an assiduous study of treat- indicated), suggesting a systematically higher ment for benign hypertrophy of the prostate. mortality in patients who underwent transur- Such treatments, of course, represent the mid- ethral resection than open operation'7 (table 4). dle ground of small area variation and the great Such evidence about relative efficacy and bulk of hospital admissions. It is precisely safety as this provides is, of course, wholly treatments like this for which problems of compromised by the possibility of selection; appropriateness and quality are almost com- the worse prognosis patients being offered the pletely uninformed by reliable data on out- more benign treatment while similar patients come. Practice styles have emerged and been are refused an open operation. Several studies established using exactly the kind of criteria since this publication have tried to adjust for

confounding by taking account of case mix, http://jech.bmj.com/ Table 4 One year age standardised mortality after prostatectomy in three large and the evidence is inconclusive. The latest, by databases, 1963-85 Alvan Feinstein et al,'8 seems to suggest case mix as the explanation but the study is far too No ofprocedures Mortality (%) small to refute an independent effect of the Transurethral resection Open prostatectomy surgery. '9 Denmark 36 000 7-6 5-7 Manitoba 12000 60 4-1 7-6 on October 3, 2021 by guest. Protected copyright. 5 000 10-3 ESTABLISHMENT VIEWS Reprinted with permission from the New England Journal of Medicine 1989;320:1120-4. Most interesting are the responses of the medi- cal/urological establishment to these results. In the USA a meeting of the executive council 35r of the American Urological Association, con- vened shortly after the Roos study was pub- 3-0O lished,'7 emerged with the following state- Co ment, which explicitly acknowledges the 0 2-5 E uncertainty associated with the choice of open 20k surgery and transurethral surgery. "Though the mechanisms of a direct causal Ln0 are not at this the 1 5[- relationship apparent time, -I weight of statistical evidence suggests that a ._na) I 4' >o randomized prospective study of international scope involving large numbers of patients, is 0-5 indicated." In my view this is the most rational response to these because the is Manitoba Denmark Manitoba New Haven HCFA Kaiser data, question clearly 1 2 Inc one of attribution which can only be resolved a randomised controlled trial.'0 In Figure 3 Adjusted relative risks of S year mortality after transuretheral and open by contrast, prostatectomy in various published studies. an editorial in the British Medical J7ournal" The best and the enemy of the good 11

does not budge an inch in the direction of Table 5 Common conditions for which patient J Epidemiol Community Health: first published as 10.1136/jech.48.1.6 on 1 February 1994. Downloaded from recognising any uncertainties whatsoever: preferences may be dominant "The fact that a recent review has suggested Benign conditions of the uterus Angina pectoris that TURP may not be as good as urologists Cataracts have claimed does not detract from the import- Benign hypertrophy of the prostate Gall stones ant advantages of this technique in treating Menopausal symptoms benign prostatic hyperplasia". Breast cancer A wonderful quote which manages to dis- miss the finding completely by calling it a "review" and then attributing a mere sugges- PATIENT PREFERENCE tion to that review. The important "advan- Meanwhile an important point receives little tages" of transuretheral prostatectomy sug- attention in the effectiveness literature; that for gested by the data from a complete some decisions patient preferences are domin- enumeration of all prostatectomy patients in ant while for others they are of no practical three databases are an excess in one year mor- consequence, except for extreme cases of re- tality of around 40%. Of course, since the data ligious bigotry. This depends of course on the are inconclusive and inconvenient they can be probabilities of different outcomes associated ignored, while inconclusive and convenient, with each choice and the nature of the out- but almost all small, selected, uncontrolled comes. For some decisions the probability of case series, can be cited, essentially as estab- death associated with one choice is unity, while lished. This surely ought not to happen. for another choice it is less than one in a thousand - which of course is no choice. How- ever, much in medicine is dominated by choice WHAT DOES THE DIFFERENCE MEAN? and hence it is hoped by communicating the One is, of course, left wondering whether the probabilities associated with different choices. observed difference in the databases is attribu- Some common examples are shown in table 5. table to the operation type, for there are many plausible explanations for the transuretheral prostatectomy really having a higher risk of THRESHOLD QUESTIONS mortality. These are mostly concerned with Most often the nature of the therapeutic choice hypothermia associated with unheated irriga- is not about which procedure, but is with the tion fluid, for instance, or the nature of the timing and whether to do something or not; it irrigation fluid itself. The results of the only is a threshold question, which dominates much randomised study22 is interesting because it is of the variation in observed rates (see fig 4). It consistent with an increased mortality. The is simple to demonstrate that quite small dif- five year mortality was 21% in the transureth- ferences in the threshold of signs or symptoms eral prostatectomy group and only 6% in the at which intervention happens on average can open prostatectomy group, however, the be responsible for dramatic differences in the numbers were very small, some 40 in the rates of intervention. By the nature of the transuretheral prostatectomy group and 30 in problem these differences in treatment policy the other. are seldom assessed unbiasedly by randomised A question we are left with is the extent to comparison; partly because the uncertainties http://jech.bmj.com/ which consumers of this kind of health care are not sufficiently stark and partly because would, given all the evidence outlined above, consumer choice is (often illegitimately23) cited actually choose a transuretheral prostatectomy as the driving force behind the decision. But in preference to an open operation. the cost differences are often enormous. The nature of the choices can be illuminated by using the methodologies in section (3), and (4) The role of patient choices in sometimes it can be seen that the medical decision making certainties are based on implausible theory. on October 3, 2021 by guest. Protected copyright. In deciding upon an appropriate medical or Rarely can the demonstration be sufficiently surgical treatment the choice obviously ought convincing to cause appropriate randomised to be determined by the probabilities of par- comparison, however. As was shown above if ticular outcomes associated with each choice, vested interests do not want to believe plaus- combined with the individual preferences for those outcomes. Where there is uncertainty about the true size of these probabilities, the Hi( procedure is complicated, not least because of c the demonstrated placebo effect of medical 0 certainty. That is to say, the paternalistic doc- tor may actually be doing his (gender chosen 0 cJ advisedly) patients more good than one who 0 a) expresses the uncertainties honestly. So per- C) haps the above editorial in the British Medical > 7ournalP is dedicated only to making patients -0~ with benign hypertrophy of the prostate better - who would otherwise not improve if they Low suspected that the treatment could possibly Symptom severity have a significant downside. I will pursue some Figure 4 Relationship of symptom severity and of the implications of this below. prevalence in prostatectomy. 1212 ~~~~~~~~~~~~~~~~~~~~~~~~~~McPherson

suboptimal. These preferences may all be J Epidemiol Community Health: first published as 10.1136/jech.48.1.6 on 1 February 1994. Downloaded from compatible with current medical knowledge which may be subject to very wide uncertain- ties (see above) nonetheless. Supplier induced CU~~~~~~~~~...... 6. demand clearly has strong theoretical, if not

0~~~~~~~~. empirical, support.24 co

V 4242~~~~~~...... 4-V...... E.

RELIABLE INFORMATION ~~~ ~ ~ ~ ~ ~ ~ .' So the solution is, of course, to evaluate the ..0. 0. outcomes to provide reliable information from which to make these decisions. In the short ..0 term this remains a forlorn hope, for reasons discussed above. Meanwhile, to test the above et have ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ hypothesis directly, Wennberg al5 ~~~ ~ ~ ~ ~ ~ ~ ~ ~ ~ produced interactive video discs designed to inform about the extent of current 1988.... 1989..1990 patients and associ- years...... scientific knowledge uncertainty .5 .Men .rostati .pe. aier.la ated with particular clinical decisions. We are evaluating this technology in the UK using randomised methodology. The questions we are trying to answer are the following. Firstly, do patients like having extra information and the chance to participate in the decision making process? Secondly, do the clinical decisions change as a consequence of providing this information? Thirdly, do patients do better, with respect to symptoms, survival, or quality of life, as a consequence? First indications in the USA are that the decisions made with more information are very different from those made without and that patients are in favour of the extra information. (See fig 5.)

ible but inconclusive argument, they will not. Hence randomisation is "unethical"! EFFECT OF CHOICE ON OUTCOME In the case of prostatectomy (fig 4) the A more fascinating aspect still of the role of operation at an early stage in the development in clinical decision making of urinary symptoms was thought to be advan- patient preference comes from the effect that choice tageous compared with operating later. By studying

on outcome. This is http://jech.bmj.com/ would be more intractable itself has therapeutic my then the symptoms third and surgery would be more dangerous. De- proposal. cision analysis and the combination of all avail- able data showed this to be a most implausible reality. However, the work required to reach Proposal 3: Investigate areas systematically this conclusion, in the absence of a randomised where treatment preferences affect outcome comparison of operating at different thresh- importantly olds, was both formidable and obviously I will cite two important examples of observa- on October 3, 2021 by guest. Protected copyright. essentially inconclusive. tions from clinical trials where the indirect evidence strongly suggests that choice itself effects hard outcomes such as survival.29 SUBOPTIMAL CHOICES Table 6 shows a summary of the results of This does raise the interesting suggestion, double blind clinical trials in which the outcome however, that if the determinants of treatment among patients given placebo is compared policy are driven ultimately by the supplier according to whether patients actually took the imposing a set of preferences that are different prescribed pills. Choosing to take the placebo from those of a well informed consumer of pills is then the treatment comparison in cir- services, then treatment choices may be cumstances when the patients did not know that they were on placebo. Both trials, after for all measured con- Table 6 Mortality of patients on placebo in double blind trials adjusting possible founders, give a strong hint that choice itself Patient Trial can effect measures of adherence dramatically important Coronary drug project research group" fi Blocker Heart Attack Trial outcome. There are many other examples of Sy mortality 1 y mortality this phenomenon in the published reports and Crude Adjusted Crude Adjusted the mechanisms could, of course, be various, a combination of immu- Poor 28-2 25-8 7-0 7-1 synergistic biology, Good 15-1 16-4 3-0 2-9 nology and pathology, with psychology30 much p

ness of treatment. We are familiar with the only to understand about outcomes and their http://jech.bmj.com/ idea that efficacy might give an euphoric estim- determinants, which is formidable enough, but ate of effectiveness, because things that matter also to assess routinely knowledge on the qua- might change outside a clinical trial for the lity of life and functioning from a patient's worse. I an now suggesting a possible counter- point of view. Most importantly, the main vailing influence where things might get better challenge is to create a decision making en- outside randomised comparisons. This would vironment which is capable of optimally using be true if and only if choice or control itself reliable information on and data about patient on October 3, 2021 by guest. Protected copyright. effects outcome. If true, another reason for not outcomes. generalising altogether from randomised trial is that they might thus be systematically biased. Hence a sixth reason for not wanting (6) Conclusions always to attribute outcomes by using ran- The evidence for common uncertainties in domised controlled trials is: clinical practice is overwhelming and, clearly, (6) Choice or preference or control affects important carving away at key parts of it is the outcome dominant component of Archie Cochrane's There are, of course, many plausible mechan- legacy. I have sought to draw attention to some isms by which such an effect could be import- of the potential problems in priority setting ant. The literature on the physiology of the and in the evaluation of treatments, where placebo effect is dominant here34 as well as the evidence, for instance, implicating social or professional control in coron- Table 7 Preference trials - a way of investigating the aetiology of choice ary heart disease.'5 (1) Measure therapeutic effect of choice, if any: for example randomised controlled trial of patients who have seen interactive video disc versus normal care (5) The role of choice in evaluating or (2) Wennberg proposal: randomised controlled trial of outcomes preferences with information versus randomised Clearly the role of choice or control needs to be controlled trial with information - for example surgery, drugs, watchful waiting systematically studied in the following con- 14 McPherson

consumer input is important and possibly neg- tectomy in English health regions in the J Epidemiol Community Health: first published as 10.1136/jech.48.1.6 on 1 February 1994. Downloaded from lected. I believe that these considerations are 1970s.38 These variations in practice style have important methodologically as well as substan- yielded no information whatsoever about their tively and their methodological implications consequences for the patients, and there must are poorly investigated as yet. be some. The true role of uncertainty in medicine has yet to be properly investigated also, and this is partly because it fits uneasily into the domin- OBSERVATIONAL METHODS ant medical paradigm. Much of what happens I believe that some of the observational metho- in medical school would change, much of what dology proposed is, in principle, cheap and happens in clinical practice would change, could be routine. It is not so cheap now much that happens in media coverage of medi- because one has to develop mechanisms not cine would change, and perhaps much of what already in existence to capture all cases and to happens in the evaluation of outcome would allow useful follow up of outcome as well as to change, if the true nature and extent of uncer- attract the collaboration of clinicians. It cannot tainty in medicine were readily acknowledged. replace randomised controlled trials except in Perhaps effectiveness itself would change too, circumstances where the attribution of cause but until we know when and how, and in with respect to outcome is unambiguous, and which direction, there seems little point in not this will sometimes happen. We will have to investigating whether it would. Certainly the get better at determining the causes of politics of health care planning would change. observed differences by knowing about the potency of plausible confounders.39 What I am proposing is systematic study of enormous STYLE DIFFERENCES numbers of cases so that even rare and long My final proposal is again something I have term outcomes can be systematically com- often suggested which is to take greater advan- pared. tage of a manifestation of uncertainty, dif- ferences in practice style. CRITERION FOR RANDOMISED COMPARISON Proposal 4: Practice style differences exploited Randomised comparisons are only practically in natural experiments of outcome, where feasible when the major primary outcome(s) of randomisation is not feasible interest need to be unambiguously attributed Systematic practice style differences are com- to treatment choice and cannot be attributed mon and they exist because clinicians adopt without them. We are often concerned with different criteria for intervening, largely as a secondary outcomes (like long term rare side consequence of assuming that informal, theor- effects) here and nobody is going to randomise etical, and anecdotal methods of assessment of patients when the primary outcome compari- outcome are good enough. Since they exist it sons (functional status at 30 days for example) always seems an enormous waste not to take are essentially, or even approximately, under- advantage of these natural experiments where stood. What is often required is a clear cohort treatment at the margin is not chosen, as David collected at a time when there are uncertainties

Byer would have described things, entirely by about the primary questions of outcome attri- http://jech.bmj.com/ objective measures of prognosis but much bution and hence when the cohort is relatively more as a random consequence of history. uncontaminated by biased clinical selection on Figure 6 shows variations in the use of prosta- the basis of prognosis. An example is the collaboration with the 130 NW Thames Royal College of Surgeons in their audit of all South West cases of prostatectomy for benign hypertrophy SW Thames ay 120 Yorkshire of the prostate. For the first time, and essen- on October 3, 2021 by guest. Protected copyright. NE Thames tially incited by the other observational work 110 Oxford on already quoted, they are a, Wessex prostatectomy CLQ 100 SE Thames studying outcome on some 4000 consecutive a) East Anglia cases. The hypotheses being investigated are E 90 Wales 0 to do with the characteristics of patients who 0 Northern 0 80_ Trent might benefit from this operation from among 0 those now being offered the operation. Testing 0 70 transuretheral prostatectomy against open sur- CDa) North West co gery remains a long way off I suspect. Clearly 60 -0 West Midlands large numbers are not in themselves enough, a) somewhat .a_ 50 but it emerges (conditioned by pos- E. sible response bias) that fascinating dif- ferences, and in some cases fascinating similar- :o ities, in outcomes themselves raise important c) hypotheses to be tested individually in the future.

POSTAL ASSESSMENTS what emerges clearly from this Figure 6 Age standardised rates for prostatectomy in However, England and Wales. study, and another I have recently completed The best and the enemy of the good 15 J Epidemiol Community Health: first published as 10.1136/jech.48.1.6 on 1 February 1994. Downloaded from Table 8 Response rates to a postal questionnaire in 388 men who underwent 10 Wennberg JE, Gittlesohn A. Small area variations in health care delivery. Science 1975;182:1 102-8. prostatectomy 11 McPherson K, Wennberg JE, Hovind OB, Clifford P. Small area variations in the use of common surgical Time procedures: an international comparison of New England, England, and Norway. N Engl I Med 1982;307:1310-4. At surgery At 3 mth At 6 mth At 12 mth 12 Wennberg JE, McPherson K, Caper P. Will payment based on diagnostic related groups control hospital costs. N Engl Patients (%) 100 93 92 91 J Med 1984;311:295-300. Surgeons (%) 99 81 79 100 13 Wennberg JE. What is ? In: Gelijns AC, ed. Medical innovations at the crossroads. Vol I Modern methods of clinical investigation. Washington DC: National Academy Press, 1990:33-46. 14 Bunker J, Barnes B, Mosteller F. Costs, risks and benefits of surgery. Oxford: Oxford University Press, 1977. 15 Wennberg J, Roos N, Sola L, Schori A, Jaffe R. Use of in collaboration with Nick Black and others, is calim's data systems to evaluate health care outcomes: that patients who have received treatment are mortality and reoperation following prostatectomy. JAMA 1987;257:933-6. very willing to complete, even detailed, postal 16 Barry MJ, Mulley AG, Fowler FJ, Wennberg JE. Watchful questionnaires about the state of their health.40 waiting vs immediate transurethral resection of the pro- state for symptomatic prostatism. JAMA 1988;259:3010- It seems slightly scandalous to waste the good 17. will evidenced in table 8 in obtaining informa- 17 Roos NP, Wennberg JE, Malenka DJ, et al. Mortality and reoperation after transurethral resection of the prostate tion which could prove to be extremely valu- for benign prostatic hyperplasia. N Engl Jf Med able. Record linkage has often been imple- 1989;320:1 120-4. 18 Concato J, Horwitz RI, Feinstein AR, et al. Problems of mented to obtain follow up information on comorbidity in morality after prostatectomy. JAMA outcome, but it has obvious limitations. 1992;267: 1077-82. 19 Roos N, Roos L, Cohen M, et al. Therapies for benign prostatic hyperplasia (letter). JAMA 1992;268(10):1269. 20 Bracken MB. Clinical trials and the acceptance of uncer- Proposal 5: (Postal) outcome assessments by tainty. BMJ 1987;294:1111-2. 21 Chisholm GD. Editorial. Benign prostatic hyperplasia: the patients become routine, with and without best treatment. BMJ 1989;299:215-6. treatment 22 Meyhoff HH. Transurethral versus transvesicle prostatec- tomy: a randomised study. Scand Jf Urol Nephrol Thus, if collected in a uniform and systematic 1987;4(S102):26-32. manner, basic information not now available in 23 Coulter A, McPherson K. Socioeconomic variations in the use of common surgical operations. BMJ 1985;291:183-7. any analysable form could usefully inform out- 24 Wennberg JE, Barnes BA, Zubkoff M. Professional uncer- come research. This must eventually include, tainty and the problem of supplier induced demand. Soc Sci Med 1982;16:811-24. of course, patients with symptoms not deemed 25 Kasper JP, Mulley AG, Wennberg JE. Developing shared appropriate for intervention. decision making programs to improve quality of health care. Quality Review Bulletin. Journal of Quality Im- provement 1992;18:182-90. I conclude with a summary of the main 26 Sidney S, Queensberry CP, Sadler MG, et al. Reoperation and mortality after surgical treatment of benign prostatic points I have tried to make which encapsulate hypertrophy in a large medical care program. Med Care a hierarchy of methodologies designed simply 1992;30: 117-25. 27 Winslow R. Videos, questionnaires aim to expand role of to establish the effectiveness of health inter- patients in treatment decisions. Wall St Journal, 25 ventions. These main points are: February 1992. 28 Coronary Drug Project Research Group. Influence of ad- (1) Monitor practice variations. herence to treatment and response of cholesterol on (2) Enhance multi-disciplinary generation mortality in the coronary drug project. N Engl J Med 1980;30: 1038-41. of appropriate uncertainties. 29 Horowitz RI, Viscolli CM, Berkman L, et al. Treatment (3) Determine where treatment preferences adherence and risk of death after a myocardial infarction. outcome. Lancet 1990;336:542-5. affect 30 Philips DP, Todd RE, Wagner LM. Psychology and survi- http://jech.bmj.com/ (4) Undertake natural experiments of prac- val. Lancet 1993;342:1142-5. 31 Asher L, Harper HW. Effect of human chorionic gonado- tice style. trophin on weight loss, hunger, and feeling of well-being. (5) Routine long term postal follow up for Am J Clin Nutrition 1973;26:211-18. 32 Fuller R, Roth H, Long S. Compliance with disulfiram important questions can be valuable. treatment of alcoholism. Journal of Chronic Disease (6) Randomised trials for small, important, 1983;36: 161-70. 33 Pizzo PA. Oral antibiotic prophylaxis in patients with possibly confounded effects are essential. cancer: a double-blind randomized placebo-controlled trial. J Pediatr 1983;102(1):125-33.

1 Cochrane AL with Blythe M. One man's medicine. London: 34 Wolf S. Effects of suggestion and conditioning on the action on October 3, 2021 by guest. Protected copyright. Memoir Club, BMJ, 1989. of chemical agents in human subjects - the pharmacology 2 Cochrane AL. Effectiveness and efficiency. Random reflec- of placebos. J Clin Investigation 1950;29:100-9. tions on health services. London: Nuffield Provincial Hos- 35 Siegrist J, Peter R, Junge A, et al. Low status control, high pital Trust, 1972. effort at work and ischaemic heart disease: prospective 3 Chalmers I, Dickerson K, Chalmers TC. Getting to grips evidence from blue-collar men. Soc Sci Med with Archie Cochrane's agenda. BMJ 1992;305:786-8. 1990;31(10):1 127-34. 4 Early Breast Cancer Trialists' Collaborative Group. Sys- 36 Brewin CR, Bradley C. Patient preferences and randomised tematic treatment of early breast cancer by hormonal, clinical trials. BMJ 1989;299:313-5. cytotoxic, or immune therapy. An overview of 133 ran- 37 Henshaw RC, Naji SA, Russell IT, Templeton AA. Com- domised trials involving 31,000 recurrences and 24,000 parison of medical abortion with surgical vacuum aspi- deaths among 75,000 women. Lancet 1992;339:1-15,71- ration and acceptability of treatment. BMJ 1993;307:714- 85. 7. 5 Anonymous. Databases for health care outcomes. Lancet 38 McPherson K. Variations in hospitalization rates: why and 1989;335:195-6. how to study them. London: The King's Fund Institute, 6 Feinstein AR. Current problems and future challenges in 1988. RCTs. Circulation 1984;70(5):767-74. 39 McPherson K. Diversity and similarity of health: organisa- 7 Dollery C. An end of an age of optimism. London: Nuffield tion, practice and assessment. In: Normonds C, Vaughan Hospital Hospitals Trust, 1978. P, eds. Europe without frontiers. Chichester: J Wiley, 8 Green SB, Byer DP. Using observational data from regis- 1993. tries to compare treatment: the fallacy of omnimetrics. 40 Doll HA, Black NA, McPherson K, et al. Mortality and Stat Med 1984;3:361-70. complications following transurethral resection of the 9 Byar DP. Problems with using observational databases to prostate for benign prostatic hypertrophy. J Urology compare treatments. Stat Med 1991;10:663-6. (Balti) 1992;147:1566-73.