Assessing the effectiveness of new technologies Adrian K Dixon Introduction • Technical performance; Rapid changes in the way in which • Diagnostic performance; healthcare is delivered continually • Diagnostic impact; necessitate different methods of • Therapeutic impact; and assessing new technology. Economical • Impact on health. issues have become paramount. Perhaps even more than mere fi scal assessment comes the need to avoid Technical performance hospitalisation and reduce the number The fi rst level is that of technical of patient contacts with expensive performance, which assesses whether secondary care. Whether these fi nancial the new equipment or technique does imperatives are perceived by the patient deliver what it is expected to do on as optimal often remains unanswered, the technical front. For a new piece even in these days of supposed ‘patient of diagnostic imaging equipment, this choice’. The main methods of new might assess whether or not the new There should be no let technology assessment are discussed machine yields anatomical images of in this article, along with possible future spatial (and/or contrast) resolution equal assessments in the light of recent to or better than existing equipment3. For up in the quest for even changes in the way in which diagnostics nuclear medicine and other functional are being introduced into the United imaging techniques this might assess better techniques for Kingdom (UK). the additional physiological data that is obtained. For a new interventional identifying women at New technology stent, it might be a more mechanical assessment about tensile strength and risk from this dreaded assessment biocompatibility. The original hierarchical fi ve level method For the average radiology department, disease (breast cancer). of assessing new technology within the there is little involvement in such imaging fi eld is now well established with assessments as manufacturers only minor variations1,2. These are will not bring novel techniques or 12 IMAGING & ONCOLOGY | 2007 technologies to the market place numbers of true negative fi ndings in without all such technical performance normal women. The predictive value of information available. However, research a positive result is not all that high and departments become involved with this means that a fair number of normal assessments of prototypes, and some of women still have to undergo a traumatic these may have been developed by their biopsy. The sensitivity for some of the staff members. more aggressive lesions is rather lower and MRI may be better, especially in younger women6. Consequently, despite Diagnostic performance the considerable advances which have Diagnostic performance, namely how arisen as a result of the NHSBSP, there well the new technique fares with should be no let up in the quest for even regards to making the diagnosis, is better techniques for identifying women often regarded as the be all and end at risk from this dreaded disease. all of technology assessment. It is The assessment of diagnostic often regarded, erroneously, as being performance is relatively straightforward synonymous with the diagnostic in breast screening where a fi nal accuracy of the new technique. It is diagnosis (cancer or no cancer) is now realised that studies describing defi nitively established. Many workers overall accuracy are signifi cantly have pointed out how diffi cult this infl uenced by the prevalence of disease becomes when the diagnosis is elusive in the population under scrutiny. In and the patient may not necessarily fact, the markers of most importance undergo early biopsy or surgery – viz are the sensitivity and specifi city of magnetic resonance imaging (MRI) in the new investigation and these are the diagnosis of multiple sclerosis7. Even often combined to produce a receiver for something relatively straightforward operator curve for the new investigation4. such as MR of the knee, the fact that Again, many prestigious research only a selection of patients undergo Whilst the overall departments become involved in such arthroscopy hinders the assessment of early assessments. But of course, the diagnostic performance8. accuracy is very high prevalence of various disease processes in these institutions may be far removed (NHSBSP), this is based from the real world of a typical general Diagnostic impact hospital. Hence, there is need for If a new technique successfully passes on the very large numbers early, large, multicentre studies of new through the above two stages of methodologies, which will include a assessment, it should be possible to spectrum of different practices. prove that it helps make an impact of true negative fi ndings in The data now available from the very on the clinician’s diagnosis, either by high quality UK National Health Service providing a new, unexpected diagnosis normal women. Breast Screening Programme (NHSBSP) or by improving the clinician’s confi dence provides interesting conclusions: whilst in their working clinical diagnosis. Such the overall accuracy is very high5, information is extremely diffi cult to obtain this is based mainly on the very large unless the confi dence in the working 2007 | IMAGING & ONCOLOGY 13 diagnosis is established before the Therapeutic impact a randomised trial14,15. However, fi rst imaging investigation. Historically, Now that the investigation under scrutiny randomised trials are notoriously diagnostic confi dence information has has passed the fi rst three levels of the diffi cult to perform in diagnostic been obtained in patients who were technology assessment hierarchy, it radiology and ethical review referred for VQ scintigrams for possible must be shown to have therapeutic boards may refuse to pulmonary embolus (PE) where the a value. If, after an investigation has sanction a study where one priori clinical probability was necessary been performed, the clinician ends arm of patients is ‘denied’ in order to provide a defi nitive report. up doing what he or she would have access to the diagnostic This made it relatively simple to compare done anyway, it could be argued that test under scrutiny, no and prove the benefi cial diagnostic the test was unnecessary. In the days matter how new or impact of computed tomography (CT) when radiologists had to scrutinise experimental. pulmonary angiography9 and has led requests assiduously to avoid too to general acceptance in the UK that many referrals and a departmental Impact on nuclear medicine (NM) referrals for overspend, they frequently asked ‘will possible PE should now be limited to the result of this investigation change health those patients with a relatively normal your management?’. Indeed, this is It should follow that any chest radiograph, with all other patients still a very valid question, particularly investigation of proven directed towards CT. when radiation exposure is at stake11,12. technical and diagnostic Relatively few clinicians state their Again, clinicians must be encouraged performance with positive possible differential diagnosis in general to state before an investigation details clinical and therapeutic impact referrals for imaging examinations, let about their proposed management; would be associated with a alone their confi dence in the leading and they must also be asked what their benefi cial impact on health. But diagnosis. Only when this information management will be once the result of this may be diffi cult to prove: the side is demanded at the outset can the true the investigation is apparent. Only by this effects of surgery may mask the health diagnostic impact of a new investigation method can the true therapeutic impact gain in the immediate post-operative be measured. Some might argue that all be measured. This can be achieved by period and sometimes the diagnosis may imaging is aimed at reducing diagnostic means of a prospective observational have such an unfavourable outcome (for dangerous investigations. The use of uncertainty10 and that any investigation study13 but a purer study is where the example, pancreatic cancer) that the MRI instead of Endoscopic Retrograde which improves diagnostic confi dence new investigation is assessed alongside benefi cial infl uence of the diagnostic Cholangiopancreatography (ERCP) is a ‘good thing’. But an investigation a conventional investigation on the test is masked by the natural history of is one example; the development which merely confi rms the clinician’s same patient population by means of the disease. Even for benign disease, of fl uid sensitive techniques diagnosis may be an unnecessary luxury. it is diffi cult to prove the benefi t of allowed the introduction of MR Alternatively, an investigation which high technology diagnosis in terms cholangiopancreatography (MRCP)18, completely alters the clinician’s diagnosis of improvement in quality of life16. It which has virtually replaced diagnostic (say from benign disease to probable is much easier to prove the benefi cial ERCP. Interestingly, this change in cancer) may completely change the infl uence of interventional radiological practice has come about with no patient’s clinical course – and save the techniques, where great advances have full health technology assessment clinician the embarrassment of going been made recently17. As a result, several – merely common sense! In these days up the wrong diagnostic alley. And such surrogate markers have been used to of patient choice, it may be that we changes in diagnosis (diagnostic impact) assess impact of new techniques on should ask the patients which test they must be measured as radiologists and health. Clearly, the avoidance of ionising would prefer – all other information clinicians need to justify the considerable radiation is one marker; and another being equal; the presumed preference expense of diagnostic certainty. may be the avoidance of potentially for MR over myelography was never 14 IMAGING & ONCOLOGY | 2007 formally assessed but can be assumed. in imaging capacity, fi rst on the back subjective clinical examination, even for However, the preference for MR over of various cancer initiatives and now something simple such as the presence some other techniques may not be as from additional independent sector or absence of an abdominal mass14.