Educational Workshop EW11: Development of molecular assays: do's & don't's arranged with the ESCMID Study Group for Molecular Diagnostics (ESGMD) Convenors: Marijjyke Raymaekers ( Hasselt,,) BE) Paul Savelkoul (Amsterdam, NL) Faculty: Anton M. van Loon (Utrecht, NL) Kate Templeton (Edinburgh, UK) Marijke Raymaekers (Hasselt, BE) Udo Reischl (Regensburg, DE) Paul Savelkoul (Amsterdam, NL) Van Loon - QC controls Development of Molecular Assays: Do’s and Don’t’s QC controls Anton M van Loon Department of Virology, University Medical Centre Utrecht, the Netherlands QCMD, Glasgow Scotland 1 Anton M van Loon, ECCMID 2010 Molecular Diagnostics is hot! Scientific impact •The human genome project •Elucidation of new pathogenesis pathways •Discovery of new infectious agents Increased public awareness •Criminal investigations and court room cases (Castro –Ponce, Nienke K, etc.) • Fuelled by Film and TV industry (CSI, ‘Waking the Dead’, 6th Day, etc.) 2 Anton M van Loon, ECCMID 2010 DNA tells the truth General perception: DNA doesn’t lie –or does it? •Famous court cases, for example Castro –Ponce •Incorrect results on BCRA‐1/2 detection • Missed HIV diagnosis in neonate • Conflicting reports in literature, for instance on the etiology of chronic fatigue syndrome, ALS, schizophrenia, etc. So, maybe DNA does not lie, but it still requires human efforts to develop assays, to collect and investigate samples, and to interpret the results: need to monitor quality of the overall process! 3 Anton M van Loon, ECCMID 2010 3 Van Loon - QC controls Quality Concerns In Molecular Diagnostics • Sensitivity and specificity •Genotypic variation • Contamination • Clinical significance •Large number of in‐house assays •Lack of robustness and standardisation •Lack of international reference material 4 Anton M van Loon, ECCMID 2010 PCR False‐Positives: early years 5 Anton M van Loon, ECCMID 2010 Valenthine‐Thon, JCV, 2002 Improvements to molecular diagnostics • Anti-contamination measures • Technological developments: reagents, automation, real-time platforms • Introduction of commercial assays • International Standards (unfortunately so far only available for BBV’s) • Use of universal internal controls • External quality assessment programmes Anton M van Loon, ECCMID 2010 6 4 Van Loon - QC controls Need for Standardisation Standardisation: the case of NASA’s Mars Orbitor The likely cause...... Lack of standardization when determining thruster impulse One team used pound force (Imperial unit) Other team used Newton (Metric measurement). One pound force = 4.45 Newton ……. Therefore the spacecraft's trajectory were 4.4 times greater than NASA navigation Mars team believed. Source: Report NASA mission failure investigation board November 1999 7 Anton M van Loon, ECCMID 2010 Units used for reporting HCV, HIV‐1 and HBV viral load results in EQA distributions 2002 & 2008 Geq/ml 100% Meg/ml pg/ml 90% IU/ml 80% Geq/ml 70% IU/ml 60% 50% HCV HIV HBV 40% Copies/ml Copies/ml 30% 20% 10% Copies/ml 0% 2002 2008 2002 2008 2002 2008 8 Anton M van Loon, ECCMID 2010 Need for Reference Materials: Accuracy of Molecular Assays Virus International SD range of % of commercial Standard geometric mean assays (log10) HIV Y 0.17 – 0.27 > 95% HCV Y 0.20 – 0.30 > 95% HBV Y 0.30 – 0.45 ~ 50% BKV N 0.5 – 0.6 < 15% HSV N 0.6 – 0.7 < 15% EV N >1.0 < 10% Anton M van Loon, ECCMID 2010 9 5 Van Loon - QC controls Run Controls and Internal controls • Run controls: assess the validity of the results of a particular run • Positive control sample • Low-positive control sample • Negative control sample • Internal controls: assess the validity of the results of every sample • Incorporation (spiking) of an internal control in specimens allows for control of the effect of inhibitors inhibitors, and thus improves accuracy and precision. • Provides valuable information on new equipment, reagents, isolation procedures, and new or improved methods • Homologous or heterologous IC’s (target/ non-target) • DNA targets • RNA targets 10 Shewart plot positive run control JCV‐PCR Shewartplot 40 38 36 34 32 aarde 30 w 28 Ct- 26 24 22 20 29-11-2006 18-1-2007 9-3-2007 28-4-2007 17-6-2007 6-8-2007 25-9-2007 14-11-2007 Data Mean Ct value: 29.7 + 0.6 Range Ct value: 28.5 – 30.9 11 Anton M van Loon, ECCMID 2010 Shewart plot positive run control TOX‐PCR Shewartplot 40 38 36 34 e 32 30 t-waard 28 C 26 24 22 20 10-okt- 29-nov- 18-jan- 9-mrt-07 28-apr- 17-jun- 6-aug-07 25-sep- 14-nov- 3-jan-08 22-feb- 06 06 07 07 07 07 07 08 Data Mean Ct value: 30.3 + 0.55 Range Ct value: 29.2 – 31.4 12 Anton M van Loon, ECCMID 2010 6 Van Loon - QC controls Universal Internal Controls DNA viruses Phocine HerpesVirus 1 (PhHV) RNA viruses Encephalomyocarditis virus (EMCV) 13 Molecular diagnostic assay setup Department of Virology UMC Utrecht Clinical samples Run controls Negative controls (spiked with PhHV or EMCV as universal internal control) MNAPMagNAPure Exttitraction Taqman Amplification and Detection Diagnostic Target PhHV EMC Anton M van Loon, ECCMID 2010 14 Comparison various clinical samples Ct values internal control PhHV‐1 MagnaPure LC 38 36 * 34 * 32 ABI7700 on 30 values Ct 28 26 24 Plasma Serum CSF Faeces Swabs Urine 15 Anton M van Loon, ECCMID 2010 Data: Bert Niesters 7 Van Loon - QC controls Ct values Internal Controls 16 Anton M van Loon, ECCMID 2010 Data: Bert Niesters External Quality Assessment (EQA/PT) •Examination of blinded panels, distributed by EQA organisations •Integral part of a laboratory quality assurance procedures •Helps identify weak spots in a laboratory’s general performance •Helps improves reliability of results, and gives confidence in reporting results • Allows comparison of performance with other laboratories & methods •Provides a reference level in the absence of International Standards. • Mandatory requirement in many countries. 17 Anton M van Loon, ECCMID 2010 Under requirements of ISO 15189… ISO 15189 : Medical laboratories – Particular requirements for quality and competence 5645.6.4: “Th“ The la boratory s ha ll part ic ipate in inter la boratory comparisons… . The laboratory shall monitor the results… . Inter laboratory comparison programs shall be in substantial agreement with ISO/IEC Guide 43-1.” Anton M van Loon, ECCMID 2010 8 Van Loon - QC controls External Quality Control in Europe : 1998 (1) • No uniform regulatory system for medical laboratories across Europe (and there still isn’t one) • QC organisations for external proficiency testing in many countries (NEQAS, INSTAND, EQUALIS, SKMM, Labquality, etc.) • Differences in approaches, but regular proficiency testing mainly focused on (virus)culture and serology. • Lack of International Standards or reference reagents for molecular testing • Molecular methods: high rate of false-positives ( > 20 % ); sensitivity often unclear Anton M van Loon, ECCMID 2010 19 External Quality Control in Europe : 1998 (2) • European Union sponsored Concerted Action for Quality Control (EU-QCCA) of Molecular Methods in Diagnostic Virology ( co-ordinator G Cleator): Start in March 1998 • Annual programme of proficiency panel distributions for limited number of agents (HSV, VZV, EV, CMV, BBV’s) and CT • Endorsement by European Society for Clinical Virology and major diagnostic industries (ILC) 20 Anton M van Loon, ECCMID 2010 External Quality Control in Europe : 2001 • EU sponsoring of EU-QCCA would stop by September 2001; at that time, no suitable EU program identified for further sponsoring (5th FP: QL, Growth; DG-SANCO) • Request for continuation (ESCV, ESCMID, Industry) • Business plan prepared, including grants from Industry • QCMD : Quality Control for Molecular Diagnostics, an independent, not-for-profit organization run by experts from the field (www.qcmd.org) • Since 2005 TUV UKAS certified to ISO 9001: 2000 21 Anton M van Loon, ECCMID 2010 9 Van Loon - QC controls Quality Control for Molecular Diagnosis: QCMD AIM • To establish an international external quality assessment programme for evaluation of existing and evolving nucleic acid amplification procedures in diagnostic microbiology STRATEGY • Preparation and distribution of proficiency panels • Provision of run control programmes • Support in development of reference materials, international standards Anton M van Loon, ECCMID 2010 22 QCMD: organisational structure QCMD EXECUTIVE ADVISORS TO THE BOARD SCIENTIFIC COUNCIL Scientific organizations ESCV, ESCMID QCMD CENTRAL OFFICE World QC Organisations EXECUTIVE OFFICE NEUTRAL OFFICE Industrial Liaison Committee (ILC) Contractors Program Participants 23 Anton M van Loon, ECCMID 2010 The QCMD EQA panels • QCMD prepares a panel of artificial samples containing various amounts of virus • Panel materials prepared or selected by expert laboratories, including new agents/ variants (CT, hMPV, HPeV, INFL A H5N1) • Panel materials subjected to in-process and independent release testing • Panel materials (prefereably) inactivated and lyophilised • Ships to lab without key for sample identification • Labs test the panel and report results by mail, fax or web to QCMD’s Neutral Office (> 99% via web) • QCMD sends to lab key for sample identification • Results from labs are analysed; a report is prepared and sent to labs 24 Anton M van Loon, ECCMID 2010 10 Van Loon - QC controls The QCMD EQA cycle December Sample … following Berlin Program Design 2006 meeting, October 2006 Selection February August Quality Control 2007 2007 Participant and Reports Validation The QCMD Pre-Analytic EQA cycle (JCV/BKV) stage Analysis of Sample March results (Neutral 2007 Distribution