8th European HIV Drug Resistance Workshop
M. Zazzi on behalf of the EuResist Network (www.euresist.org)
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy EuResist status
Funded by the EU JAN-06 to JUN-08, then set as a European Network (legal entity)
Data collected from ~40,000 patients (Italy, Germany, Sweden, Luxembourg, Belgium, Spain)
Data modeling by IBM Israel, Max Planck Institute for Informatics, Informa & Rome TRE University
Several methods investigated, much effort on feature selection and derivation
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy EuResist – TCE definition
CD4 Viral load Genotype Treatment switch Viral load
time 0 to 12 weeks Short-term model: 4-12 weeks Pre-therapy HIV RNA Patient demographics (age, gender, race, route of infection) Past AIDS diagnosis Past treatments Past genotypes
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy EuResist – labeling therapies
Baseline data HIV genotype at 0 to 12 weeks before treatment VL at 0 to 12 weeks before treatment Additional variables when available
Treatment switch
VL at 4 to 12 weeks (8-week outcome)
SUCCESS FAILURE Undetectable or >2 log Detectable and not >2 decrease VL log decrease VL
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy EuResist – engines
Three prediction engines developed independently Generative-Discriminative (by IBM) Evolutionary (by Max-Planck Institute) Mixed effects (by Rome TRE & Informa)
Then, combined into a unique engine and made freely available on the web
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy The EuResist combined engine
3143 therapies, Short-term outcome (8 weeks)
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy Rosen-Zvi, Bioinformatics 2008 EuResist vs. Expert interpretation (EVE study)
Form the invitation letter:
The requested response include a categorical (C) answer and a quantitative (Q) estimate:
C) Given this HIV genotype and patient information, will the indicated therapy be successful (i. e. will it make HIV RNA decrease
by at least 2 logs or to undetectable levels in 8 weeks) ?
Q) Given this HIV genotype and patient information, what probability of success would you estimate for the indicated therapy?
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy EuResist vs. Expert interpretation (EVE study)
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy Distribution of the 25 therapies by year and type
5 4 No. of 3 cases 2 1 1998 1999 2000 2001 2002 2003 2004 2005 2006
Year of therapy
NRTI-only therapy NNRTI-based therapy *PI-based therapy
*17 cases with boosted PI, 2 unboosted ATV, 1 NFV Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy
Patient characteristics
FEATURE MEDIAN (IQR) Baseline log viral load 4.67 (4.38-4.99) Baseline CD4 counts 298 (134-412) Number of previous treatment lines 5 (3-6) Number of NRTI mutations at baseline 3 (3-4) Number of NNRTI mutations at baseline 1 (0-2) Number of PI mutations at baseline 2 (0-3)
Number of available previous viral load data 15 (8-25) Number of available previous CD4 counts 14 (10-30) Number of available previous genotypes 1 (0-3)
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy Distribution of wrong and correct calls
25 HAART cases randomly selected form the EuResist db:
•Obsolete therapies excluded
•Wild type genotype excluded •All clinical and virological information available
12 experts enrolled, response obtained from 10:
•On‐line anonymous rating •Only European (E) vs. non‐ European (N) setting traceable
•Use of any interpretation system allowed (and declared)
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy Sensitivity, specificity and accuracy
76%
Specificity = ability to call failure Sensitivity = ability to call success
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy 100
90
80
70
60 Average Expert Best Expert 50 EuResist Worst Expert Sensitivity 40
30
20
10
0 0 10 20 30 40 50 60 70 80 90 100 100-Specificity
AUC 95% CI Best Expert 0.853 0.655 - 0.961
EuResist 0.787 0.578 - 0.923 P = 0.011 Average Expert 0.777 0.567 - 0.917
Worst Expert 0.653 0.438 - 0.830
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy EuResist vs. Expert interpretation (EVE study)
Correlation between EuResist and avg(expert) probability of success Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy EuResist vs. Expert interpretation (EVE study)
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy EuResist vs. Expert interpretation (EVE study) EuResist & most experts incorrect
EXPERT 1 EXPERT 2 EXPERT 3 EXPERT 4 EXPERT 5 EXPERT 6 EXPERT 7 EXPERT 8 EXPERT 9 EXPERT 10 EuResist
ACTUAL OUTCOME F S F S F F F S S S S S S S S S F S F F S S F F S Incorrect prediction F Treatment failure S Treatment success
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy EuResist vs. Expert interpretation (EVE study) Unexpected drug efficacy
EXPERT 1 EXPERT 2 EXPERT 3 EXPERT 4 EXPERT 5 EXPERT 6 EXPERT 7 EXPERT 8 EXPERT 9 EXPERT 10 EuResist
ACTUAL OUTCOME F S F S F F F S S S S S S S S S F S F F S S F F S Incorrect prediction F Treatment failure S Treatment success
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy EuResist vs. Expert interpretation (EVE study) Adherence issues?
EXPERT 1 EXPERT 2 EXPERT 3 EXPERT 4 EXPERT 5 EXPERT 6 EXPERT 7 EXPERT 8 EXPERT 9 EXPERT 10 EuResist
ACTUAL OUTCOME F S F S F F F S S S S S S S S S F S F F S S F F S Incorrect prediction F Treatment failure S Treatment success
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy Why do we err (so much)?
Limitations in the definition of success can “create” prediction errors Expected adherence cannot be accounted for based on presently available training data set Genotype shortcomings Impact of past mutations Short-term drug activity on partially resistant strains (e. g. 215 revertants, Y181C with EFV) Unweighted factors (host genetics)
Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy Perspectives
EuResist is being updated regularly and expanding Other European countries Non-European countries
Engines being retrained with the expanded dataset (~40,000 patients) Small increase in accuracy ○ Upper bound of accuracy already reached? Tipranavir, Darunavir, Etravirine now computed Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy If you have data and data sharing attitude, please consider becoming a EuResist partner!
www.euresist.org Presented at the 8th European HIV Drug Resistance Workshop, March 17-19 2010, Sorrento, Italy