Calibration and Discrimination of the Framingham Risk Score and the Pooled Cohort Equations

RESEARCH

Calibration and discrimination of the Framingham Risk Score and the Pooled Cohort Equations

Dennis T. Ko MD MSc, Atul Sivaswamy MSc, Maneesh Sud MD, Gynter Kotrri BHSc, Paymon Azizi MSc, Maria Koh MSc, Peter C. Austin PhD, Douglas S. Lee MD PhD, Idan Roifman MD MSc, George Thanassoulis MD MSc, Karen Tu MD MSc, Jacob A. Udell MD MPH, Harindra C. Wijeysundera MD PhD, Todd J. Anderson MD n Cite as: CMAJ 2020 April 27;192:E442-9. doi: 10.1503/cmaj.190848

ABSTRACT

BACKGROUND: Although accurate risk who underwent cholesterol testing and Score and 3.51% by the Pooled Cohort prediction is essential in guiding treat- blood pressure measurement from Equations at 5 years overestimated ment decisions in primary prevention of Jan. 1, 2010, to Dec. 31, 2014. We com- observed event rates by 101% and atherosclerotic cardiovascular disease, pared predicted event rates generated by 115%, respectively. The degree of over- the accuracy of the Framingham Risk the Framingham Risk Score and the estimation differed by age and ethnicity. Score (recommended by a Canadian Pooled Cohort Equations with observed The C statistics for the Framingham Risk guideline) and the Pooled Cohort Equa- event rates at 5 years using linkages from Score (0.74) and Pooled Cohort Equa tions (recommended by US guidelines) validated administrative databases. tions (0.73) were similar. has not been assessed in a large contemporary Canadian population. Our RESULTS: Our study cohort included INTERPRETATION: The Framingham primary objective was to assess the cali- 84 617 individuals (mean age 56.3 yr, Risk Score and Pooled Cohort Equations bration and discrimination of the Fram- 56.9% female). Over a maximum follow- significantly overpredicted the actual ingham Risk Score and Pooled Cohort up period of 5 years, we observed 2162 risks of atherosclerotic cardiovascular Equations in Ontario, Canada. (2.6%) events according to the outcome disease events in a large population definition of the Framingham Risk from Ontario. Our finding suggests the METHODS: We conducted an observa- Score, and 1224 (1.4%) events according need for further refinement of cardio- tional study involving Ontario residents to the outcome definition of the Pooled vascular disease risk prediction scores aged 40 to 79 years, without a history of Cohort Equations. The predicted event to suit the characteristics of a multieth- atherosclerotic cardiovascular disease, rate of 5.78% by the Framingham Risk nic Canadian population.

ccurate individual risk estimation is essential in guiding cardiovascular events but do not consider the potential compet- informed treatment decisions in primary prevention, ing impact of noncardiovascular deaths during follow-up. The because treatment has greater benefit in those at accuracy of commonly used risk scores has also not been evalu- increasedA risk of developing the disease.1,2 Primary prevention ated in a large contemporary Canadian cohort and has not been guidelines in cardiology around the world have endorsed the use of measured using newer statistical techniques. Our main objective risk prediction scores to estimate the risk of atherosclerotic cardio- was to assess the calibration and discrimination of the Framing- vascular disease as an initial step in assessing the need for preven- ham Risk Score and Pooled Cohort Equations in Ontario, Canada. tive treatment.2–6 The Canadian Cardiovascular Society (CCS) Second, we aimed to assess the validity of these risk scores in pre- guideline recommends using the Framingham Risk Score to assess specified subgroups to gain additional perspectives. risk of atherosclerotic cardiovascular disease.2 The American Col- lege of Cardiology/American Heart Association (ACC/AHA) practice Methods guidelines currently endorse the use of the Pooled Cohort Equa- tions over the Framingham Risk Score for risk estimations.1,4 Study design and data sources Since the Framingham cohort was assembled more than We performed an observational study of Ontario adults to 3 decades ago, there have been substantial declines in the risk of evaluate the calibration and discrimination of the Framingham cardiovascular events.7 Furthermore, these risk scores predict Risk Score and Pooled Cohort Equations using the Electronic

E442 CMAJ | APRIL 27, 2020 | VOLUME 192 | ISSUE 17 © 2020 Joule Inc. or its licensors RESEARCH - E443 This cohort cohort This 2 The observed rate was The observed rate was 16 Third, because smoking status was Third, because smoking status was 14 We generated 95% confidence intervals (CIs) of (CIs) of We generated 95% confidence intervals 17 ISSUE 17 ISSUE | We compared 5-year predicted and observed rates of predicted and observed rates of We compared 5-year 15 The US guidelines, using the Pooled Cohort Equations, Equations, The US guidelines, using the Pooled Cohort 18 A series of additional analyses was performed. First, we First, we A series of additional analyses was performed. included individuals aged 40–75 years, who did not have diabetes, whose low-density lipoprotein (LDL) cholesterol was less than 5 mmol/L, who did not have chronic kidney disease and who were not taking statins. We also excluded those with substantial comorbidities (i.e., history of cancer and receiving hemodialysis) because statins may not be indicated in these subgroups. Analyses were conducted at ICES using SAS version 9.4 and R version 3.1.2. categorized 10-year risk as less than 5%, 5%–7.5%, 7.5%–10% categorized 10-year risk as less than 5%, 5%–7.5%, 7.5%–10% and greater than 10%. missing for about 15% of included individuals, we performed missing for about 15% of included individuals, we performed an additional multiple imputation analysis using 10 imputed data sets. Fourth, we repeated our analyses in a “statin-eligible” guidelines. CCS 2016 the to according cohort observed risk using a Wald-based approach with Aalen variance with Aalen observed risk using a Wald-based approach in the predicted estimator. The relative percentage of difference (discordance = to observed risk was defined as discordance rate). Calibration was (predicted rate – observed rate)/observed the predicted versus also assessed qualitatively by plotting risk. We assessed dis- observed risks across deciles of predicted using the R package crimination by calculating the C statistic risks. (timeROC), which took into account competing in prespecified sub- evaluated the accuracy of the risk scores there were discrep- groups (age and sex) to evaluate whether we compared the ancies in the discordance ratios. Second, - cardio and observed rates of atherosclerotic 5-year predicted The Canadian vascular disease in different risk categories. Score, categorized guidelines, using the Framingham Risk than greater and 10%–20% 10%, than less as risk 10-year 20%. Ethics approval This project was approved by the research ethics board at Sun- nybrook Health Sciences Centre, Toronto. Statistical analysis Statistical Equa- Score and Pooled Cohort Framingham Risk For both the which refers to the agreement tions, we assessed calibration, predicted risk, and discrimination, which between observed and risk different between differentiates model well the to how refers levels. date of Dec. 31, 2016, until individuals had 5 years of follow-up, follow-up, of years 5 had individuals until 2016, 31, Dec. of date out- disease cardiovascular to unrelated cause a from death until an end point had occurred. come or until calculated using the Kalbfleisch–Prentice estimator of cumula of estimator Kalbfleisch–Prentice the using calculated - for the competing risk of nonout tive incidence that accounts for causes noncirculatory to due deaths as (defined death come Risk Score and deaths from causes validating the Framingham for validating the other than myocardial infarction or stroke shown to generate Pooled Cohort Equations), which has been estimate in a simu- estimates that are closer to the true survival lation study. atherosclerotic cardiovascular disease using the equations from disease using the equations from atherosclerotic cardiovascular Score and Pooled Cohort Equations, and the Framingham Risk function at 5 years. the baseline survival VOLUME 192 VOLUME | These data These data 9 ginal weightingginal and had blood and had blood APRIL 27, 2020 | The Pooled CohortPooled The 12,13 The ori CMAJ 14 The database is housed housed The database is 8 /cmaj.190848/-/DC1), and the /cmaj.190848/-/DC1), and the We excluded patients with prior We excluded patients with prior 1 10,11 - Nonresidents of Ontario and those with 10 fatal myocardial infarction or stroke. myocardial fatal ian Institute for Health Information (CIHI) Discharge Information (CIHI) Discharge ian Institute for Health out a valid Ontario Health Card were excluded. We also excluded. We also out a valid Ontario Health Card were care physicians had excluded individuals if their primary 1 year to ensure we been enrolled in EMRALD for less than medical records on had complete capture of the electronic eligible individuals. Equations outcome definition included a composite of death due foradmission hospital and stroke, and disease heart coronary to non of the risk factors for the Framingham Risk Score and Pooled Pooled and Score Risk Framingham the for factors risk the of Cohort Equations are shown in Appendix 1 (available at www. cmaj.ca/lookup/suppl/doi:10.1503 codes used to define outcomes are shown in Appendix 1, Supple- mental Table 1. Follow-up continued until either the study end

Outcomes To assess the accuracy of the Framingham Risk Score, we used the Framingham outcome definition that included a composite of - infarc myocardial for admission hospital and death, circulatory tion, angina, cerebrovascular events (ischemic stroke, hemor- rhagic stroke and transient ischemic attack), heart failure and peripheral artery disease (also including abdominal aortic aneur stent). or endarterectomy carotid and ysm Study cohort care primary their visited who years 40–79 aged Individuals 2014, physician from Jan. 1, 2010, to Dec. 31, pressure and cholesterol measurements taken within a year taken within a year pressure and cholesterol measurements This age restric- of each other were considered for inclusion. recently developed tion was chosen to be consistent with the Pooled Cohort Equations. Medical Record Administrative Data Linked Database Database Linked Data Administrative Record Medical study cohort. the identify to (EMRALD) hospital admissions for myocardial infarction, cerebrovascu- hospital admissions for myocardial infarction, disease and cor- lar disease, heart failure, peripheral vascular the date individuals onary revascularization from 1988 to entered the cohort. at ICES and includes electronic medical records of more than more than medical records of includes electronic at ICES and 500 000 indi- for more than care physicians caring 350 primary to various data set was linked Ontario, Canada. This viduals in the included the following: data sets that administrative - Database, a registry of Ontario res Ontario Registered Persons under the Ontario Health Insurance Plan; idents with coverage created using information from the Can multiple databases ad - capture comorbid conditions and hospi Abstract Database to Ontario Cancer Registry to capture prior tal admissions; the the Registrar General of Ontario Database diagnosis of cancer; - of death; the Ontario Laboratories Infor to determine cause laboratory results; and the Can mation System to determine data- Resident Permanent Citizenship and Immigration adian of the ethnicity of landed immigrants base, which includes surname algorithm to Ontario, used with the application of a study. determine ethnicity of individuals in the sets were linked using unique encoded identifiers and ana- sets were linked using unique encoded these databases lyzed at ICES. Additional information about can be found elsewhere. RESEARCH E444 myocardial infarction, PAD=peripheralarterydisease, PCI=percutaneouscoronaryintervention. Electronic Medical Record Administrative Data Linked Database (EMRALD) cohort. Note: CABG = coronary artery bypass grafting, HF = heart failure, MI = Figure 1:Creationflowchart.Atotalof84 617 individualswereincludedinourstudycohortafterinclusionand exclusion criteriawereappliedtothe disease eventsaccordingtothePooledCohortEquationsout- and 0.74inwomen. tic oftheFraminghamRiskScorewas0.74overall,0.69inmen estimated riskby110%inmenand85%women.TheCstatis- relative discordanceof101%.TheFraminghamRiskScoreover- Score was5.78%,leadingtoanabsolutedifferenceof2.9%anda mean 5-yearpredictedriskasestimatedbytheFraminghamRisk circulatory death was 2.88% (95% CI 2.76%–3.00%) (Table 2). The 5-year incidencerateaccountingforthecompetingriskofnon- the FraminghamRiskScoreoutcomedefinition.Theobserved 2162 atheroscleroticcardiovasculardiseaseeventsaccordingto During thefollow-upperiod(maximumof5yr),weobserved Score andPooledCohortEquations Predicted andobservedeventsforFraminghamRisk 1.1% wereblack. were of Chinese ethnicity, 1.6% were of SouthAsianethnicity and 16.9% werecurrentsmokers(Table1).Oftheparticipants,2.2% tolic bloodpressurewas127mmHg,13.4%haddiabetesand 5.1 mmol/L,meanLDLcholesterolwas3.1sys- were women. The participants’ mean total cholesterol level was teria, ourcohortincluded84 617individualsinthegeneralcohort. cohort (Figure1).Afterweappliedtheinclusionandexclusioncri- of eachotherfrom2010to2014werecapturedintheEMRALD pressure measurementandcholesterolassessmentwithin1year A totalof133 892Ontarioresidentsallageswhohadblood Results During thefollow-upperiod,weobserved1224cardiovascular The meanageofthegeneralcohortwas56.3years,and56.9% each other, Ontario residents Ontario residents and lipid history values collected CMAJ from of cardiovasculardiseases Main cohort n =133 n =8461 n =99 2010 EMRALD who aged 40–79yrwithout | APRIL27, 2020 to 2014,identifiedby had 892 7 5 Missing blood • • Excluded within HF, Prior Age pressure 1 yrof smoking PCI, < history

40 or≥80yrn=26981 n =33 | VOLUME 192 CABG a

overall studycohort(Figure2). Framingham RiskScoreandPooledCohortEquationsinthe 0.72 inwomen(Table2).Therewasalackofcalibrationforthe of thePooledCohortEquationswas0.73overall,0.67inmenand relative overestimationof115%inallindividuals.TheCstatistic mated bythePooledCohortEquationswas3.51%,resultingina were observed.Themeanpredictedoveralleventrateasesti- event ratewas1.63%(95%CI1.53%–1.72%),and1443events come definition.The5-yearobservedPooledCohortEquations 2 riskscoreswassimilarformenandwomen. individuals (232%inthegroupaged75–80yr).Discordanceof group aged40–45yr)andamuchlargerdiscordanceinolder tions hadsmallerdiscordanceinyoungeragegroups(51%the older agegroup(75–79yr).Incontrast,thePooledCohortEqua- est agegroup(40–45yr)andasmallerdiscordanceof44%inthe Framingham RiskScorehadadiscordanceof183%intheyoung- and riskthresholdsareshowninFigure3.Foragestrata,the The plotsofoverallcalibrationinsubgroupsage,sex,ethnicity Cohort Equationsaccordingtostrata Calibration oftheFraminghamRiskScoreandPooled Equations overestimatedriskby118%. estimated riskby104%inallindividuals,andthePooledCohort dix 1,SupplementalTable4).TheFraminghamRiskScoreover- actual riskofeventsaftersmokingdatawereimputed(Appen- and thePooledCohortEquationssignificantlyoverestimated smoking statuswereexcluded,boththeFraminghamRiskScore Similar totheoriginalanalysisinwhichindividualswithmissing Validation ofriskscoresaftersmokingimputation information 897 of hospital

or PAD

n =6916 | admission ISSUE 17

n =15378

for MI,

stroke, RESEARCH E445 0.74 0.72 0.67 0.69 0.73 0.74 C statistic Women 5.3 ± 1.1 1.6 ± 0.4 3.1 ± 0.9 691 (1.4) 516 (1.1) 28.6 ± 7.0 1140 (2.4) 56.3 ± 10.1 n = 48 184 7138 (14.8) 8675 (18.0) 9141 (19.0) 9685 (20.1) 5654 (11.7) 7085 (14.7) 5616 (11.7) 125.8 ± 17.6 10 292 (21.4) 12 748 (26.5) 28 768 (59.7) 12 331 (25.6) 11 204 (23.3) 85 107 119 110 115 101 Discordance, %* Discordance, 1.1 1.7 2.9 4.5 1.9 2.9 Men 5.0 ± 1.1 1.3 ± 0.4 3.0 ± 0.9 746 (2.0) 652 (1.8) 398 (1.1) Absolute Absolute 29.4 ± 5.8 56.1 ± 10.0 n = 36 433 5639 (15.5) 6401 (17.6) 7010 (19.2) 7900 (21.7) 9302 (25.5) 8081 (22.2) 5657 (15.5) 7186 (19.7) 9486 (26.0) 6453 (17.7) 129.6 ± 16.7 17 211 (47.2) 12 036 (33.0) difference, % difference, ISSUE 17 ISSUE | No. (%) of participants* (%) of No. VOLUME 192 VOLUME (95% CI) | Observed, % Observed, 1.03 (0.93–1.12) 1.98 (1.84–2.11) 2.43 (2.26–2.60) 4.09 (3.87–4.31) 1.63 (1.53–1.72) 2.88 (2.76–3.00) Overall 5.1 ± 1.1 1.4 ± 0.4 3.1 ± 0.9 914 (1.1) 28.9 ± 6.5 1886 (2.2) 1343 (1.6) 56.3 ± 10.1 n = 84 617 127.4 ± 17.3 12 777 (15.1) 15 076 (17.8) 16 151 (19.1) 18 192 (21.5) 22 050 (26.1) 17 766 (21.0) 11 311 (13.4) 45 979 (54.3) 14 271 (16.9) 24 367 (28.8) 20 690 (24.5) 12 069 (14.3) APRIL 27, 2020 | (95% CI) CMAJ Predicted, % Predicted, 2.13 (2.10–2.15) 3.67 (3.64–3.70) 5.33 (5.28–5.39) 8.57 (8.50–8.65) 3.51 (3.48–3.54) 5.78 (5.74–5.82) Pooled Cohort Equations Pooled Women Risk Score Framingham Pooled Cohort Equations Pooled Men Risk Score Framingham Pooled Cohort Equations Pooled Variable Table 2: Five-year predicted and observed events of the Framingham Risk Score and Pooled Cohort Equations and Pooled Risk Score the Framingham of events and observed predicted 2: Five-year Table Note: HDL = high-density lipoprotein, LDL = low-density lipoprotein, SD = standard deviation. SD = standard lipoprotein, LDL = low-density HDL = high-density lipoprotein, Note: otherwise. *Unless stated Age, yr, mean ± SD mean yr, Age, Characteristic Table 1: Baseline characteristics of Ontario residents who had blood pressure measurement and cholesterol assessment assessment and cholesterol measurement pressure who had blood residents Ontario of 1: Baseline characteristics Table 2014 2010 to from other each of within 1 year Income quintile Income 1 (lowest) 2 3 4 5 (highest) residency Rural Ethnicity Chinese South Asian Black ± SD mmol/L, mean Lipid measurements, cholesterol Total HDL cholesterol LDL cholesterol mellitus Diabetes Body mass index, mean ± SD mean Body mass index, ± SD mm Hg, mean blood pressure, Systolic Smoking Nonsmoker Smoker smoker Former Medications Antihypertensive Statin interval. CI = confidence Note: × 100. event rate – observed event rate)/observed event rate as (predicted is calculated *Discordance Overall Risk Score Framingham

RESEARCH E446 Each dotshowspredicted versusobservedrateineach declineofrisk.Thesolidlineshows whenpredictedriskisequalto observed risk. Figure 2: Predictedandobserved 5-year riskforFramingham Risk Score and Pooled Cohort Equations in the overall cohort (40–79 yr), bydecile of risk. mated itmoreinolderpatients.Giventheimportanceofaccu- younger patients,whereasthePooledCohortEquationsoveresti- statins. The Framingham Risk Score overestimated risk more in study cohort,aswellinsubgroupsofage,sexandeligibilityfor actual observedrisk.Overestimationwasseenintheoverall sclerotic cardiovasculardiseasethatweremorethandoublethe Score andthePooledCohortEquationspredictedrisksofathero- primary care setting, we found that both the Framingham Risk Using alargecontemporarycohortofindividualsinanOntario Interpretation and 0.74forthePooledCohortEquationsinthiscohort. risk by83%.Cstatisticswere0.75fortheFraminghamRiskScore individuals, whereasthePooledCohortEquationsoverestimated ple, theFraminghamRiskScoreoverestimatedriskby129%inall who could be considered eligible for statin treatment. For exam- tion ofriskthanthePooledCohortEquationsamongpatients In general,theFraminghamRiskScorehadgreateroverestima- tics inthiscohortisshownAppendix1,SupplementalTable3. tions (15%v.24.6%). uals intheoverallcohorttobetakingantihypertensivemedica- Hg and17.3%weresmokers.Theylesslikelythanindivid terol was3.1mmol/L,meansystolicbloodpressure126mm mean totalcholesterollevelwas5.25mmol/L,LDLcholes- 53.8 years(v.56.3yr);58.6%werewomen.Theparticipants’ cohort wasyoungerthantheoverallcohort,withameanageof characteristics ofthe59 126individuals.Thestatin-eligible guidelines, andAppendix1,SupplementalTable2showsthe the cohort of individuals eligible for statins according to CCS Appendix 1, Supplemental Figure 1 details the construction of statin-eligible individuals Characteristics andvalidationofriskscoresamong A summaryofthecalibrationanddiscriminationcharacteris-

5-year predicted event rate, % 10 12 14 16 18 20 0 2 4 6 8 0 246 5-year observedeventrate,% Framingham RiskScore 81 01 CMAJ 21 41 | APRIL27, 2020 61 82 0 | VOLUME 192 lected inthe1970s. data onindividualsresidinginFramingham,Massachusetts,col- treatment decisions. risk scoresforatherosclerotic cardiovascular disease toimprove prediction wouldstronglysupportredevelopmentof validation studiesacrossCanada.Evidenceofconsistentover- diovascular disease,ourfindingssuggesttheneedforadditional rate riskestimationinprimarypreventionofatheroscleroticcar- endorses QRISK 78%. 51% andthePooledCohortEquationsoverestimatedriskby Coronary HeartDiseasepredictionoverestimatedactualriskby sclerosis (MESA) cohort, DeFilippis found that the Framingham that theparticipantswereenrolledfrom1971to1976. findings arelikelynotapplicabletocontemporarypracticegiven and discriminationoftheFraminghamRiskScore.However,these data on1173Canadianparticipants,andfoundgoodcalibration from the Lipid Research Clinics Follow-up Cohort, which included score calledPREDICT. atic CoronaryRiskEvaluation(SCORE), rently, theEuropeanSocietyofCardiologyendorses the System- risk scoresthatarebettercalibratedtotheirpopulations.Cur- among blackAmericans. rate riskestimationsofatheroscleroticcardiovasculardisease contemporary, diversecohortsfromtheUSthatenabledaccu- developed thePooledCohortEquationsbyassemblingmore resentative ofacontemporaryUSpopulation, theACC/AHA cohorts. several studiesthatexaminedvalidationofthesescoresusingUS Score andPooledCohortEquationsareconsistentwiththoseof data onCanadianparticipants. the performanceoforiginalFraminghamRiskScoreusing The FraminghamRiskScorewasinitiallydevelopedusing Our results showing overestimation of the Framingham Risk

5-year predicted event rate, % 21 To the best of our knowledge, only 1 study has evaluated Tothebestofourknowledge,only1studyhasevaluated 10 12 14 16 18 20 0 2 4 6 8 19–23 02468 Forexample,usingtheMulti-EthnicStudyofAthero- | ISSUE 17 3 andNewZealandhasrecentlycreatedanew 5-year observedeventrate,% 12 Becausetheseindividualsmaynotberep- 6 Pooled CohortEquations 14 Otherjurisdictionshavedeveloped 24 Grover and colleagues used data Groverandcolleaguesuseddata 10 12 5 theUnitedKingdom 14 16 24 18 20 RESEARCH E447 75–79 ≥ 10 General 70–75 Male 65–70 7.5–10 outh Asian 60–65 Sex Ethnicity Observed Observed Observed –7.5 55–60 hineseS Age category, yr Observed 50–55 Female Pooled Cohort Equations Pooled Cohort Predicted Predicted Predicted < 55 10-year predicted risk threshold, % BlackC 45–50 Predicted ISSUE 17 ISSUE

| 8 6 4 2 0 8 6 4 2 0 8 6 4 2 0

40–45

20 18 16 14 12 10 10 10

% risk, 5-year % risk, 5-year % risk, 5-year

8 6 4 2 0

14 12 10

% risk, 5-year VOLUME 192 VOLUME | 75–79 General ≥ 20 APRIL 27, 2020 | 70–75 Male 65–70 CMAJ South Asian 60–65 Sex 10–20 Ethnicity 55–60 Observed Observed Observed Chinese Age category, yr Observed 50–55 Female Framingham Risk Score Framingham < 10 Predicted Predicted Predicted Black 10-year predicted risk threshold, % 45–50 Predicted

8 6 4 2 0 8 6 4 2 0 8 6 4 2 0

40–45 20 18 16 14 12 10 10 10

% risk, 5-year % risk, 5-year % risk, 5-year

8 6 4 2 0

14 12 10

% risk, 5-year A D C B

Predicted and observed 5-year risk for the Framingham Risk Score and Pooled Cohort Equations by (A) age category, (B) sex, (C) ethnicity and Figure 3: Predicted and observed 5-year risk for the Framingham Risk Score and Pooled Cohort Equations (D) risk threshold recommended by practice guidelines at 10 years. RESEARCH developed in6.1%ofblackparticipantsat10years. come developedin5.3%ofwhiteparticipants,andanoutcome mated bloodpressuremeasurement, Third, althoughCanadianfamily physicianscommonlyuseauto- to ensureourfindingscanbeextendedoveralongerduration. from 2010to2014.Additional follow-up of thiscohortis needed who hadcholesteroltestingandbloodpressuremeasurement have 10-yearfollow-upbecauseourcohortincludedindividuals Score andPooledCohortEquationsat5years,butwedidnot contemporary cohort and we validated theFraminghamRisk ings. Second,wewereabletoestimatecardiovascularriskin a Canada wouldbeimportanttoensuretherobustnessofourfind - have producedmorebiasedestimates. accounted forcompetingrisksofnoncardiacdeath,whichcould Framingham RiskScorenorthePooledCohortEquations vices, whichisavailabletomostresidents.Finally,neitherthe benefited fromuniversal coverage ofhospitalandphysicianser- control ofcardiovascularriskfactors.Third,ourOntariocohort Second, individualslikelyhadearlierdiagnosisandimproved cohort. estimated the actual risks of cardiovascular events in our minority groupsinCanada. Chinese or South Asians, which are the most common ethnic of theexistingriskscoresincludedethnicminoritiessuchas E448 provinces. a primarycarephysicianorthoseresidinginotherCanadian have notshownthattheyarerepresentativeofpatientswithout the Ontario patients who had a primary care physician, but we ous research,wehaveshownthattheywererepresentativeof may bedifferentfromthegeneralOntariopopulation.Inprevi- it ispossiblethatpatientswhosedataareincludedinEMRALD Several potential limitations of our study merit discussion. First, Limitations part explained by the year the cohorts were assembled. monly usedriskscoresalsofoundthatthediscrepancywasin 1.4%. Researcherswhorecalibratedtheperformanceof4com- cohort and a Pooled Cohort Equations outcome developed in ham RiskScoreoutcomedevelopedinonly2.6%ofourstudy we observedmuchlowereventratesat5yearsinthataFraming- 12 years. disease outcomedevelopedin11.4%oftheparticipantsat were derived.IntheFraminghamHeartStudy,acoronaryheart disease eventsoccurlessfrequentlythanwhenthosescores contemporary periodinwhichatheroscleroticcardiovascular contributed to an overestimationofrisk. usually higherthanintheresearch setting,andthiscouldhave our studycohort.Bloodpressure readingsinclinicalpracticeare tion onthewaybloodpressure measurementswereobtainedin mates usinganage-squaredterminthepredictionmodel. ancy maybeattributabletothePooledCohortEquationsriskesti- mated riskamongindividualsinolderagegroups.Thisdiscrep- whereas thePooledCohortEquationssubstantiallyoveresti- overestimated risk among individuals in younger age groups, There maybeseveralreasonswhyexistingriskscoresover We observed that the Framingham Risk Score substantially 20,25–27 12 Inthepooledcohort,aPooledCohortEquationsout- 8 Accordingly,additionalstudiesusingcohortsacross First,individualsinourstudywererecruiteda 28 wedidnothaveinforma- CMAJ 29 Fourth,theCCS rec- | APRIL27, 2020 14 Incontrast, 14 None None 27

| VOLUME 192 inflated risk. information wouldleadtofurtheroverestimationofthealready recommendation completely.However,theuseoffamilyhistory mation onfamilyhistory,wewereunabletoduplicatethisCCS References of adiverseCanadianpopulation. atherosclerotic cardiovasculardiseasetosuitthecharacteristics refine thecurrentlyrecommendedpredictionriskscoresfor history ofcardiovasculardisease.Furthereffortsareneededto Pooled CohortEquationsinanOntariopopulationwithoutprior vascular diseaseriskwithuseoftheFraminghamRiskScoreand Our findingssuggestoverestimationsofatheroscleroticcardio- Conclusion

16. 15. 14. 13. 12. 11. 10. sclerotic cardiovascular disease. doubles theriskifthereisafamilyhistoryofprematureathero- ommends amodificationoftheFraminghamRiskScorethat 6. 2. 3. 9. 8. 5. 4. 1. 7.

management ofdyslipidaemias.EurHeartJ2016;37:2999-3058. Catapano AL,GrahamI,DeBackerG,etal.2016ESC/EASguidelinesforthe blood cholesterol.JAmCollCardiolNov.2018;doi:10.1016/j.jacc.2018.11.003. Grundy SM,StoneNJ,BaileyAL,etal.2018Guideline on themanagementof of QRISK2.BMJ2012;344:e4181. the UnitedKingdom:independentandexternalvalidationofanupdatedversion Collins GS,AltmanDG.Predictingthe10-yearriskofcardiovasculardiseasein cular Diseaseintheadult.CanJCardiol2016;32:1263-82. Guidelines fortheManagementofDyslipidemiaPreventionCardiovas- Anderson TJ,Gregoire J, PearsonGJ, et al.2016Canadian Cardiovascular Society adults. JAmCollCardiol2014;63:2889-934. treatment ofbloodcholesteroltoreduceatheroscleroticcardiovascularriskin Stone NJ,RobinsonJG,LichtensteinAH,etal.2013ACC/AHAGuidelineonthe poses? ACanadianperspective.BMCMedInformDecisMak2015;15:67. electronic medicalrecordssuchthatthedatacanbeusedforsecondarypur- Tu K, Widdifield J, Young J, et al.Arefamily physicians comprehensively using 2017;33:378-84. vascular diseasesinOntario:ClinicalandHealthSystemsImpact.Can J Cardiol Tu JV,KhanAM,NgK,etal.Recenttemporalchangesinatheroscleroticcardio- study. in 400 Pylypchuk R,WellsS,KerrA,etal.Cardiovasculardiseaseriskpredictionequations cardiovascular disease PooledCohortRiskEquations.JAMA2014;311:1406-15. Muntner P,ColantonioLD,CushmanM, etal.Validationoftheatherosclerotic 33:517-35. bration oflogisticregressionmodelsby usingloesssmoothers.StatMed2014; Austin PC,SteyerbergEW.Graphicalassessment ofinternalandexternalcali- Circulation 2014;129:S49-73. Cardiology/ assessment ofcardiovascularrisk:areporttheAmericanCollege Goff DCJr,Lloyd-JonesDM,BennettG,etal.2013ACC/AHAguidelineonthe 2008;117:743-53. for useinprimarycare:theFraminghamHeartStudy.Circulation D’Agostino RBSr,VasanRS,PencinaMJ,etal.Generalcardiovascularriskprofile using riskfactorcategories.Circulation1998;97:1837-47. Wilson PWF,D’AgostinoRB,LevyD,etal.Predictionofcoronaryheartdisease 2015;132:1549-59. immigrants toOntario,Canada:theCANHEARTImmigrantStudy.Circulation Tu JV,ChuA,RezaiMR,etal.Theincidenceofmajorcardiovasculareventsin 8:204-12. vascular healthandhealthcareservices.CircCardiovasc Qual Outcomes 2015; Research Team(CANHEART):usingbigdatatomeasureandimprovecardio- Tu JV, Chu A, Donovan LR, et al. The Cardiovascular Health in Ambulatory Care Res Methodol2010;10:42. ethnicity fromsecondarydatainOntario,Canada:avalidationstudy.BMCMed Shah BR,ChiuM,AminS,etal.SurnameliststoidentifySouthAsianandChinese Lancet 2018;391:1897-907. 000 primarycarepatientsinNewZealand:aderivationandvalidation | ISSUE 17 American HeartAssociationTaskForce onPracticeGuidelines. 2 Because we did not have infor-

RESEARCH E449 The authors are indebted to Dr. Jack V. Tu V. Tu Jack to Dr. The authors are indebted ISSUE 17 ISSUE | This study was supported by ICES, which is funded by an by funded is which ICES, by supported was study This Kaczorowski J, Myers MG, Gelfer M, et al. How do family physicians measure Gelfer M, et al. How do family physicians measure Kaczorowski J, Myers MG, practice? National survey of Canadian family blood pressure in routine clinical 2017;63:e193-9. physicians. Can Fam Physician Myers MG. Comparing automated office blood pres- Roerecke M, Kaczorowski J, of blood pressure measurement for identifyingsure readings with other methods meta-analysis. and review a systematic hypertension: possible with patients JAMA Intern Med 2019;179:351-62. Pennells L, Kaptoge S, Wood A, et al. Equalization of four cardiovascular risk A, et al. Equalization of four cardiovascular risk Pennells L, Kaptoge S, Wood recalibration: individual-participant meta-analysis algorithms after systematic 2019;40:621-31. of 86 prospective studies. Eur Heart J Grover SA, Hemmelgarn B, Joseph L, et al. The role of global risk assessment in risk assessment role of global L, et al. The B, Joseph SA, Hemmelgarn Grover therapy. Can J Cardiol 2006;22:606-13. hypertension the impact Risk score overestimation: Young R, McEvoy JW, et al. DeFilippis AP, on the per- and preventive therapies cardiovascular risk factors of individual Cardiology- College of American Heart Association-American formance of the risk score in a modern multi-ethnic Cardiovascular Disease Atherosclerotic 2017;38:598-608. cohort. Eur Heart J et al. Evaluation of the Pooled Cohort Risk EquationsMora S, Wenger NK, Cook NR, Women’s the from cohort multiethnic a in Prediction Risk Cardiovascular for 2018;178:1231-40. Health Initiative. JAMA Intern Med

29. 28. 24. 25. 26. 27. Award from the Heart and Stroke Foundation of Canada. Douglas Lee isAward from the Heart and Stroke Foundation of JacobOutcomes. Function Heart in Chair Rogers Ted the by supported Clin Udell is supported in part by a National New Investigator/Ontario Foundation of Canadaician Scientist Award from the Heart and Stroke of Ontario. Karenand an Early Researcher Award from the Government from the Department ofTu is supported by a Research Scholar Award of Toronto. Harindra Family and Community Medicine, University Award from Heart andWijeysundera is supported by a Clinician Scientist funded through a CIHRStroke Foundation of Canada. Maneesh Sud is Clinician–Scientist Pro- Post-Doctoral Fellowship, the Eliot Phillipson (South Asian Networkgram at the University of Toronto and a SANSAR Young Investigator Supporting Awareness and Research)–Burgundy Clinician Scholar AwardAward. George Thanassoulis is supported by a from the Fonds de recherche Québec — Santé. in coded Data sharing: The data set from this study is held securely prohibit ICES from form at ICES. While data-sharing agreements thoseto granted be may access available, publicly set data the making at available access, confidential for criteria prespecified meet who plan and underlying www.ices.on.ca/DAS. The full data set creation request, understandinganalytic code are available from the authors on templates or macrosthat the computer programs may rely on coding inaccessible or may that are unique to ICES and are therefore either require modification. Acknowledgement: (deceased May 30, 2018), who contributed to the idea, early work of this study and funding from the Foundation Grant (FDN 143313) from CIHR. Disclaimer: Care Long-Term and Health of Ministry Ontario the from grant annual (MOHLTC). The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should Parts of this material are based on data and information inferred. be compiled and provided by the Canadian Institute for Health Information (CIHI). The analyses, conclusions, opinions and statements expressed herein are those of the authors, and not necessarily those of CIHI. Accepted: Mar. 9, 2020 Correspondence to: Dennis Ko, [email protected] VOLUME 192 VOLUME | APRIL 27, 2020 | Maneesh Sud, Gynter CMAJ Dennis Ko and Todd Anderson made substantial contri- Ann Intern Med 2015;162:266-75. Jung KJ, Jang Y, Oh DJ, et al. The ACC/AHA 2013 Pooled Cohort Equations Equations Cohort Pooled 2013 ACC/AHA The al. et DJ, Oh Y, Jang KJ, Jung Prediction Model for atherosclerotic cardiovascular compared to a Korean Risk disease. Atherosclerosis 2015;242:367-75. MD, et al. Accuracy of the Atherosclerotic Car- Rana JS, Tabada GH, Solomon in a large contemporary, multiethnic population. J diovascular Risk Equation Am Coll Cardiol 2016;67:2118-30. Xia F, Ning J, Huang X. Empirical comparison of the Breslow Estimator and the Estimator and the Breslow of X. Empirical comparison J, Huang Xia F, Ning 2018;9. Functions. J Biom Biostat Estimator for Survival Kalbfleisch Prentice [Epub 2018 Feb.]. pii: 392. doi: 10.4172/2155-6180.1000392. guidelines: al. The new dyslipidemia Mancini GB, Genest J Jr, et Anderson TJ, Can J Cardiol 2015;31:605-12. what is the debate? of car- guidelines for prevention NR. Statins: new American Ridker PM, Cook Lancet 2013;382:1762-5. diovascular disease. insight into the cardiovascular risk calculator: the Cook NR, Ridker PM. Further and underascertainment in the Women’s roles of statins, revascularizations, 2014;174:1964-71. Health Study. JAMA Intern Med CJ, et al. An analysis of calibration and discrim- DeFilippis AP, Young R, Carrubba risk scores in a modern multiethnic ination among multiple cardiovascular cohort. Competing interests: George Thanassoulis reports personal fees from grants and personalAmgen, personal fees from Sanofi and Regeneron, and personal feesfees from Ionis, grants and personal fees from Servier, Jacob Udell has re- from HLS Therapeutics, outside the submitted work. Amgen, AstraZeneca, ceived consulting or speaker honoraria from and Sanofi, and researchBoehringer Ingelheim, Janssen, Merck, Novartis and Sanofi. No othergrants to his institution from AstraZeneca, Novartis competing interests were declared. This article has been peer reviewed. Affiliations: Schulich Heart Centre (Ko, Sud, Roifman, Wijeysundera), Sivaswamy, Sud, Kotrri,Sunnybrook Health Sciences Centre; ICES (Ko, Institute of HealthAzizi, Koh, Austin, Lee, Roifman, Udell, Wijeysundera); Lee, Roifman, Tu,Policy, Management and Evaluation (Ko, Azizi, Austin, Health NetworkUdell, Wijeysundera), University of Toronto; University (Thanassoulis), McGill(Lee, Tu), Toronto, Ont.; Department of Medicine Montréal, Que.;University; McGill University Health Centre (Thanassoulis), Family and CommunityNorth York General Hospital (Tu), Department of Research Institute Medicine, University of Toronto; Women’s College Alberta of Institute Cardiovascular Libin Ont.; Toronto, (Udell), University of Cal- (Anderson); Cumming School of Medicine (Anderson), gary, Alta. Contributors: butions to the conception and design of the work. Atul Sivaswamy, butions to the conception and design of the work. Atul Sivaswamy, Maneesh Sud, Gynter Kotrri, Paymon Azizi and Peter Austin contributed to the analysis and interpretation of data. Maria Koh contributed to the acquisition, analysis and interpretation of data. Douglas Lee, Idan Roifman, George Thanassoulis, Karen Tu, Jacob Udell and Harindra Wijeysundera contributed to the interpretation of data. Dennis Ko and Todd Anderson drafted the work. Atul Sivaswamy, Kotrri, Paymon Azizi, Maria Koh, Peter Austin, Douglas Lee, Idan Idan Lee, Douglas Austin, Peter Koh, Maria Azizi, Paymon Kotrri, Roifman, George Thanassoulis, Karen Tu, Jacob Udell and Harindra Wijeysundera revised the work critically for important intellectual con- tent. All of the authors gave final approval of the version to be published and agreed to be accountable for all aspects of the work. Funding: Early work on this study was funded by a Foundation Grant (FDN 143313) from the Canadian Institutes of Health Research (CIHR). Additional funding was provided by Foundation Grants (FDN 154333). Mid-Careera by are supported Lee Douglas and Austin Peter Ko, Dennis

22. 23. 17. 18. 19. 20. 21.