Supplement 1: Prescription Placed in Patient Chart 11- 12, Printed on Yellow Paper

Online Supplement

Supplement 1: Prescription placed in patient chart ’11-’12, printed on yellow paper

1 Supplement 2: Laminated reminder placed in patient chart ’12-‘13

2 Supplement 3: Pink sticker placed on clinic visit medication reconciliation form ’12-‘13

3 Supplement 4: Influenza vaccination reminder placed in public patient areas

4 Supplement 5 A. Provider survey ’12-‘13 1) Did you find the laminated reminder for flu shot included in the patients’ charts/clinic desk helpful in terms of prescribing the flu shot? 2) Did you find the laminated reminder for flu shot included in the patients’ charts/clinic desk helpful in terms of addressing whether the patient has been vaccinated against influenza in the current flu season? 3) Were you aware of the Best Practice Alerts regarding influenza vaccination in MiChart? 4) Did you find the Best Practice Alerts embedded in MiChart helpful in terms of prescribing the flu shot? 5) Did you find the Best Practice Alerts embedded in MiChart helpful in terms of addressing whether the patient has been vaccinated against influenza in the current flu season? 6) Do you believe that we should continue these intervention(s) at the next flu season, in order to prompt providers to address influenza vaccination?

Supplement 6 C: Logistic model for effect of the intervention The unit of analysis is each patient, and we posit a model for the probability of a previously unvaccinated patient being vaccinated after his/her first appointment of the flu season. Make the following definitions. Let V be the outcome indicating vaccination status. Let Yr be the year beginning each flu season (Yr = 2005,2006, … ,2012). Let Mo be a length-4 vector of indicators for the month of the patient’s first appointment of the flu season, with Mo = (1, 0, 0, 0)T denoting September, Mo = (0, 1, 0, 0)T denoting October, Mo = (0, 0, 1, 0)T denoting November, and Mo = (0,0,0,1)T denoting the remaining months of the flu season (December, January, February, and March). Let Age be a length-3 vector of indicators of 20-year age groups, beginning at age 17, with the youngest group being the reference category. So, an adult who is younger than 37 during the current flu season has Age = (0, 0, 0)T, an adult at least 37 but younger than 57 has Age = (1,0,0)T , etc. The oldest age group contains all ages over 77. Let H1N1 be an indicator variable for Yr = 2009, which was the H1N1 flu season. Finally, let Int indicate the presence or not of the intervention, i.e. Int =1 when an eligible patient’s first appointment was after October 3, 2011 in the ’11-’12 flu season or after September 24, 2012 in the ’12-’13 season, which are the start dates of the intervention (see Methods), and Int =0 for appointments before those dates as well as for all previous flu seasons. A logistic model relating the covariate pattern X = {Yr, Mo T, Age T, H1N1 , Int }T to the probability of vaccination is given by

The coefficient α assumes a constant yearly change in the log-odds of being vaccinated. The vector µ allows for the odds of vaccination to differ by month of an eligible patient’s first appointment of the season. The intercept in (1) is implicitly in the vector µ, because there is no reference category in Mo. The vector δ allows the odds of vaccination to change depending on age. The vector γ allows the odds of vaccination to differ in each month of the H1N1 season. The vector θ is of primary interest and determines whether there is any additional month-by-month change in the log-odds of being vaccinated during the intervention period of the ’11-’12 and ’12-’13 flu seasons. Importantly, because the intervention began on October 3 and September 24 for the ’11-’12 and ’12-’13 flu seasons, respectively, we parameterized the model accordingly, and θ may only differentially affect the probability of vaccination T with appointments that were after these dates. A hypothesis test of H0: θ = (0, 0, 0, 0) corresponds to a formal test for an effect of the intervention.

5 Fitting (1) to our data, we have, which suggests that the intervention did indeed increase the rate of vaccinations. The p-values for the first, second and fourth components of , corresponding respectively to patient’s first appointment being in September, October, or December/January/February/March, are all less than 0.0001, and the p-value for the third component, November, is 0.618. A joint test of θ = (0, 0, 0, 0) T is highly significant, with p<0.0001. Thus, we estimate that the log-odds of vaccination increased during the ’11-’12 and ’12-’13 flu seasons during the months of September, October, and December/January/February/March, in comparison to previous flu seasons, and the probability of being vaccinated in the most recent two flu seasons is larger than what would be expected if there were no effect of the intervention.

Supplement 7 D: Estimates and standard errors of the relative and absolute increase in vaccination rate Define , the model-based estimate of the probability an eligible patient with covariate pattern X being vaccinated. Also, define , where is the vector with the intervention indicator Int constrained to be zero. In other words, is the estimated probability of an eligible patient with covariate pattern being vaccinated, had the intervention not occurred. The quantities and are the primary ingredients for constructing Figure 3 of the manuscript. Let be the number of eligible patients with covariate pattern X. We estimate two

quantities. First, R is the relative change in the 2011 vaccination rate as a result of the intervention and 11 is be estimated by . The summation is over all possible values of the covariate pattern X, constrained to

the year 2011.Second, A is the absolute increase in the 2011 vaccination rate as a result of the 11 intervention and is estimated by. Similarly for 2012, we have and .

Standard errors can be estimated using the delta method and the multivariate central limit theorem. For , we estimate the confidence interval on the log-scale and then transform back to the natural scale. We have

The quantity is the observed number of vaccinations at covariate pattern , and is the number of unique covariate patterns. The expression for includes an estimate of an overdispersion parameter to account for variability in the data beyond what is expected from the binomial distribution.1 Using this approximation, a 95% confidence interval for is given by . The calculation is analogous for . For, , we have

6 Using this approximation, a 95% confidence interval for is given by . The calculation is analogous for .

REFERENCES 1. McCullagh P NJ. Generalized linear models. 2nd ed. Boca Raton: CRC press; 1989.