When effects cannot be estimated: redefining estimands to understand the effects of naloxone access laws

Kara E. Rudolph∗1, Catherine Gimbrone1, Ellicott C. Matthay2, Iván Díaz3, Corey S. Davis4, John R. Pamplin II5,6, Katherine Keyes1, and Magdalena Cerdá5

1Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, New York 2Center for Health and Community, School of Medicine, University of California, San Francisco 3Division of Biostatistics, Department of Population Health Sciences, Weill Cornell Medicine, New York, New York 4Network for Public Health Law, Los Angeles, California 5Center for Opioid Epidemiology and Policy, Department of Population Health, School of Medicine, New York University, New York, New York 6Center for Urban Science and Progress, New York University, New York, New York

Abstract

Background: All states in the US have enacted at least some naloxone access laws (NALs) in an effort to reduce opioid overdose lethality. Previous evaluations found NALs increased nalox- one dispensing but showed mixed results in terms of opioid overdose mortality. One reason for mixed results could be failure to address violations of the positivity assumption caused by the co-occurrence of NAL enactment with enactment of related laws, ultimately resulting in bias, increased variance, and low statistical power.

Methods: We reformulated the research question to alleviate some challenges related to law co-occurrence. Because NAL enactment was closely correlated with Good Samaritan Law (GSL) arXiv:2105.02757v1 [stat.AP] 6 May 2021 enactment, we bundled NAL with GSL, and estimated the hypothetical associations of enact- ing NAL/GSL up to 2 years earlier (an amount supported by the observed data) on naloxone dispensation and opioid overdose mortality.

Results: We estimated that such a shift in NAL/GSL duration would have been associated with increased naloxone dispensations (0.28 dispensations/100,000 people (95% CI: 0.18-0.38) in

∗Corresponding author: [email protected] 722 W. 168th St, NY, NY 10032 tel. +12123422926

1 2013 among early NAL/GSL enactors; 47.58 (95% CI: 28.40-66.76) in 2018 among late enactors). We estimated that such a shift would have been associated with increased opioid overdose mor- tality (1.88 deaths/100,000 people (95% CI: 1.03-2.72) in 2013; 2.06 (95% CI: 0.92-3.21) in 2018).

Conclusions: Consistent with prior research, increased duration of NAL/GSL enactment was associated with increased naloxone dispensing. Contrary to expectation, we did not find a protective association with opioid overdose morality, though residual bias due to unobserved confounding and interference likely remain.

2 Introduction

The opioid overdose crisis continues to be a public health threat, with overdose deaths increasing nine-fold since 1999.(22) In 2019, opioids, either alone or in combination with other substances, were responsible for nearly 50,000 deaths.(21, 54) Provisional data show that overdose-related deaths have accelerated since then, with more deaths recorded in the 12 month period ending May 2020 than in any other twelve month period on record.(15)

In response to this unprecedented increase in opioid-related harm, states have turned to policy change. One of the most widely adopted of these changes are laws designed to increase access to the medication naloxone, an opioid antagonist that quickly reverses opioid overdose if adminis- tered in time.(7) Naloxone access laws (NALs) allow naloxone to be prescribed and dispensed to people who use opioids, or to laypersons such as nonmedical first responders, bystanders, and fam- ily/friends. They can include provisions such as: 1) permission for “third-party prescriptions” that allow healthcare practitioners and pharmacists to prescribe and dispense naloxone to people who are not at direct risk of overdose, 2) removal of liability for providing or administering naloxone, 3) per- mission for a naloxone “standing order” prescription for persons at risk of overdose, 4) permission for pharmacists to dispense naloxone to individuals without a prescription, among other provisions.(8)

Prior research has found mixed evidence for the effects of NALs on their intended outcomes. While several studies have found that NALs are associated with increases in access to naloxone from both pharmacies and non-pharmacy sources,(18, 28, 43, 55) findings regarding whether laws are associ- ated with reduced overdose mortality vary somewhat based on the time period and specific NAL components studied.(1, 2, 10, 13, 30, 37, 42)

In addition to variation in timing and the specific laws under investigation, another rea- son for mixed findings could be challenges stemming from the co-occurrence of NAL with other related opioid laws. For example, overdose Good Samarian Laws (GSL), which provide some le- gal protections to those witnessing or experiencing an overdose to encourage help-seeking,(8, 44) are frequently enacted in conjunction with an NAL (in 31% of all states in the US) or within 2 years of NAL enactment (in 63% of states). When laws that may affect a common outcome are passed either close together in time or even at the same time, it is difficult or even impossible to estimate the effect of any individual law, disentangled from the others.(29) In an extreme example,

3 if two laws have only ever been implemented together, there exists no variation in Policy A that is independent from Policy B, meaning that it is not possible to identify the independent effect of

Policy A. Alternatively, one may mistakenly attribute the effects of Policy B to Policy A, if their covariation is not considered. In this case, between 48% and 78% of the place-time variation in NAL provisions is explained by other opioid-related state laws in the United States (considering years

2007-2017). This situation, albeit less extreme than 100% covariation, nonetheless poses a problem for estimating effects of NALs.

Formally, this challenge is due to violation of the positivity assumption that is required to identify most causal effects.(34) The positivity assumption means that there is a positive prob- ability of each level of exposure for every strata of observed confounder combinations.(23, 34) The assumption is violated if there is zero chance of having a particular exposure level given the observed covariates. For example, if a state had an NAL passed in an earlier year, that state would have zero chance of having the absence of an NAL in a subsequent year (as these laws have not been repealed).

The positivity assumption may also be “practically” violated in cases of very small, though nonzero, probabilities of a particular exposure level given covariates.(34) Because one would have a moderately high probability of correctly predicting whether an NAL is present in a state-year given the presence of other related laws, the positivity assumption is likely to be practically violated.

Taking into account other covariates in addition to the other related laws would exacerbate the practical positivity violations. Practical positivity violations are compounded in the small sample sizes common in policy analysis,(34) in multiple timepoint settings, as well as with most continuously valued exposures (e.g., if the exposure is the number of years a policy has been in place).(52)

When trying to evaluate the effect of a policy like an NAL that frequently co-occurs with other related policies, researchers typically choose between one of two options. 1) Not controlling for the other policies, meaning that the effect of the policy in question is conflated with the effect of the uncontrolled policies. Or, 2) controlling for the other policies, greatly reducing the independent variation in the primary policy. Depending on the degree of space-time correlation between the policies, the second option can result in: a) low statistical power, b) increased variance, sometimes even rendering the estimate no longer informative, and c) bias, which is a less widely acknowledged consequence.(34, 38)

4 With these challenges in mind, we adopt option 1) to address the frequent co-enactment of NAL with GSL, considering them together as a bundle, and adopt option 2) to address other, opioid-related laws, refining the research question to correspond to a hypothetical intervention that “shifts” the exposure distribution to weaken the positivity assumption.(11) We estimate the associations between longer duration of a state NAL/GSL and naloxone pharmacy dispensations, one of the proximal outcomes that NALs are designed to affect, and opioid overdose mortality rates, the longer-term outcome and ultimate target of such laws. Thus, our objective is not only to provide evidence of the extent to which NAL/GSLs may affect relevant opioid-related outcomes, but also to offer a proposed strategy for answering such policy questions more generally.

Methods

Data/Sample

We consider US county-level data 2007-2017, analyzed in two strata. We stratified by states that were early NAL/GSL-enactors (enacting any one of the NAL or GSL provisions considered from

2007 to 2012) and those that were late NAL/GSL-enactors (enacting any one of the NAL or GSL provisions considered in 2013 or later) (Table 1). We discuss the decision to combine NAL with GSL below. We stratify by early-/late-enactor status, because the drivers of the opioid epidemic changed meaningfully during this time period: the years 2007-2012 represents a period where deaths were driven largely by prescription opioids and heroin, whereas in years 2013-2017 deaths were driven by fentanyl.(6) We also stratified for estimation-related reasons, which we discuss further below.

Table 1: Early and late NAL/GSL enactors

Year of NAL/GSL State Analytic Strata with exposure period enactment 2007 NY Early: 2007-2012 2008 CA 2009 MD 2010 IL,WA 2012 CO,RI,MA,FL 2013 DC,NC,NJ,VT,OR,KY,VA,DE,OK Late: 2013-2017 2014 OH,UT,IN,WI,GA,ME,MI,TN,LA,MI,PA 2015 WV,NE,NH,SC,AL,ID,MS,HI,AK,ND,TX,NV 2016 IA,SD,AZ,MO 2017 MT,KS,WY

5 For the early-enactor strata, our data include: baseline covariates in 2007 (W ); exposure of number of years the law provisions of interest were in place 2007-2012 (A), using exact dates to calculate partial years of enactment; and outcomes of naloxone dispensations and opioid overdose deaths in 2013 (Y ). For the late-enactor strata, our data entailed: baseline covariates in 2013 (W ); number of years the law provisions of interest were in place 2013-2017 (A), again, including partial years of enactment; and outcomes of naloxone dispensations and opioid overdose deaths in 2018

(Y ).

We considered all US states and DC except for: Alaska, due to county boundary changes, and New Mexico and Connecticut, given their anomalous early enactment of NAL in 2001 and

2003, respectively. We also excluded 5.1% of counties with missing naloxone dispensations (the only variable with missingness) in the early-enactor strata and 12.6% in the late-enactor strata

(more detail below). Our final sample size was 409 early-enactor counties and 2,298 late-enactor counties.

Measures and their refinement

Policy exposure variable

The Prescription Drug Abuse Policy System(44) provided state data on effective dates of laws, including NAL provisions. While there are variations between states, most of these laws have one or more of the four provisions enumerated in the Introduction. These provisions were operationalized as the proportion of year that they were in effect.

Using the variable importance measure associated with the random forests algorithm,(3) we first assessed whether any of the NAL provisions were unimportant in predicting subsequent naloxone dispensing rates and consequently could be considered an irrelevant component of the exposure. Variables deemed “unimportant” are those that, when permuted, do not affect, or min- imally affect, the predictive value of the algorithm. Provisions 1-3, described in the Introduction, all contributed to predicting naloxone dispensing and were retained, meaning that their inclusion in a random forest-based model predicting naloxone dispensing rates improved predictive accuracy.

In contrast, inclusion of pharmacist prescriptive authority did not improve prediction of naloxone dispensing rates, so was not retained.

6 Figure 1: Absolute (range 0-1) correlations between presence of each of three Naloxone Access Law provisions and a Good Samaritan Law across relevant states, years 2007-2017.

NAL Provision(a) Correlations: Early-enactor Early NAL or GSL states Adopters NAL Provision(b) Correlations: Late-enactor Late NAL or GSL states. Adopters

1 1

GSL GSL

3rd Party 3rd Party Presc Presc

0.5 0.5 Absolute correlation Absolute correlation Protocol/ Protocol/ Stand Order Stand Order

Presc Presc 0 Immunity 0 Immunity GSL GSL Protocol/ Stand Order Protocol/ Stand Order 3rd Party Presc Presc Immunity 3rd Party Presc Presc Immunity

The co-occurrence of an NAL with other opioid or other substance policies that also affect the outcomes of interest would preclude our ability to identify any effect specifically attributable to an NAL due to their violation of the positivity assumption. We considered the following “covariate" policies: pain management clinic laws (PMCL), medical marijuana laws (MML), prescription drug monitoring program laws (PDMPL), and Good Samaritan Laws (GSL). We were also interested in laws related to opioid prescribing limits(9, 46) and recreational marijuana laws,(48) but were unable to incorporate them into the analysis due to lack of variation over the time periods considered. To assess the extent of law co-occurrence, we examined their correlation with the three NAL provisions of interest across the years included in this study (2007-2017). The absolute value of the correlations are shown in Figure 1. GSL passage appears highly correlated with NAL passage. Consequently, we redefine the exposure of interest as passage of any NAL provision 1)-3) or passage of a GSL, and operationalize it as the proportion of year that any of the 3 NAL provisions or GSL were in effect.

Naloxone pharmacy dispensations

Data for retail pharmacy naloxone dispensations came from IQVIA Xponent,(27) providing 87-94% of complete information for US counties 2007-2018. IQVIA masks dispensing data for counties with

7 less than three retail pharmacies in a given year. In order to determine if a county/year was indeed masked or simply had no naloxone dispensing, we cross-referenced our data with both county-level opioid dispensing data analyzed by the Centers for Disease Control and Prevention (CDC) and sourced from IQVIA Xponent(14) and county retail pharmacy information provided by the Census

Bureau’s County Business Patterns (CBP) data.(4) If, according to the CDC, a county/year had some opioid dispensing but no naloxone dispensing, then it was assigned 0 naloxone dispensations.

Similarly, if a county/year had greater than two or 0 retail pharmacies then it was assumed to have

0 naloxone dispensations.

Retail pharmacy naloxone dispensations included prescriptions of the generic naloxone,

Naloxone HCl, as well as two branded naloxone products, Narcan and Evzio. We calculated the yearly county naloxone dispensing rate by dividing total naloxone dispensations by the population aged 12 years or older, multiplied by 100,000.

Opioid overdose death rate

Opioid overdose mortality data came from the National Center for Health Statistics.(17) Opioid overdose deaths included all opioid overdose-related deaths where cause of death was coded as

ICD-10(16) codes X40-44, X60-64, X85, Y10-14, T40.0-T40.4, and T40.6. We calculated the yearly county opioid overdose death rate by dividing the total number of opioid overdose deaths by the population aged 12 years or older, multiplied by 100,000.

Covariates

Analyses included numerous covariates measured at baseline (2007 for the early-enactor states, 2013 for the late-enactor states). At the county-level, this included the proportion of families under the poverty threshold; proportion unemployed; median household income; proportion male; proportion white, black, Hispanic/Latino; population density; proportion in the following age categories: 0-

19, 20-24, 25-44, 45-64, 65+. These county-level demographic data came from GeoLytics.(26)

We included the outcome measured at baseline as a covariate as well. At the state-level, covariates included policies that 1) varied over the early-/late-enactor time periods considered, and 2) were not highly correlated with our exposure: PMCL, MML, and PDMPL (see Policy Exposure subsection above). Each was operationalized as the proportion of year the law was in effect. PMCL was

8 categorized by enactment of any pain management clinic law, using data from PDAPS.(47) MML was categorized as present if a state had a law allowing for medical marijuana home cultivation or contained an operational dispensary for medical marijuana, using data from PDAPS,(45) the

Marijuana Policy Project,(36) ProCon.org,(35) and Pacula et al. 2015.(32) PDMP was categorized by Horwitz et. al. 2020(25) and ranged from 0 - 2, summing proportion of year a state had a modern operational PDMP system and proportion of year a state had a law that required PDMP system querying prior to prescribing or dispensing.

Estimands

We defined our estimands to reflect our research questions while also minimizing positivity problems.

Problem: We could have chosen a more typical average treatment effect (ATE) estimand, such as:

1. the difference in overdose risk in 2015 had all states been early NAL/GSL versus late NAL/GSL

enactors, E(Y1 −Y0), using notation Ya to denote the counterfactual outcome had the exposure value been a (possibly contrary to fact); or

2. the difference in overdose risk in 2015 had all state-years had an NAL/GSL enacted versus if

none of them had one, E(Y1¯ − Y0¯), using notation Ya¯ to denote the counterfactual outcome had the exposure history over time been a¯.

To intuitively understand positivity problems stemming from formulation of the estimand in example

1), consider using prior opioid overdose mortality rate as the only covariate. States with very few opioid overdose deaths may be unlikely to pass NAL/GSL, resulting in practical positivity violations.

Adding covariates, as we do here, only exacerbates the issue. In addition, for the longitudinal estimand in example 2), to appropriately account for time-varying covariates affected by prior exposure, one would need to rely on the sequential exchangeability assumption and positivity such that there would be a nonzero and not too small probability of having an NAL/GSL enacted at each time point given past exposure history and covariate history.(24) However, given that NALs/GSLs have never been repealed, positivity problems would arise for states in years after enactment, because they would have no real probability of not having an NAL/GSL given their history of having an

NAL/GSL.

9 Proposal: Instead of explicitly comparing states as the above ATEs would entail, we instead reformulate the research question to correspond to the effect of shifting the distribution of the number of years since NAL/GSL enactment, corresponding to what is called a “modified treatment policy”.(11, 20) Under this reformulation, we ask: what would the effect on the opioid overdose mortality rate have been had late-enactor (early-enactor) states had their NAL/GSL for 2 years longer than their actual duration up to some maximum number of years (e.g., 5 years)? We can formulate it in terms of a hypothetical regime d(a) that depends on the natural exposure level a

(i.e., under no intervention).

In the case of the late-enactor analysis, A represents the exposure of years that a state had an NAL/GSL in place from March 19, 2013 to December 31, 2017, and ranged from 0.50 to 4.79 years. That is, the maximum number of years that a late-enactor state had an NAL/GSL was 4.79.

Then, one may define a hypothetical shift of the exposure distribution Aδ = d(A) that increases the years by δ ∈ {δ1, δ2} (Eq. 1), where d(a) is defined:

  a + δ2 if a ≤ 4.79 − δ2, where δ2 = 2,   d(a) = a + δ1 if 4.79 − δ2 < a ≤ 4.79 − δ1, where δ1 = 1, (1)    a if a > 4.79 − δ1.

For the subset of late-enactor states and outcome of opioid overdose mortality rate, this effect can be written: E(Yd) − E(Y ), where Y is the opioid overdose mortality rate and d denotes the hypothetical increase of NAL/GSL duration for up to 2 years longer than observed up to a maximum of the number of years in the time period. For the early-enactor states, we can apply the same approach, with the number of years that a state had an NAL/GSL ranging from 0.25 to 5.91 years.

To summarize, this reformulation moves away from an all or nothing (1/0) comparison and instead compares hypothetical outcomes under a shift of a continuous exposure. Consequently, it weakens the positivity assumption. One can intuitively understand this by considering a simpler intervention that would implement a shift of δ without bounds. In such a case, instead of relying

1 on the stability of g(a|w) for all observed (a, w) in estimation, the shift intervention would rely on g(a+δ|w) the more stable ratio g(a|w) .(31, 38)

10 We consider such an exposure shift estimand for two outcomes: the difference in 1) naloxone dispensing rates and 2) opioid overdose rates in 2018 (2013) had late-enactor (early-enactor) states had their NAL/GSL for 2 years longer than the true duration up to a maximum of 4.79 (5.91) years, controlling for baseline covariates in 2013 (2007).

The estimand E(Yd) − E(Y ) (where d is defined in Eq. 1) is identified under the random- ization assumption (Ya⊥⊥A|W ) as well as positivity for the exposure shift (if (a, w) ∈ supp{A, W }, then d(a, w) ∈ supp{A, W }), and the consistency rule E(Yd = y|W = w, A = d) = E(Y = y|W = w, A = d).(33).

Estimator

To estimate these quantities, we use a doubly robust substitution estimator of a modified treatment policy.(11) We incorporate an ensemble of machine learning algorithms into model fitting(51)— specifically, generalized linear models and gradient boosted machines. Interference(49) is likely present,(13) as characteristics (covariates, exposures) of one county may influence the potential naloxone dispensing rate and potential fatal opioid overdose rate in a nearby county, which may bias estimates. Therefore, in predicting the outcome for a given county, Yi, we use not only that county’s covariates, wi, and exposure, ai, but also a summary measure of each of the covariate values and exposure values of other counties in the same state, wj∈1,..,n\i and aj∈1,..,n\i. Standard errors and confidence intervals were calculated using the sample variance of the efficient influence curve, modified to reflect the clustering of counties within states.

Sensitivity Analysis

As a sensitivity analysis, we also considered a longitudinal alternative to the estimand in exam- ple 2), that appropriately accounts for feedback between time-varying covariates and time-varying exposures and compares the potential outcomes of naloxone dispensation rate and opioid overdose mortality rate had states delayed their NAL/GSL enactment for up to 2 years versus what was observed. We note that a longitudinal estimand more analogous to the estimand in the primary analysis would compare the potential outcomes of naloxone dispensation rate and opioid overdose mortality rate had states enacted their NAL/GSL up to 2 years earlier instead of up to 2 years later versus what was observed, but such an estimand would involve conditioning on future exposures,

11 meaning that one could not rely on the standard sequential exchangeability identification strategy.

We use a longitudinal version of the same estimator described above.(11) A more detailed discussion of this sensitivity analysis, including results, is in the Appendix.

R version 4.0.3 was used for all analyses. The estimator can be implemented using the lmtp R package.(53) Code to replicate the analyses is available: blindedforreview.

Results

We show the point estimates and associated 95% confidence intervals for each effect estimate in

Figure 2. As expected, we find that hypothetically increasing how long an NAL/GSL had been in effect was associated with greater expected naloxone dispensing rates in both the early period

(0.28 greater (95% CI: 0.18-0.38) naloxone dispensations per 100,000 people in 2013) and in the late period (47.58 greater (95% CI: 28.40-66.76) naloxone dispensations per 100,000 people in 2018).

Figure 3 demonstrates that the observed naloxone dispensing increased substantially over time.

This increase contributes to why the effect of increasing the amount of time having an NAL/GSL is stronger in the later period, because dispensations increased drastically between the early and late periods.

12 Figure 2: Predicted average additive associations of increasing the number of years that the state had a Naloxone Access Law or Good Samaritan Law by up to 2 years on A) naloxone dispensing rate (total county pharmacy dispensations per 100,000 population 12 years and older) and B) fatal opioid overdose rate (total county opioid overdose-related deaths per 100,000 population 12 years and older).

(b) Association with opioid overdose mortality (a) Association with naloxone dispensing rate. rate

3 60

●

● 2 40 ● Estimate Estimate

20 1

0 ● 0

early late early late

Figure 3: Naloxone dispensing rate over time

However, in contrast to expectation and the intended effects of the laws, we find that hypothetically increasing how long an NAL/GSL had been in effect was associated with greater expected opioid overdose mortality rates in both the early period (1.88 additional opioid overdose deaths per 100,000 people in 2013 (95% CI: 1.03-2.72)) and in the late period (2.06 greater (95%

CI: 0.92-3.21) opioid overdose mortality rate in 2018).

13 The longitudinal sensitivity analyses were mostly consistent with the primary analyses

(with the exception of the association with naloxone dispensations among late enactors), and are presented and discussed in the Appendix.

Discussion

States universally enacted NALs in response to the opioid epidemic in an effort to reduce the lethality of opioid overdose. Previous evaluations of the effectiveness of NALs found that they increased dispensing(18, 28, 43, 55) but demonstrated mixed results in terms of whether or not they increased,(13) decreased,(1, 2, 30, 37) or had no effect(5, 10) on overdose mortality rates.(42) One reason for these mixed results could be failure to adequately address violations of the positivity assumption caused by the co-occurrence of NAL enactment with the enactment of other, related laws, potentially resulting in bias, increased variance, and low statistical power. All but one of the previously mentioned studies controlled for co-occurring laws, including GSL, potentially resulting in positivity problems.(1, 2, 10, 13, 37) The remaining study chose not to control for co-occurring policies, conflating their potential effects with those of NALs.(30) The challenges posed by law- co-occurrences are not unique to NALs or even to substance use laws more generally, but are near-universal in policy evaluation.(29)

We proposed an approach of reformulating the research question of interest to alleviate some of these challenges that included: 1) bundling laws that cannot be disentangled together, and 2) considering an estimand of an effect of hypothetically shifting an exposure quantity by an amount supported by the observed data.(11) Because NAL enactment was closely correlated with

GSL enactment, we bundled NAL with GSL, and estimated the hypothetical effects of increasing duration of NAL/GSL enactment by up to 2 years on naloxone dispensation and opioid overdose mortality. We estimated that such a shift in NAL/GSL duration would have been associated with increased naloxone dispensations, consistent with prior studies.(18, 42, 43, 55) We also estimated that such a shift in NAL/GSL duration would have been associated with slightly increased opioid overdose rates, in contrast to the goal of NAL/GSL enactment and generally in contrast to findings of prior studies.(1, 2, 30, 37)

The unexpected associations with increased opioid overdose mortality may be due to un-

14 observed confounding, as it is unlikely we measured all of the relevant drivers of the opioid epidemic

(including factors that could affect both likelihood of enacting an NAL/GSL and overdose deaths).

For example, we did not control for any broader social/economic policies, even though these could be related to how long an NAL/GSL policy had been in place and related to naloxone dispensa- tions and opioid overdose rates. Future work could focus on better addressing endogeneity, perhaps by focusing on lethality of opioid overdose as the outcome (proportion of opioid overdoses that are fatal), attempting to control for opioid supply factors such as fentanyl penetration into the opioid market. However we acknowledge that data for nonfatal overdoses and for fentanyl supply can suffer from measurement error issues. For example, approximately 40% of people who experi- ence nonfatal overdoses never have contact with the emergency department or hospital (e.g., those that are reversed by a bystander),(12) and among those that do reach the emergency department,

ICD-based identification is insensitive.(39) Accurately capturing fentanyl supply is also plagued by measurement error,(19) for example due to lack of testing for fentanyl in overdose deaths in some jurisdictions.(41)

Another potential limitation is that the decision to combine NAL/GSL was based on the heatmaps in Figure 1 and was not the result of a formal statistical test or power calculation. To our knowledge, a formal approach for estimating the statistical power to detect some threshold of a law’s unique contribution to an outcome, controlling for co-occurring laws and other relevant confounding variables does not yet exist, and represents an area for future work.

Although we accounted for both interference (one county’s characteristics influencing a nearby county’s potential outcome) and spatial dependence among counties within a state, we treated each state as an independent unit. However, interference and spatial dependence likely exists across states,(13) and addressing this is another area for future work.

In summary, estimating the effects of policies is a practical, important endeavor, but one that entails multiple challenges, particularly related to policy co-occurrence and violations of the positivity assumption. These issues have been highlighted in some recent epidemiolgic(29, 40) and biostatistical(11) publications. Our goal was to add to this recent discussion, synthesizing and applying previous recommendations to gain traction in formulating a research question about the effect of NALs that would be identified based on observed data and then in estimating the expected effects of hypothetical shifts in NAL duration.

15 References

[1] Rahi Abouk, Rosalie Liccardo Pacula, and David Powell. Association between state laws facil-

itating pharmacy distribution of naloxone and risk of fatal overdose. JAMA internal medicine,

179(6):805–811, 2019.

[2] Janice Blanchard, Audrey J Weiss, Marguerite L Barrett, Kimberly W McDermott, and

Kevin C Heslin. State variation in opioid treatment policies and opioid-related hospital read-

missions. BMC health services research, 18(1):1–12, 2018.

[3] Leo Breiman. Random forests. Machine Learning, 45(1):5–32, 2001.

[4] United States Census Bureau. County business patterns (cbp).

[5] Guido Cataife, Jing Dong, and Corey S Davis. Regional and temporal effects of naloxone access

laws on opioid overdose mortality. Substance abuse, pages 1–10, 2019.

[6] Magdalena Cerdá, Noa Krawczyk, Leah Hamilton, Kara E Rudolph, Samuel R Friedman, and

Katherine M Keyes. A critical review of the social and behavioral contributions to the overdose

epidemic. Annual Review of Public Health, 42, 2020.

[7] James M Chamberlain and Bruce L Klein. A comprehensive review of naloxone for the emer-

gency physician. The American journal of emergency medicine, 12(6):650–660, 1994.

[8] Corey S Davis and Derek Carr. Legal changes to increase access to naloxone for opioid overdose

reversal in the united states. Drug and alcohol dependence, 157:112–120, 2015.

[9] Corey S Davis and Amy Judd Lieberman. Laws limiting prescribing and dispensing of opioids

in the united states, 1989-2019. Addiction, 2020.

[10] Jennifer L Doleac and Anita Mukherjee. The moral hazard of lifesaving innovations: naloxone

access, opioid abuse, and crime. Opioid Abuse, and Crime (March 31, 2019), 2019.

[11] Iván Díaz, Nicholas Williams, Katherine L Hoffman, and Edward J Schneck. Non-parametric

causal effects based on longitudinal modified treatment policies. arxiv, 2020.

16 [12] Matthew D Eisenberg, Brendan Saloner, Noa Krawczyk, Lindsey Ferris, Kristin E Schneider,

B Casey Lyons, and Jonathan P Weiner. Use of opioid overdose deaths reported in one state’s

criminal justice, hospital, and prescription databases to identify risk of opioid fatalities. JAMA

internal medicine, 179(7):980–982, 2019.

[13] Elham Erfanian, Daniel Grossman, and Alan R Collins. The impact of naloxone access laws

on opioid overdose deaths in the us. Review of Regional Studies, 49(1):45–72, 2019.

[14] Centers for Disease Control and Prevention. U.s. opioid dispensing rate maps.

[15] Centers for Disease Control and Prevention. Overdose deaths accelerating during covid-19,

2020.

[16] Centers for Disease Control and National Center for Health Statistics Prevention. International

classification of diseases, tenth revision, clinical modification (icd-10-cm).

[17] Centers for Disease Prevention and National Center for Health Statistics Control. Detailed

mortality – all county files 2006 - 2018 as compiled from data provided by the 57 vital statistics

jurisdictions through the vital statistics cooperative program.

[18] Alex K Gertner, Marisa Elena Domino, and Corey S Davis. Do naloxone access laws increase

outpatient naloxone prescriptions? evidence from medicaid. Drug and alcohol dependence,

190:37–41, 2018.

[19] Michael Gilbert and Nabarun Dasgupta. Silicon to syringe: Cryptomarkets and disruptive

innovation in opioid supply chains. International Journal of Drug Policy, 46:160–167, 2017.

[20] S Haneuse and A Rotnitzky. Estimation of the effect of interventions that modify the received

treatment. Statistics in Medicine, 32(30):5260–77, 2013.

[21] Holly Hedegaard, Arialdi M Miniño, and Margaret Warner. Drug overdose deaths in the united

states, 1999–2019.

[22] Holly Hedegaard, Margaret Warner, and Arialdi M Miniño. Drug overdose deaths in the united

states, 1999–2015. 2017.

[23] Miguel A Hernán and James M Robins. Causal inference: what if, 2020.

17 [24] Miguel A Hernán and James M Robins. Causal inference: what if, 2020.

[25] Jill R Horwitz, Corey Davis, Lynn McClelland, Rebecca Fordon, and Ellen Meara. The im-

portance of data source in prescription drug monitoring program research. Health Services

Research, 2020.

[26] Geolytics Inc. Geolytics estimates premium, 2018.

[27] IQVIA. Prescription information.

[28] Barrot H Lambdin, Corey S Davis, Eliza Wheeler, Stephen Tueller, and Alex H Kral. Naloxone

laws facilitate the establishment of overdose education and naloxone distribution programs in

the united states. Drug and alcohol dependence, 188:370–376, 2018.

[29] Ellicott C Matthay, Erin Hagan, Spruha Joshi, May Lynn Tan, David Vlahov, Nancy Adler,

and M Maria Glymour. The revolution will be hard to evaluate: How simultaneous change in

multiple policies affects policy-based health research. medRxiv, 2020.

[30] Chandler McClellan, Barrot H Lambdin, Mir M Ali, Ryan Mutter, Corey S Davis, Eliza

Wheeler, Michael Pemberton, and Alex H Kral. Opioid-overdose laws association with opi-

oid use and overdose mortality. Addictive behaviors, 86:90–95, 2018.

[31] Iván Díaz Muñoz and Mark van der Laan. Population intervention causal effects based on

stochastic interventions. Biometrics, 68(2):541–549, 2012.

[32] Rosalie L Pacula, David Powell, Paul Heaton, and Eric L Sevigny. Assessing the effects of

medical marijuana laws on marijuana use: the devil is in the details. Journal of Policy Analysis

and Management, 34(1):7–31, 2015.

[33] Judea Pearl. On the consistency rule in causal inference: axiom, definition, assumption, or

theorem? Epidemiology, 21(6):872–875, 2010.

[34] Maya L Petersen, Kristin E Porter, Susan Gruber, Yue Wang, and Mark J van der Laan.

Diagnosing and responding to violations in the positivity assumption. Statistical methods in

medical research, 21(1):31–54, 2012.

[35] ProCon.org. 33 legal medical marijuana states and dc: Laws, fees, and possession limits.

18 [36] Marijuana Policy Project. State policy.

[37] Daniel I Rees, Joseph J Sabia, Laura M Argys, Dhaval Dave, and Joshua Latshaw. With a little

help from my friends: The effects of good samaritan and naloxone access laws on opioid-related

deaths. The Journal of Law and Economics, 62(1):1–27, 2019.

[38] James Robins, Mariela Sued, Quanhong Lei-Gomez, and Andrea Rotnitzky. Comment: Per-

formance of double-robust estimators when" inverse probability" weights are highly variable.

Statistical Science, 22(4):544–559, 2007.

[39] Christopher Rowe, Eric Vittinghoff, Glenn-Milo Santos, Emily Behar, Caitlin Turner, and

Phillip O Coffin. Performance measures of diagnostic codes for detecting opioid overdose in

the emergency department. Academic emergency medicine, 24(4):475–483, 2017.

[40] Megan S Schuler, Beth Ann Griffin, Magdalena Cerdá, Emma E McGinty, and Elizabeth A Stu-

art. Methodological challenges and proposed solutions for evaluating opioid policy effectiveness.

Health Services and Outcomes Research Methodology, pages 1–21, 2020.

[41] Svetla Slavova, Chris Delcher, Jeannine M Buchanich, Terry L Bunn, Bruce A Goldberger,

and Julia F Costich. Methodological complexities in quantifying rates of fatal opioid-related

overdose. Current epidemiology reports, 6(2):263–274, 2019.

[42] Rosanna Smart, Bryce Pardo, and Corey S Davis. Systematic review of the emerging literature

on the effectiveness of naloxone access laws in the united states. Addiction, 116(1):6–17, 2021.

[43] Minji Sohn, Jeffery C Talbert, Zhengyan Huang, Michelle R Lofwall, and Patricia R Freeman.

Association of naloxone coprescription laws with naloxone prescription dispensing in the united

states. JAMA network open, 2(6):e196215–e196215, 2019.

[44] Prescription Drug Abuse Policy System.

[45] Prescription Drug Abuse Policy System. Medical marijuana dispensaries.

[46] Prescription Drug Abuse Policy System. Opioid prescribing guidelines for acute and emergency

care.

[47] Prescription Drug Abuse Policy System. Pain management clinic laws.

19 [48] Prescription Drug Abuse Policy System. Recreational marijuana laws.

[49] Eric J Tchetgen Tchetgen and Tyler J VanderWeele. On causal inference in the presence of

interference. Statistical methods in medical research, 21(1):55–75, 2012.

[50] Mark J van der Laan and Susan Gruber. Targeted minimum loss based estimation of causal

effects of multiple time point interventions. The international journal of biostatistics, 8(1),

2012.

[51] Mark J Van der Laan, Eric C Polley, and Alan E Hubbard. Super learner. Statistical Applica-

tions in Genetics and Molecular Biology, 6(1), 2007.

[52] Daniel Westreich and Stephen R Cole. Invited commentary: positivity in practice. American

journal of epidemiology, 171(6):674–677, 2010.

[53] Nicholas T Williams and Iván Díaz. lmtp: Non-parametric Causal Effects of Feasible Interven-

tions Based on Modified Treatment Policies, 2020. R package version 0.0.5.

[54] Nana Wilson, Mbabazi Kariisa, Puja Seth, Herschel Smith IV, and Nicole L Davis. Drug and

opioid-involved overdose deaths—united states, 2017–2018. Morbidity and Mortality Weekly

Report, 69(11):290–297, 2020.

[55] Jing Xu, Corey S Davis, Marisa Cruz, and Peter Lurie. State naloxone access laws are associated

with an increase in the number of naloxone prescriptions dispensed in retail pharmacies. Drug

and alcohol dependence, 189:37–41, 2018.

20 A Sensitivity Analysis

As in the primary analysis, we stratified by states that were early enactors and those that were late enactors. The longitudinal data can be represented: O = (W, A1,L2, ..., AT ,Y ), where W represents baseline covariates, including the baseline measure of the outcome; At represents 0/1 exposure of whether or not the county had an NAL/GSL enacted in year t; Lt represents time-varying covariates at time t, including the outcome measure at time t; and Y is the outcome at the final year, T . For the early enactors, t ∈ {2007 − 2013}. For the late enactors, t ∈ {2013 − 2018}.

We are interested in an estimand that compares the potential outcomes of naloxone dis- pensation rate and opioid overdose mortality rate had states delayed their NAL/GSL enactment for ¯ up to 2 years versus what was observed. This estimand can be written E(Yd¯) − E(Y ), where d is defined as:   at = at+1 = 0 if at−1 = 0 and at = 1and t + 1 ≤ T,   dt(at) = at = 0 if at−1 = 0 and at = 1and t = T, (2)    a otherwise, for all at ∈ 0, 1 and all t ∈ 2013-2018 (2007-2013) for late-enactors (early-enactors). We use a cross-fitted (5 folds), longitudinal targeted minimum loss-based estimator of a longitudinal additive treatment effect,(11, 50) including the following machine learning algorithms: intercept-only models, generalized linear models, lasso, multiple additive regression splines, and extreme gradient boosting.

Table A 2 gives the results. Among early enactors, delaying NAL/GSL enactment by up to

2 years during the period 2007-2013 would have had no effect on the pharmacy naloxone dispensation rate (estimate: -0.0005, 95% CI: -0.003, 0.002) and reduced the opioid overdose mortality rate by 0.109 deaths/100,000 people (95% CI: -0.139, -0.079) in 2013. Among late enactors, delaying

NAL/GSL enactment by up to 2 years during the period 2013-2017 would have increased the pharmacy naloxone dispensation rate by 36.4 dispensations/100,000 people (95% CI: 34.6, 38.3) and reduced the opioid overdose mortality rate by 0.556 deaths/100,000 people (95% CI: -0.612,

-0.501) in 2018.

21 Table A 2: Longitudinal effect of delaying NAL/GSL enactment by up to two years.

Outcome Early Enactors Late Enactors Estimate 95% CI Estimate 95% CI Naloxone dispensation rate -0.000 -0.003, 0.002 36.4 34.6, 38.3 Opioid overdose rate -0.109 -0.139, -0.079 -0.556 -0.612, -0.501

22