Identifying Cases of Spinal Cord Injury/Disorder in an Ontario Primary Care Electronic Medical Record Database

John Dean Shepherd

A thesis submitted in conformity with the requirements for the degree of Master of Science Rehabilitation Sciences Institute University of Toronto

John Shepherd

Master of Science

Rehabilitation Sciences Institute University of Toronto

2020 Abstract Currently, little health system information exists on the longitudinal course of spinal cord injury

(SCI). The increasing availability of information from electronic medical records (EMR) offers an opportunity to fill this gap. An initial keyword search of all eligible records in a database of primary care EMR in Ontario, Canada, was conducted to identify possible cases of SCI. A random sample of possible cases was selected (n=803) and reviewed using a structured chart review process. 126 validated cases of SCI were identified and this reference standard cohort was used to develop and test potential algorithms. The optimal algorithm uses a combination of free-text keywords and ICD-9 codes, with a sensitivity of 70.6%, a positive predictive value of

89.9%, and an F-score of 79.1%. This cohort can be linked with administrative data to study longitudinally the health status and health care utilization of this population.

Acknowledgments

The analysis documented in this thesis was the work of many people and I am grateful to all those who gave me support and guidance, and from whom I have learned so much.

First and foremost, I owe a tremendous debt of gratitude to my supervisor, Dr. Susan Jaglal. She has been unfailing in her support at all times and without peer as a mentor. She has always encouraged me to follow my research interests and has equipped me with the tools to do so effectively.

My committee members, Drs. Cathy Craven, Rahim Moineddin, and Karen Tu, have been extremely generous with their time and guidance, and provided many insightful suggestions and comments. I was fortunate to be able to build on the strong foundation of their prior work and the methods they shared with me.

My work at ICES was supported by a very welcoming and helpful team including Jin Park, Bogdan Pinzaru, and Dinah Thorpe. I am particularly grateful to Senior Analyst Jacqueline Young, who skillfully and patiently guided me through the analytical process, and whose work is evident particularly in the results section of this thesis.

I am also fortunate to have extremely supportive labmates and colleagues; I want to recognize in particular Drs. Sonya Allin, Leslie Carlin, and Sarah Munce, who each gave very generously of their time and expertise to provide feedback and suggestions on my drafts and to encourage me through the writing process.

Finally, I want to thank my family: my parents Glen and Eleanor, my sister Elizabeth and her husband Johan, and my nieces Sanna and Alma; they have been, since long before the beginning of this project right through to the end, a source of encouragement, support, and joy.

iii

Table of Contents

Acknowledgments ...... iii

Table of Contents ...... iv

List of Tables ...... vii

List of Figures ...... ix

List of Appendices ...... x

Introduction ...... 1

Background ...... 3

2.1 Spinal Cord Injury as a Diagnostic Entity ...... 3

2.1.1 Overview ...... 3

2.1.2 Definition of SCI ...... 4

2.1.3 Traumatic and Non-traumatic SCI ...... 6

2.1.4 Classification of SCI ...... 8

2.2 Frequency of Spinal Cord Injury ...... 8

2.2.1 Recent Studies of TSCI Incidence ...... 10

2.2.2 Prevalence of SCI ...... 10

2.2.3 Canadian Estimates of SCI Incidence and Prevalence ...... 12

2.3 Longitudinal Patterns of Chronic SCI ...... 13

2.3.1 Individual Burden and Healthcare System Impact of Chronic SCI ...... 13

2.3.2 Longitudinal Datasets in Chronic SCI ...... 14

2.3.3 Administrative Data Studies of SCI ...... 16

2.4 Using EMR Data for Research ...... 17

2.4.1 Primary Care EMR Databases ...... 17

2.4.2 Issues with Case Finding in EMRs ...... 18

2.4.3 Problem List Use and Reliability ...... 19

2.4.4 Case Finding in EMRALD ...... 20

2.4.5 Case Finding in SCI ...... 21

2.4.6 Case Finding Criteria in NTSCI ...... 22

2.5 Thesis Objectives ...... 22

Methods ...... 24

3.1 Overview ...... 24

3.1.1 Three-stage Case Identification/Algorithm Development Process ...... 24

3.1.2 EMRALD Overview ...... 25

3.1.3 EMRALD Composition and Representativeness ...... 25

3.1.4 Overview of EMRALD Charts ...... 26

3.1.5 ICES Policies and Working Arrangements ...... 27

3.1.6 Inclusion and Exclusion Criteria ...... 28

3.2 Stage 1—Keyword Search Strategy Development ...... 28

3.2.1 Preliminary Chart Review: Round 1 ...... 29

3.2.2 Preliminary Chart Review: Round 2 ...... 29

3.2.3 Preliminary Chart Review: Round 3 ...... 30

3.2.4 Addition of NTSCI-Related Keywords from Literature ...... 31

3.3 Stage 2—Chart Review and Case Identification ...... 33

3.3.1 Considerations for Identification of Chronic SCI Cases ...... 33

3.3.2 Case Definition of Chronic SCI ...... 33

3.3.3 Chart Entries Used for Case Identification ...... 35

3.3.4 Hierarchy of Evidence ...... 35

3.3.5 Developing a Reference Standard: Chart Review Process ...... 37

3.4 Stage 3—Algorithm Development ...... 40

3.4.1 Keyword Analysis and Keyword Lists ...... 41

3.4.2 Other Algorithm Components Considered ...... 43

3.4.3 Evaluating Algorithms ...... 44

Results ...... 46

4.1 Stage 1—Keyword Search Strategy Development ...... 46

4.1.1 Round 1 ...... 46

4.1.2 Round 2 ...... 47

4.1.3 Round 3 ...... 49

4.2 Stage 2—Chart Review and Case Identification ...... 50

4.2.1 Overview ...... 50

4.2.2 Chart Review Ratings by Round ...... 52

4.2.3 Demographic Characteristics ...... 53

4.2.4 EMR Documentation ...... 55

4.2.5 Primary Care Utilization ...... 55

4.3 Stage 3—Algorithm Development ...... 56

4.3.1 Keyword Analysis ...... 56

4.3.2 Algorithm Component Evaluation ...... 58

4.3.3 Case Identification Algorithm Evaluation ...... 60

Discussion ...... 66

5.1 Statement of Principal Findings ...... 66

5.1.1 Summary ...... 66

5.1.2 Similarities and Differences with Respect to Other Studies ...... 66

5.2 Strengths and Limitations ...... 72

5.3 Meaning and Implications ...... 74

5.4 Future Research Directions ...... 75

References ...... 76

Appendices ...... 85

List of Tables

Table 1 Comparison/evolution of definitions of TSCI in literature

Table 2 Studies of TSCI prevalence

Table 3 Longitudinal SCI datasets

Table 4 Case-finding studies using EMRALD

Table 5 EMRALD chart entry types

Table 6 Conclusions from preliminary chart review round 3

Table 7 Case definition: criteria for chronic TSCI and NTSCI case identification

Table 8 EMRALD chart entries used for case identification

Table 9 Chart review process steps

Table 10 CPP keyword lists

Table 11 Summary of preliminary chart review round 1

Table 12 Summary of preliminary chart review round 2

Table 13 Summary of preliminary chart review round 3

Table 14 Chart review process summary: ratings by process round

Table 15 Demographic characteristics of RSC compared with EMRALD

Table 16 Demographic characteristics of RSC: TSCI and NTSCI cases

Table 17 EMR documentation of RSC compared with EMRALD

Table 18 CPP keyword occurrences

Table 19 CONS/PN keyword occurrences

Table 20 CPP algorithm component performance

Table 21 CONS/PN algorithm component performance

Table 22 Medication and ICD-9 code algorithm component performance

Table 23 Algorithms with optimal F-score

Table 24 Algorithms with optimal PPV

vii

Table 25 Algorithms with optimal sensitivity

Table 26 Two-stage case finding studies using EMRALD

Table 27 Studies including TSCI and NTSCI cohorts

viii

List of Figures

Figure 1 Comparison of TSCI definitions

Figure 2 Descriptors for classifying SCI

Figure 3 Estimates of TSCI prevalence in recent review articles

Figure 4 Methods overview: key process steps

Figure 5 Final keyword search strategy

Figure 6 Hierarchy of evidence

Figure 7 Chart review process

Figure 8 CONS/PN keyword list

Figure 9 Flowchart for SCI case identification and algorithm development

Figure 10 Chart review ratings by round

Figure 11 Primary care utilization: encounters per year

Figure 12 Algorithms ranked by sensitivity

Figure 13 Precision-recall curve for case identification algorithms

Figure 14 Results overview: results by process step

List of Appendices

Appendix A NTSCI chart abstraction case-finding criteria (From Ho (2017))

Appendix B Final keyword search strategy: versions 1-3

Appendix C Charts rated non-cases

Appendix D Case-finding algorithm components

Appendix E Complete list of algorithms considered

Appendix F ICES Dataset Creation Plan

Chapter 1 Introduction Introduction

Spinal cord injury (SCI) is thought to be characterized by relatively small incidence and prevalence but long duration and many secondary health conditions, leading to high healthcare utilization and cost (Dryden et al., 2004). To date, most research has looked at the early/inpatient management of SCI, with the few large longitudinal datasets and registries focusing on this early (acute) portion of the care continuum (Dvorak et al., 2017). The years of life following discharge from inpatient rehabilitation, during which these patients make frequent use of the healthcare system, remain poorly described.

Community-based and primary care management of SCI are not extensively studied in the literature, which is problematic given the significance of long-term health care utilization by people with SCI in terms of both their health outcomes and the cost burden on the healthcare system. This gap has recently been addressed by analysis of administrative data, looking at hospital discharges, emergency department visits and overall physician billings for people living in the community with SCI (S J T Guilcher et al., 2010; Sara J. T. Guilcher, 2009; S B Jaglal et al., 2009; Munce et al., 2013). What such studies have yet to provide is a detailed description of the primary care environment (diagnoses, treatments/medications, lab diagnostics, etc.). To do this, one first needs to identify cases of SCI in primary care.

As primary care practices adopt electronic medical records (EMRs), data from these records is being aggregated and made available for research, particularly for the surveillance of chronic condition management (Birtwhistle & Williamson, 2015). To do this, it is necessary to reliably identify cases of the chronic condition of interest, ideally flagging every individual with the condition (no false negatives) without falsely flagging individuals without the condition (no false positives). Methods have recently been demonstrated to do this, by developing and validating case-finding algorithms for various chronic conditions (McBrien et al., 2018). This study will use a similar method to identify cases of SCI in primary care. The objectives of this thesis are to:

1 2

Primary objectives:

• Develop a method for identifying cases of SCI in a primary care EMR database • Establish a reference standard cohort of validated cases of SCI • Develop and evaluate case finding algorithms for use within the EMR database Secondary objectives:

• Develop a comprehensive case definition of SCI • Calculate the prevalence of SCI in a primary care database • Describe the characteristics of the reference standard cohort of validated SCI cases

Chapter 2 Background Background 2.1 Spinal Cord Injury as a Diagnostic Entity

2.1.1 Overview

Spinal cord injury is damage to the spinal cord resulting from traumatic or non-traumatic causes; vertebral injury in the absence of cord/cauda equina damage is not considered SCI (Chhabra, 2015). Traumatic SCI (TSCI) is defined as any injury to the spinal cord that is caused by trauma or damage resulting from an external force (Bickenbach, 2013). Non-traumatic SCI (NTSCI) is less well-defined in the medical literature, and there is no universally accepted term to designate this population (Peter W New & Delafosse, 2012), but the International Spinal Cord Injury Society (ISCoS) has agreed to use the term NTSCI with two major subgroups according to etiology: congenital/genetic (including spina bifida) and acquired (including degeneration, infection, cancer, etc). (P. W. New & Marshall, 2014a).

SCI can result in a range of neurological impairments depending on the location and nature of the damage to the spinal cord, potentially including impairments to motor, sensory, and autonomic function. Generally, lesions that are higher (more rostral) result in greater impairment; functional impairment limited to the lower extremity is called “paraplegia,” and impairment involving the upper limbs is called “tetraplegia.” Moreover, lesions that affect the entire cord (such as a total transection) are called “complete” and result in greater impairment than lesions that affect only a portion of the cord (called “incomplete”). The International Standards for the Neurological Classification of Spinal Cord Injury (ISNCSCI) define a complete injury as one where there is no motor or sensory function in the lowest sacral segments (S4-5); whereas an incomplete injury is one where some function is preserved at this level (Kirshblum et al., 2011). Incomplete lesions can present patterns of impairment that are variable and more limited than in the case of a complete lesion at the same level. Common types of lesions and the patterns of impairment they result in have been classified in a number of clinical syndromes (Anterior Cord syndrome, Brown-Sequard, Cauda Equina syndrome, Central Cord syndrome, Posterior Cord syndrome) (Chhabra, 2015); however, each SCI is unique in terms of the precise pattern of

3 4 impairments and the degree of functional recovery over time. Because of its potentially extensive impact on physical function, health and self-care, and social participation, SCI engages all aspects of a patient’s life.

Prior to the advent of modern, multidisciplinary rehabilitation medicine in the first half of the 20th century, the prognosis for those experiencing SCI was poor and life expectancy following injury was typically a few years at most (Dick, 1969). In subsequent decades, outcomes have improved and people living with SCI have demonstrated the ability to participate actively in society and engage in many forms of occupation; nonetheless, life expectancy is still meaningfully diminished by SCI and persons living with chronic SCI contend with a range of secondary medical complications in addition to their neurological impairment (M J DeVivo, 2007).

2.1.2 Definition of SCI

A case of SCI involves damage to the spinal cord and resulting functional impairment. In an early and influential definition by Kraus (1975), SCI is an “Acute, traumatic lesion of the spinal cord, including trauma to the nerve roots which result[s] in varying degrees of motor and/or sensory deficit or paralysis.” The diagnosis therefore requires two elements: the lesion to the spinal cord and some form of motor and/or sensory deficit and/or autonomic impairment.

There is, however, no consensus case definition of SCI. A survey of the literature shows that the definition has evolved over time and exhibits several key points of divergence between various studies with respect to duration and type of impairment and injury site (see Table 1).

Table 1. Comparison/Evolution of Definitions of TSCI in Literature Reference Definition Kraus (1975) Acute, traumatic lesion of the spinal cord, including trauma to the nerve roots which result[s] in varying degrees of motor and/or sensory deficit or paralysis. Griffin (1985) Definition from Kraus (1975); injuries to the spinal cord extending from the foramen magnum down to and including the cauda equina and complete nerve root avulsions were also included. Persons who had a spinal cord injury but had no residual neurologic deficit or only reflex changes were excluded. Acton (1993) Acute traumatic lesion of the spinal cord, excluding lesions resulting from chronic degenerative disease, which often results in a loss of motor or sensory function. Prevalent cases are included only if they exhibit any three of the four following functional characteristics: lack of normal motor control, lack of normal sensation, lack of normal bowel function, or lack of normal bladder function.

Thurman Acute, traumatic lesion of neural elements in the spinal canal (including spinal (1994) cord and cauda equina) resulting in any degree of sensory deficit, motor deficit, or neuropathic deficit of bowel or bladder function, either transient or persistent. CDC/WHO An occurrence of an acute traumatic lesion of neural elements in the spinal canal, (1995) resulting in temporary or permanent sensory deficit, motor deficit, or autonomic dysfunction. Johnson Injury caused by trauma that results in some degree of spinal cord-related (1997) motor, sensory, and bowel and/or bladder impairment, which can be temporary or permanent. Hagen (2008) Definition from Kraus (1975); although injuries of cauda equina were included, the definition excluded isolated injuries of other nerve roots. Transient paresis or impermanent deficits lasting less than 1 week were not included. ISNCSCI/ Impairment or loss of motor and/or sensory function in the spinal cord due to Kirshblum damage of neural elements within the spinal canal. Includes cauda equina and (2011) conus medullaris injuries, but not brachial plexus or lumbosacral plexus lesions, nor injury to peripheral nerves outside the neural canal.

The definitions found in these 8 studies show three key points of divergence and each needs to be considered when operationalizing a definition of chronic SCI for the purpose of case identification:

• The first has to do with the duration of impairment. Some definitions exclude transient impairment by stipulating that the impairment has to be present for a minimum period, from 24 hours to one week. Others have no minimum duration. This is because there is often some level of neurological and functional recovery after a spinal cord injury that may happen more or less rapidly. • The second divergence concerns types of impairment: motor, sensory, autonomic, bowel, and bladder. Some definitions require more than one of these five possible types. Others state that any type of impairment is sufficient. • The third point of divergence concerns injury site. Some definitions explicitly include injuries to the cauda equina and nerve roots. One definition excludes injuries to the nerve roots other than cauda equina. Other definitions leave it unspecified.

Although comparison of these definitions of SCI is instructive, it is complicated by the fact that not every definition is explicit with respect to every characteristic. However, certain characteristics can be inferred to be included in some definitions even though they are not

6 mentioned explicitly. Using this analysis, the inclusion/exclusion of various characteristics in the case definitions of SCI used in the 8 studies from Table 1 is represented visually in Figure 1.

Duration of Impairment Type of Impairment Site of Lesion Temporary >24h, Spinal Cauda Nerve >=1 wk Motor Sensory Bladder Bowel (<=24h) <1wk cord equina roots Kraus (1975) Griffin (1985) Acton (1993) 3 of 4 Thurman (1994) CDC/WHO (1995) Johnson (1997) Hagen (2008) ISNCSCI (2011)

Legend: Explicitly included Implicitly included Excluded

Figure 1: Comparison of TSCI definitions

2.1.3 Traumatic and Non-traumatic SCI

Historically, SCI as a diagnostic entity only included traumatic injuries; until fairly recently, spinal cord impairment of a nontraumatic origin was typically not categorized as a form of spinal cord injury. “Spinal cord injury,” and “traumatic spinal cord injury” were, practically, co- extensive (as reflected in the titles of many journal articles which refer to “spinal cord injury” and where the population in question only includes traumatic injuries). NTSCI was fragmented into a variety of etiology-related diagnoses (degeneration, inflammation, cancer, spina bifida, etc).

Over time, there has been an evolution with respect to the inclusion of spinal cord damage from non-traumatic causes in the overall definition of SCI. Early studies (Acton et al., 1993; Griffin, O’Fallon, Opitz, & Kurland, 1985; Johnson et al., 1997; Kraus, Franti, Riggins, Richards, & Borhani, 1975; Thurman, Burnett, Jeppson, Beaudoin, & Sniezek, 1994) define all SCI as injury caused by trauma, either implicitly or in some cases explicitly ruling out non-traumatic causes. In more recent studies, researchers have sought to characterize non-traumatic SCI (NTSCI) as a second subtype of SCI in which damage is caused by something other than an external force, such as a degenerative or disease process, infection, or congenital condition (Van Den Berg,

7 Castellote, Mahillo-Fernandez, & De Pedro-Cuesta, 2012). As with traumatic SCI (TSCI), there are divergences between definitions of NTSCI used in different studies: Van Den Berg et al (2012) include spina bifida and multiple sclerosis whereas Post et al (2011) exclude them (Post et al., 2011).

The importance of this population has come to be recognized due to the admission of non- traumatic injuries to specialized TSCI centres in increasing numbers; NTSCI now constitutes the majority of cases in many specialized spinal units (P. W. New, Simmonds, & Stevermuer, 2011), and the diagnostic entity of SCI has typically, for the purpose of research, come to include both TSCI and NTSCI; in some cases, the term spinal cord injury or dysfunction (SCI/D) is used to indicate the inclusion of NTSCI. Until recently, there was no authoritative classification of the etiologies of NTSCI (P. W. New & Marshall, 2014b), and medical terminologies such as the International Classification of Disease (ICD) do not facilitate categorization of NTSCI, in that they reflect the fragmentation of the diagnostic entity into a large number of etiologies; only recently has work begun to permit identification of NTSCI using ICD-10 codes (Ho et al., 2017).

Notwithstanding the difficulty, examining the epidemiology of NTSCI is important because the population is growing and has very different characteristics from the TSCI population, with important consequences for the provision of healthcare. Overall, the NTSCI population is older than the TSCI population, with a greater proportion of incomplete injuries and a greater number of co-morbidities (P. W. New, Farry, Baxter, & Noonan, 2013). In an Australian study that directly compared TSCI and NTSCI in the same population, patients with NTSCI were older, less likely to be male, and less disabled on admission (P. W. New et al., 2011).

Despite these demographic differences, however, both groups are similar with respect to rehabilitation outcomes (Kennedy & Chessell, 2013) and healthcare utilization (S J T Guilcher et al., 2010), which suggests that it is appropriate to consider them together from the standpoint of both clinical care and research. A better appreciation of the particularities of presentations within NTSCI (e.g. infection-related injuries are more likely to be complete than degenerative ones) can help with care planning, particularly in rehabilitation (Kennedy & Hasson, 2016). Finally, there is a need for research on survivorship in NTSCI, as this is essential to calculate prevalence (P. W. New et al., 2013).

8 2.1.4 Classification of SCI

A further source of complexity when considering SCI as a diagnostic entity involves the multiple ways that SCI can be classified (Chhabra, 2015) using different descriptors depending on the origin, type, and functional presentation of the injury (see Figure 2). When SCI is mentioned in clinical narrative such as in EMRs, only some of these descriptors may be mentioned.

Descriptors for classifying SCI • Cause/etiology o trauma/non-trauma, with subclassifications of each o e.g. MVA, sports (trauma), spina bifida, cervical myelopathy (non-trauma) • Syndrome o specific types of lesions, characterized by damage to a particular portion of the spinal cord and an associated functional presentation o e.g. central cord syndrome, Brown-Sequard • Neurological Level or Bony Level o can denote anatomical level of lesion and/or functional level of neurological impairment o functional level can differ e.g. between motor and sensory function, and/or between sides of the body o e.g. C5, T11 • Completeness o complete or incomplete: determined by extent of lesion and degree of functional impairment o completeness can differ between motor and sensory function, and/or between sides of the body • ASIA Impairment Scale Category o characterizes severity (completeness) of SCI using a standardized, detailed neurological examination of motor and sensory impairment o injuries are graded A-E (A: most severe, E: least severe)

Figure 2: Descriptors for classifying SCI

2.2 Frequency of Spinal Cord Injury

Basic epidemiological data concerning SCI are limited. With the exception of one estimate of global incidence of TSCI (Fitzharris, Cripps, & Lee, 2014), there are no reliable point estimates of global prevalence and incidence, and national-level statistics for both TSCI and non-traumatic SCI (NTSCI) are only available in two jurisdictions (Canada and Australia). These parameters have been estimated in studies in a small number of high-income countries, using several

9 different methods, and with a wide range of results (Furlan, Sakakibara, Miller, & Krassioukov, 2013).

A systematic review and meta-analysis of epidemiological studies of SCI is presented in the 2013 World Health Organization (WHO) report International Perspectives on Spinal Cord Injury (Bickenbach, 2013). Estimates for TSCI and NTSCI are presented separately, and data concerning NTSCI are more limited and less reliable. Six studies of TSCI reported between 1997 and 2010 show incidence ranging from 13 per million to 53 per million and prevalence ranging from 28 per 100,000 to 130 per 100,000. Two studies of NTSCI, both reported in 2010, show incidence ranging from 26 per million to 68 per million and prevalence ranging from 37 per 100,000 to 123 per 100,000.

A slightly more recent global survey of the prevalence and incidence of TSCI found a range of estimates from 8 per million to 83 per million for incidence and 25 per 100,000 to 91 per 100,000 for prevalence (Singh, Tetreault, Kalsi-Ryan, Nouri, & Fehlings, 2014). Lee et al (Lee, Cripps, Fitzharris, & Wing, 2014) report a range of prevalence estimates from 24 per 100,000 to 419 per 100,000.

Figure 3: Estimates of TSCI prevalence in recent review articles

Knowledge of the epidemiology of SCI varies globally and is minimal in many jurisdictions; in a classification of WHO zones based on data sources (Cripps et al., 2011), there was insufficient information to present estimates in 11 of 14 zones. The only global estimate of incidence available in the literature uses population modelling techniques, deriving the relationship between population at risk and incidence by regression on the data from 31 studies in 17 jurisdictions, then extrapolating to other jurisdictions by population (Fitzharris et al., 2014). This

10 method produced an estimate of 23 per million. Fitzharris et al (2014) note that there is insufficient information for a similar model and global estimate of NTSCI incidence. The measurement of SCI incidence is complicated by differences in measuring pre-hospital mortality, particularly in cases of TSCI.

2.2.1 Recent Studies of TSCI Incidence

Recent studies of TSCI in Europe (published since the WHO report) provide incidence estimates from 9.3 per million to 25.1 per million. A hospital-based study in Denmark found an incidence of 10.2 per million in the period 1990-2012 (Bjørnshave Noe, Mikkelsen, Hansen, Thygesen, & Hagen, 2015). A retrospective cohort study in Scotland found a mean incidence of 15.9 per million during the period 1994-2013 (McCaughey et al., 2016). An observational study with prospective and retrospective monitoring in Galicia (Spain) found a crude mean incidence of 21.7 per million during the period 1995-2014 (Montoto-Marqués et al., 2017). A prospective observational study in the Canary Islands (Spain) found a crude mean incidence of 9.3 per million during the period 2001-2015 (Bárbara-Bataller, Méndez-Suárez, Alemán-Sánchez, Sánchez-Enríquez, & Sosa-Henríquez, 2018). A study in Austria using national-level administrative data (ICD-10 codes in hospital discharges and deaths) found a crude incidence of 17 per million during the period 2002-2012 (Majdan, Brazinova, & Mauritz, 2016). A hospital- based study in Ireland found a crude mean incidence of 12.6 per million between 2010 and 2015 (Smith et al., 2018). Finally, a prospective study in 2 of 3 hospital districts in Finland found an incidence of 25.1 per million over a 1-year period in 2012-2013 (Koskinen et al., 2014). Most of these studies are linked to hospital admissions and, as such, do not include patients who died before hospital admission or whose injury or impairment was sufficiently mild that they did not require inpatient admission.

In contrast with the European estimates, a recent US study using a very large inpatient database (the Nationwide Inpatient Sample) reported an incidence of TSCI of 53 per million in 1993 and 54 per million in 2012 (Jain et al., 2015).

2.2.2 Prevalence of SCI

Overall, the incidence of SCI is much better studied than prevalence; in a systematic review Furlan et al (2013) cited 64 studies that reported incidence and only 13 that reported prevalence

11 (Furlan et al., 2013). Similarly, the epidemiology of TSCI is much better studied than NTSCI (Thietje & Hirschfeld, 2017). Accordingly, estimates of incidence and of TSCI can be considered relatively more reliable whereas estimates of prevalence and of NTSCI should be handled with greater caution.

An exhaustive search of the literature for studies of TSCI prevalence yielded 21 studies in different countries, using three different methods to measure prevalence (see Table 2):

Table 2. Studies of TSCI prevalence Reference Country Prev/100,000 As of i. Hospital records/Register-based Minaire (1978) France 25.0 1975 Levi (1995) Sweden 22.7 1994 Dahlberg (2005) Finland 28.0 1999 Hagen (2010) Norway 36.5 2002 BC Admin Data (2010) Canada-BC 67.3 2010 Knutsdottir (2012) Iceland 52.6 2009 Mean records/register-based 38.7 ii. Model-based Kurtzke (1975) US 50.0 1971 DeVivo (1980) US 90.6 1979 Ergas (1985) US 100.9 1984 Lasfargues (1995) US 78.6 1994 O'Connor (2005) Australia 68.1 1997 Noonan (2012) Canada 129.8 2010 James (2019) US 704.0 2016 Mean model-based 174.6 iii. Population-based Wilder (1975) US 80.0 1971 Feller (1981) US 90.0 1977 Griffin (1980) US 58.3 1980 Collins (1986) US 112.4 1981 Harvey (1990) US 72.1 1989 Cahill (2009) US 419.3 2008 CCHS (2011) Canada 360.0 2011 Armour (2016) US 464.1 2013 Mean population-based 207.0

The methods used draw on different data sources and produce widely varying estimates.

i. Hospital discharge records and registries: historically the most common data source for estimates of SCI prevalence, this method has the advantage of reliable case identification but risks missing cases that are not treated in hospital, resulting in estimates lower than the true prevalence; examination of the literature shows 6 studies of TSCI

12 prevalence using this method, with values ranging from 22.7 per 100,000 to 67.3 per 100,000 and a mean of 38.7 per 100,000. ii. Model based on the product of disease incidence and disease duration (i.e. number of new cases in one year x average postinjury life expectancy): this method is highly sensitive to assumptions made to build the model, and yields larger and more disparate prevalence estimates; 7 studies of TSCI prevalence using this method yield estimates ranging from 50.0 to 704.0 per 100,000 and a mean of 174.6 per 100,000. iii. Population-based sampling method, where every case within a representative population is identified, often using a survey: this method offers the most reliable means of capturing all cases, but is less feasible in larger populations and information provided by self-report can be problematic; 8 studies of SCI prevalence using this method range from 58.3 to 464.1 per 100,000 with a mean of 207.0 per 100,000.

In addition to differences related to the method used, some of the variance between estimates can be accounted for by other differences in methodology, such as inclusion/exclusion of cauda equina and conus medullaris injuries, inclusion/exclusion of congenital NTSCI and in particular spina bifida, and inclusion/exclusion of the pediatric population. It is not always completely clear which method is used for the various estimates presented in the literature (Thietje & Hirschfeld, 2017), making comparisons between estimates difficult. The total impact of these various methodological differences, combined with differences in the true prevalence and incidence across different geographic regions and over time, accounts for the wide range of available estimates.

2.2.3 Canadian Estimates of SCI Incidence and Prevalence

The estimates contained in a study by Noonan et al (2012) are significant because they provide the most recent and comprehensive estimates of prevalence and incidence in Canada: 41 per million incidence and 130 per 100,000 prevalence for TSCI, and 68 per million incidence and 123 per 100,000 prevalence for NTSCI (Vanessa K. Noonan et al., 2012). They are also remarkable in that they are both significantly higher than the other estimates presented in Bickenbach et al (2013), who suggest that the discrepancy of the Canadian figures may result from differences in methodology, in particular an overestimation of incidence rates.

13 2.3 Longitudinal Patterns of Chronic SCI

2.3.1 Individual Burden and Healthcare System Impact of Chronic SCI

Two factors account for the impact of SCI on individuals and the healthcare system: a long disease duration and the prevalence of secondary health conditions leading to high levels of healthcare utilization. Although the consequences of SCI can be very significant in terms of the body systems affected and the level of functional impairment, there is no subsequent degenerative process (other than accelerated aging) that causes mortality (Hitzig et al., 2008). Therefore, although life expectancy is reduced for SCI compared with non-SCI, the historically young average age at onset has meant that the average disease duration of SCI is rather long. It was calculated to be 30.2 years in 1980 (Michael J DeVivo, Maetz, Fine, & Stover, 1980), and subsequent studies have shown improved short term survival (under two years) following SCI but no significant changes in long-term survival (Strauss, Devivo, Paculdo, & Shavelle, 2006), so this figure is likely still accurate.

Individuals with SCI experience many secondary health conditions, often on a chronic basis, leading to extensive health care utilization. Several studies have documented the nature and extent of secondary health conditions in areas such as pain, bowel/bladder issues, spasms, fatigue, esophageal symptoms and osteoporosis (Jensen et al., 2013). These complications increase with age, and lead to activity limitations (Cobb et al., 2014); they are associated with higher healthcare utilization (Susan B. Jaglal, 2009; V. V. K. Noonan et al., 2014), and are predictive of mortality (Krause & Saunders, 2011). For people living with SCI, managing secondary health conditions can be a lifelong struggle (Sara J. T. Guilcher et al., 2012). Recent studies in Ontario have shown that patients with SCI experience frequent re-hospitalization following discharge from inpatient rehabilitation (Susan B. Jaglal, 2009), and a very high number of ED visits (S J T Guilcher, Craven, Calzavara, McColl, & Jaglal, 2013).

The healthcare system impact of chronic SCI arises from the occurrence of these secondary health conditions over the three decades on average that people live with a SCI. However, knowledge of the actual patterns of healthcare utilization in this population is largely limited to the initial episode of hospitalization (acute care and inpatient rehabilitation) and the subsequent year. This is because most research has focused on the acute management of SCI, as there are

14 few large longitudinal data sets to provide information on community-based management of SCI (Rowan, Chan, Jaglal, & Catharine Craven, 2019).

2.3.2 Longitudinal Datasets in Chronic SCI

The lack of longitudinal data in SCI research is well known (Devivo, 2012; Jensen et al., 2013), and is particularly problematic given the long disease duration and the high level of healthcare utilization that characterize this population. In most countries, registries of patients with SCI or large-scale cohort studies (see Table 3) gather data up to one year post injury at the most. In Canada, the Rick Hansen SCI Registry (RHSCIR) collects data on TSCI patients admitted to 30 participating sites, which are acute care and rehabilitation hospitals. The principal focus of RHSCIR is the period of inpatient hospitalization immediately following injury (V K Noonan et al., 2012). RHSCIR has proposed community follow-up interviews covering sociodemographic characteristics (such as household income, employment, and marital status) at 1, 2, 5, and 10 years post-injury and every 5 years thereafter until death or registry withdrawal, but to date results have only been reported for 1 and 5 year follow-up interviews; rates of loss to follow-up are likely high given the number of follow-up interviews reported relative to the number of database participants (Rick Hansen Institute, 2017). A conference presentation in 2017 reported national completion rates for the 1, 2, and 5 year interviews of 50%, 49%, and 32% respectively (E. Patsakos et al., Effectiveness of retention strategies to minimize participant attrition: the Rick Hansen Spinal Cord Injury Registry (RHSCIR) in (Cheng et al., 2017).

Table 3. Longitudinal SCI Datasets Dataset N Years Country Inclusion Time series NSCISC 30,892 1973- US TSCI from 28 US model Up to 40 years post- systems injury RHSCIR 6.800 2004- Canada TSCI from 31 Canadian Up to 1 year post- centres (acute & injury, 5-year follow-up rehabilitation) ASCIR 4,459 1995- Australia Traumatic SCI from 6 n/a Australian specialized units EMSCI 3,193 2001- Europe Traumatic SCI from 22 1 year post-injury units across Europe SwiSCI 2,000 2011- Switzerland Traumatic SCI from 4 1 year post-injury Swiss centres NSCID n/a 2013- UK All patients treated at 8 n/a specialized SCI centres Abbreviations: NSCISC, National Spinal Cord Injury Statistical Center; RHSCIR, Rick Hansen Spinal Cord Injury Registry; ASCIR, Australian Spinal Cord Injury Register; EMSCI, European Multicentre Study about Spinal Cord Injury; SwiSCI, Swiss Spinal Cord Injury Cohort Study; NSCID, National Spinal Cord Injury Database

15 The only major longitudinal SCI datasets (with rich data gathered from a large sample over a number of decades) are the US model systems database, housed at the National SCI Statistical Center (NSCISC) (Chen, He, & DeVivo, 2016) and the EMSCI. The US model systems database is the oldest and largest; however, it has numerous limitations, particularly for studying SCI in primary care: it covers only 13% of TSCI cases in the US (Stover, DeVivo, & Go, 1999), provides limited, survey-based longitudinal data, and does not gather objective data on healthcare utilization (M J DeVivo, Go, & Jackson, 2002). Like RHSCIR, the NSCISC database is subject to high levels of loss to follow-up: Stover et al (1999) report rates of loss to follow-up of 44.7% at year 5 and 65.7% at year 10. It is also problematic (from a Canadian standpoint) to examine primary care use in SCI with US data given the differences in primary care delivery models. It is common in the US for SCI patients to use a specialist (physiatry) based model of primary care, whereas in Canada most people receive their primary care from a family practitioner or group practice (Donnelly et al., 2007).

Most studies of the long-term outcomes of SCI use a cross-sectional design (Devivo, 2012). There is only a single large-scale long-term longitudinal cohort study of SCI: the SCI Longitudinal Aging Study, initiated in 1973, which collects data in the Midwest and southeast of the US (Krause, Clark, & Saunders, 2015). Eight assessments have been conducted at five-year intervals, using the Life Situation Questionnaire, a custom-designed instrument documenting overall psychosocial, vocational, and medical outcomes of people living with TSCI. A total of 2,208 participants have contributed one or more assessments over a period of 40 years. Compared with the NSCISC database, the SCI Longitudinal Aging Study has lower attrition/loss to follow-up and consequently a better representation of persons living with longstanding injury (>15 years).

In addition to the overall paucity of longitudinal data in SCI, there are problems with linking existing datasets, and a lack of standardized data elements and validated outcome measures to facilitate meaningful temporal analysis (Dvorak et al., 2017). The result is that the scope and cost of healthcare utilization for community-living persons with SCI are very difficult to estimate, and the economic impact of SCI in the community remains poorly understood.

16 2.3.3 Administrative Data Studies of SCI

To address the gap in research covering chronic SCI in the community, researchers have recently begun using administrative datasets that aggregate information at a large scale on categories such as physician billings, diagnoses, and medications. Recent studies in Ontario have used these data to examine healthcare utilization (S J T Guilcher et al., 2013; Sara J. T. Guilcher, 2009; Sara J.T. Guilcher et al., 2018; Susan B. Jaglal, 2009) and cost (Sara J.T. Guilcher, Bronskill, Guan, & Wodchis, 2016; Munce et al., 2013). Administrative datasets offer the benefit of detailed information about the real-world clinical environment, and in many cases cover a large proportion or even the totality of the population of interest. Examples of such datasets include the Ontario Health Insurance Plan (OHIP) which includes physician billing information, the National Ambulatory Care (NACRS) database of emergency department visits and the Discharge Abstract Database (DAD) which encompasses hospitalizations, and the Nationwide Inpatient Sample (NIS) database in the US, the country’s largest inpatient dataset, which was used to calculate the incidence of TSCI (Jain et al., 2015).

These studies using administrative data offer insight into patterns of healthcare use during the first year following discharge from inpatient rehabilitation. However, they do not comprehensively document the longitudinal perspective. Each dataset has limitations in terms of the types of data stored and the ease of extraction. One shortcoming of administrative data categories such as hospitalizations and emergency department (ED) visits is the lack of information about the primary care environment. Even physician billing data provides very limited information, only documenting diagnoses and procedures that can be associated with a billing code.

To understand the long-term healthcare experience of community-living persons with chronic SCI, it is necessary to have information about their diagnoses, characteristics and health indicators, treatments and medications over time, as well as information about their use of primary care. While information of this kind has historically been stored in the written charts of primary care practices, the adoption of electronic medical records (EMRs) has opened up the prospect of making this information accessible for research purposes (Biermans et al., 2009).

17 2.4 Using EMR Data for Research

2.4.1 Primary Care EMR Databases

The large scale use of EMRs has created a technological platform that makes possible the aggregation and sharing of data directly from primary care patient records (Birtwhistle & Williamson, 2015).

Across the world, a number of projects have undertaken the large-scale collection of data from primary care EMRs for research and public health surveillance (Gentil et al., 2017). The two largest of such databases are both located in the UK: the Clinical Practice Research Datalink (CPRD), with 13 million patients, and The Health Improvement Network (THIN), with 11.1 million patients. Other large databases include the Veterans Affairs (VA) database in the US with 17 million patients, and the Information System for Research in Primary Care (SIDIAP) database in Spain, with 6 million patients.

In Canada, there are two large databases of primary care EMR data: the Canadian Primary Care Sentinel Surveillance Network (CPCSSN), with over 1.5 million patients from across Canada (Garies, Birtwhistle, Drummond, Queenan, & Williamson, 2017), and the EMR Administrative- Linked Database (EMRALD), with approximately 400,000 patients from Ontario (Tu, Mitiku, et al., 2014). Although both databases aggregate data from EMRs of participating primary care clinics, they differ with respect to the data included and the process used to aggregate them. CPCSSN includes data in a number of fields (problem list, billings, medications, diagnoses, etc.) that has been standardized and, if necessary recoded; diagnoses, for example, are converted from varying formats, including free text, to ICD-9 codes. EMRALD comprises the verbatim free text content of all the clinically relevant fields in the chart (including progress notes and consultation letters), and the EMRALD aggregration process does not involve recoding or processing of any kind other than de-identification (replacement of all instances of patient identifiers). As a result, the free-text entries included in EMRALD are more detailed and unstructured, offering more possible methods of case identification. In CPCSSN, cases are typically identified through problem lists, medications, or using ICD-9 billing and diagnosis codes, whereas in EMRALD, cases can be identified using the addition of keywords in the free text unstructured parts of the chart.

18 2.4.2 Issues with Case Finding in EMRs

Although EMR data offers a rich and novel source of information for use in research, surveillance, and clinical improvement, there are challenges that need to be considered. Agreements with primary care practices about data sharing are necessary, as is the technical infrastructure required to aggregate data from multiple incompatible EMR systems and providers. Furthermore, any study of a particular health condition through EMR data requires a rigorous, validated method for identifying patients affected by the condition; this can be challenging given the complexity of EMRs and the variance among physicians in record keeping practices.

The quality of EMR data for use in research is subject to careful scrutiny in several respects (Hripcsak & Albers, 2013). Missingness is a serious issue due to the fragmentation of care between different clinical environments and the fact that clinical data are more frequently captured for patients who are sick than for those who are well; in this sense, EMR data is not “missing at random.” Accuracy is another concern, due to the number of steps and opportunities for error that intervene between the patient’s true state and the record preserved in the EMR (including observation of the patient, formulation of clinical opinion, and recording in the EMR); in some cases, what is recorded in the EMR can be influenced by its relationship with physician billing. Finally, there is tremendous complexity due to the varied and incompatible terminologies and ontologies used to structure and standardize medical knowledge.

With respect to missing data in the EMR, it has been demonstrated that, when hospital-based EMRs are selected for use in research using a threshold of completeness, there is a bias toward patients who are sicker on average than the population from which they were drawn (Rusanov, Weiskopf, Wang, & Weng, 2014). This is because sicker patients tend to experience a greater number of health care transactions (i.e. interventions, consultations, prescriptions, etc.), generating a greater number and volume of entries in the EMR. It is likely that, in a similar way, primary care EMRs contain greater documentation for patients with lower health status and consequently greater health care utilization. Patients with SCI are known to be high-volume users of the healthcare system, with two Ontario studies showing medians of 22 and 26 physician visits in the first year post-injury (Munce et al., 2009), (S J T Guilcher et al., 2010); as such they may be less affected by this form of bias, with the possible exception of patients with mild or transient impairment.

19 A number of recent studies have examined the use of EMRs for case-finding (also referred to as case identification/determination or as phenotyping in the context of studies related to genomic research) using a variety of methods (Barnado et al., 2017; Coleman et al., 2015; Ford, Carroll, Smith, Scott, & Cassell, 2016a; Haroon, Jordan, O’beirne-Elliman, & Adab, 2015; Teixeira et al., 2017; Xu et al., 2015). Case-finding in EMRs uses a variety of data types: diagnostic codes, medications, laboratory test results and free-text clinical narrative (Wright et al., 2011). One study demonstrated that the best performing approach to case-finding used multiple data types (W. Q. Wei et al., 2016).

Such an approach could be used in SCI; however, there are some specific constraints in SCI that need to be considered. The use of diagnostic codes is likely to be problematic because of the issues with ICD-9 and ICD-10 codes for case-finding in TSCI as documented by Hagen (Hagen, Rekand, Gilhus, & Gronning, 2008), as well as the lack of specific and validated ICD-9 and ICD-10 codes to identify NTSCI. Furthermore, it is not obvious that there are particular medications or laboratory test values that would be useful for identifying cases of SCI. For these reasons, case identification in SCI will be more challenging.

2.4.3 Problem List Use and Reliability

The problem list is the element of an EMR in which information on ongoing salient health issues is recorded and hence the most obvious place to look for evidence to support case-finding. However, there is a lack of consensus on the precise scope of the problem list, leading to issues with potential underpopulation (lack of completeness) or overpopulation (clutter) (Holmes, Brown, Hilaire, & Wright, 2012).

Problem lists are subjective and differ by physician (Krauss, Boonstra, Vantsevich, & Friedman, 2016), and inclusion of problems on the list is dependent on acuity. It has been shown that the problem list may omit problems, resulting in a risk of false negatives when used for case-finding (Szeto, Coleman, Gholami, Hoffman, & Goldstein, 2002). However, problems included in a problem list are highly reliable, with high specificity and PPV (Wright et al., 2011).

Problem lists in primary care EMRs may be more complete than those in hospital-based EMRs; one study showed that primary care physicians (PCPs) are responsible for most of the documentation in problem lists when they are shared among providers (Wright, Feblowitz, Maloney, Henkin, & Bates, 2012). A comparative study indicated that patient problems are

20 entered in the problem list with greater accuracy and granularity by PCPs than specialists (Luna et al., 2013). There is agreement among physicians that the problem list is “owned by” PCPs (Holmes et al., 2012), and in Ontario, guidelines from the College of Physicians and Surgeons of Ontario recommend ownership by PCPs of the problem list (which, in Ontario, is incorporated into a summary called the “Cumulative Patient Profile”) and recommend frequent review and updating of this information.

Overall, the problem list is an important element to consider in case-finding but is not sufficient alone; since problem lists are not necessarily exhaustive, absence of the mention of a given condition is not evidence of absence of the condition. Furthermore, if mention in the problem list is uncertain or ambiguous, additional information may help to validate the diagnosis. Other parts of the EMR, both structured and unstructured, can provide additional information to help identify cases of a condition of interest.

2.4.4 Case Finding in EMRALD

EMRALD has been used to establish the feasibility of identifying patients with a number of health conditions (see Table 4): ischemic heart disease (Ivers, Pylypenko, & Tu, 2011; Tu et al., 2010), diabetes (Tu et al., 2011), epilepsy (Tu, Wang, et al., 2014), and multiple sclerosis (Krysko, Ivers, Young, O’Connor, & Tu, 2015).

In previous case-finding studies using EMRALD, a reference standard was established by manual chart abstraction, and potential case finding algorithms were evaluated with respect to sensitivity, specificity, and positive predictive value. For example, patients with ischemic heart disease were identified using either an algorithm combining billing and hospital data (Tu et al., 2010), or one using free text entries and the problem list, medical history and treatment fields (Ivers et al., 2011). Another study examined a number of potential case-finding algorithms and found that it was possible to identify diabetics using only laboratory tests and prescriptions (Tu et al., 2011). In order to identify patients with epilepsy, data were manually abstracted from 7,500 patient records to create a reference standard against which a variety of administrative data algorithms were tested (Tu, Wang, et al., 2014).

A study examining the feasibility of identifying patients with MS in EMRALD data (Krysko et al., 2015) employed a three-step process, first using a search of EMR text fields to identify possible MS cases (flagging 943 of 73,003 patients), then manually abstracting those charts to

21 identify “true cases“ and establish the reference standard, and finally using this reference standard to evaluate a number of potential algorithms.

Table 4. Case Finding Studies Using EMRALD Study Reference Population Method Reviewed Ref cohort cohort Ivers (2011) Ischemic heart 2 keyword search algorithms (21- 19,376 969 87 disease >317 terms); tested via abstraction of 5% random sample (3 abstractors) Butt (2014) Parkinsonism Keyword search (18 terms)->2,319 73,003 2,319 231 hits; Ref cohort ID via abstraction (6 definite/ abstractors, scoring), validated with 294 AHD possible Widdifield Rheumatoid Stratified sampling (by age); Ref 73,014 9,500 69 (2014) arthritis cohort ID via abstraction (5 abstractors), validated with AHD Tu (2014) Epilepsy Random sampling (~10%); Ref 73,014 7,500 95 cohort ID via abstraction (6 abstractors, 2-step screening), validated with AHD Breiner Myasthenia Keyword search (31 terms)->645 123,997 645 49 definite/ (2015) gravis hits; Ref cohort ID via abstraction (2 256 abstractors), validated with AHD possible Krysko Multiple Keyword search (16 terms)->943 73,003 943 247 (2015) sclerosis hits; Ref cohort ID via abstraction (5 abstractors), within-EMR algorithms tested in EMR Widdifield Multiple AHD algorithms developed using 73,003 n/a 247 (2015) sclerosis reference cohort from Krysko (2015) Lee (2017) COPD Random sampling; Ref cohort ID via 73,014 5,889 364 abstraction (3 abstractors), within- EMR algorithms tested in EMR Abbreviation: Ref. cohort, reference cohort

2.4.5 Case Finding in SCI

The first step in using administrative data is cohort identification, which requires a reliable method for identifying true cases of the diagnosis being studied. A number of studies have examined the reliability of clinical terminologies such as the International Classification of Disease (ICD) for use in identifying cases of SCI. Hagen et al (2008) considered three successive versions (ICD-8, ICD-9, and ICD-10), concluding that ICD-10 was most reliable but that all three versions are insufficient for research purposes (Hagen et al., 2008).

22 Noonan et al (2012) used the RHSCIR database as a gold standard in evaluating the reliability of ICD-10 codes to identify cases of TSCI and spinal column injuries in the discharge summaries of the spine unit in an acute care hospital; they found that ICD-10 codes were only moderately reliable (V. Noonan et al., 2012). Similarly, Welk et al (2014) compared ICD-10 codes in hospital discharge summaries with other data sources (Rehabilitation Coding Groups (RCG) in the National Rehabilitation System database, and Ontario Health Insurance Plan (OHIP) billing codes), finding that ICD-10 codes were superior to OHIP fee codes for identifying cases of TSCI, but less sensitive than RCG codes (Welk, Loh, Shariff, Liu, & Siddiqi, 2014).

Ho et al (2017) examined the reliability of ICD-10 codes to identify cases of NTSCI in hospital discharge summaries, using chart review as the gold standard. They found that the codes they examined had high sensitivity and PPV, with moderate specificity and NPV (Ho et al., 2017).

Overall, these studies demonstrate that ICD codes are only moderately reliable for use in identifying cases of SCI. As Hagen et al (2008) indicate, particular caution is warranted when using ICD-9 codes rather than ICD-10.

2.4.6 Case Finding Criteria in NTSCI

In a 2017 study by Ho et al, hospital-based EMRs were reviewed to validate the identification of cases of NTSCI using a combination of ICD-10 codes (Ho et al., 2017). Four types of evidence were evaluated: evidence of sensory or motor impairment, evidence of neurogenic bladder and/or bowel, evidence of spinal surgery, and evidence of spinal cord abnormality in a diagnostic imaging report. For the first two categories of evidence, numerous potential categories of evidence and keywords were identified (see Appendix A). These criteria can potentially be used as the basis for case-finding criteria for use with primary care EMRs, adapted as necessary for the purpose of finding cases of both TSCI and NTSCI.

2.5 Thesis Objectives

A better understanding of community-based and primary care management of chronic spinal cord injury is important for the proper planning and delivery of health care to this population. The widespread adoption of EMRs in primary care has created a new data source to address this need; the rates of electronic medical record adoption have markedly increased amongst community-based family physicians in Ontario with just over 80%, supporting the secondary use

23 of EMR data for quality improvement and research (Jaakkimainen, Crampton, Pinzaru, DelGiudice, & Tu, 2018). The aggregation of EMRs into databases like EMRALD is making them available for research, and the feasibility of accurately and reliably identifying cases of a condition of interest has been demonstrated in both EMRALD and other databases across a range of health conditions.

This study will be the first to identify cases of chronic SCI in a primary care EMR database. This will require a set of case-finding criteria that adequately represent the full range and complexity of the diagnostic entity of SCI, including both TSCI and NTSCI. Using these criteria, a feasible sample of EMR charts can be manually reviewed to validate their caseness, creating a “gold standard” reference of validated cases. This reference standard cohort can then be used to develop and test case identification algorithms, which can be applied to the entire database. The prevalence of chronic SCI in the database can be calculated and the characteristics of the reference standard cohort can be described, and these can be compared with previous studies of the SCI population. Therefore the objectives of this thesis are to:

Primary objectives:

Secondary objectives:

Chapter 3 Methods Methods 3.1 Overview

3.1.1 Three-stage Case Identification/Algorithm Development Process

As this was the first study to attempt to identify cases of TSCI and NTSCI from primary care records, a considerable amount of preliminary work was needed to understand the data source and its potential to address the research objectives. Subsequently, the work required to identify cases of SCI and develop a search algorithm involved three stages, with each stage comprising several steps (Figure 4):

Figure 4: Methods overview: key process steps

• in the first stage a keyword search strategy to identify potential cases of SCI was developed via three rounds of preliminary chart review; • in the second stage a random sample of the charts identified in stage one was reviewed using a structured and exacting process to identify and validate cases of SCI, forming a “reference standard cohort”;

25 • in the third stage 125 search algorithms to identify chronic cases of SCI were developed and tested using the reference standard.

3.1.2 EMRALD Overview

EMR data were taken from the Electronic Medical Record Administrative data Linked Database (EMRALD), which aggregates data from the EMR systems of family physicians across Ontario. EMRALD is maintained by ICES (formerly known as the Institute for Clinical Evaluative Sciences).

EMRALD data are collected on a semi-annual basis from family practice clinics that use the PS Suite (formerly Practice Solutions) EMR platform and that volunteer to participate. The data used in this study were gathered from 1986 to 2016, and the available data for each patient depends on their length of time in the EMR (which depends in turn on when the patient began seeing the practice in question and when the practice adopted the PS Suite EMR system). Only practices that have been using their EMR system for at least 2 years are eligible to participate (to ensure that the EMR is adequately populated).

EMRALD contains all clinically relevant information from the EMR, including clinical notes, correspondence, prescriptions and reports on diagnostic procedures (more detail in section 3.1.5, below).

3.1.3 EMRALD Composition and Representativeness

The EMRALD is held at ICES, a health research organization in Ontario. ICES is a “prescribed entity” under Ontario provincial privacy legislation, meaning that it is able to collect individual- level patient health information without patient consent based on policies and procedures in place to protect patient privacy and confidentiality (Institute for Clinical Evaluative Sciences, 2019).

EMRALD has been validated as a data source for secondary research. An evaluation of the completeness of the records contained in EMRALD found reasonable capture when compared with administrative data (Tu, Mitiku, et al., 2014). Another study found that primary care physicians participating in EMRALD tended to show increasing completeness in almost all EMR fields with increasing time using the EMR (Tu et al., 2015). This study also indicated that the

26 EMRALD population was similar in most respects to the Ontario population with the exception of income (EMRALD patients were from higher income quartiles) and rurality (EMRALD patients were more rural than the Ontario population); with respect to age, sex, presence of major chronic conditions and measures of co-morbidity, the EMRALD population is representative of Ontario residents. At the time of data extraction for this study (December 31, 2016), EMRALD comprised 385 family physicians in 43 clinics and 443,038 patients. Average duration on EMR was 6.7 years, with a minimum of 2 years and a maximum of 25 years (Jaakkimainen et al., 2018).

3.1.4 Overview of EMRALD Charts

Häyrinen et al (Häyrinen, Saranto, & Nykänen, 2008) identify three ways of structuring an EMR: problem-based, time-based, and source-based. In the first, chart entries are associated with a specific clinical issue and follow the SOAP (subjective, objective, assessment, plan) format as first described by Lawrence Weed (1968); in the second, chart entries are arranged chronologically; in the third, chart entries are arranged according to the method by which the information was obtained (clinical note, diagnostic test report, consultation letter, etc.).

EMRALD charts are structured using elements of each of these three methods: overall, the chart entries use a time-based structure, with all entries presented in reverse chronological order following the cumulative patient profile (CPP); chart entries are colour-coded to indicate their source and type (i.e., one colour for progress notes, another for consultation letters, and so forth); and clinical progress notes typically follow the SOAP format. Many narrative entries (such as consultation letters, hospital admission/discharge reports, and some imaging reports), while not using the SOAP format per se, nonetheless use a modified version, typically presenting first the findings on examination (including details of any physical or neurological examination), followed by an assessment or diagnostic impression and a treatment plan.

The charts in EMRALD contain entries of multiple types from various authors, aggregated into a single document (see Table 5). The CPP is the first section of the chart. It is designed to provide a summary of the patient’s medical history and ongoing issues. PNs document interactions between primary care practitioners and patients, typically using the SOAP format within a problem-oriented approach to charting. Consultation letters typically begin with a medical

27 history, followed by a documentation of any examination(s) performed, an impression/assessment, and finally a treatment plan.

Table 5. EMRALD Chart Entry Types Type of Entry Abbreviation Author Description Cumulative patient profile CPP Primary care Basic information typically kept by primary care practices on all patients (includes problem list, medical history, family history, allergies, etc.) Progress notes PN Primary care Notes documenting all primary care encounters (including telephone communication) Consult letters CONS Specialists Consultation letters from specialists (including examination reports from MDs/allied health, admission/discharge summaries from hospitalizations, and surgical reports) Prescriptions n/a Primary care Prescriptions written by primary care physician Diagnostic procedure reports n/a Diagnosticians, Report of laboratory tests, laboratory diagnostic/imaging procedures, technicians etc.

3.1.5 ICES Policies and Working Arrangements

This study was undertaken in accordance with ICES policies for privacy and data security. All chart reviews were conducted by trained abstractors at the ICES Central facility. Access to the EMRALD database was governed by applicable policies and subject to approval, as documented in the project’s Dataset Creation Plan (Appendix F). In practice, access to data was mediated by ICES-appointed analysts, who responded to data requests and supported the chart review and algorithm development processes.

EMRALD chart data was reviewed via a custom online abstraction platform. The abstraction platform was developed for the use of trained chart abstractors in case identification and other analytical tasks using EMR data and has previously been used in similar studies (such as those listed in Table 4). The platform is only available on a secure computer in a secured, controlled- access ICES facility; this computer is not connected to the internet and has no input/output devices so that data sharing is physically unfeasible. These measures are in keeping with ICES’ privacy and data security practices. The abstraction platform presents data from selected records

28 in EMRALD, made available by an ICES analyst according to the requirements of the study (for example records with occurrences of particular keywords). The initial screen shows a list of available charts (identified by an anonymous number); when a chart is selected, the contents are displayed as a reverse-chronological list of entries following the CPP (as described above).

The platform was further customized during the preliminary chart review in order to support the methods used by this study; in particular, entries with a keyword occurrence were highlighted and the keyword(s) occurring in each entry were listed at the beginning of the entry. This proved helpful to reviewers when examining charts for keyword occurrences.

3.1.6 Inclusion and Exclusion Criteria

The study cohort was derived from EMRALD and had to match both inclusion and exclusion criteria. To be eligible, patients had to:

• have a valid date of birth • have a valid health insurance (OHIP card) number • be rostered to an active primary care physician • be at least 15 years old as of the date of data extraction (the pediatric population was excluded to expedite chart review and the age cutoff of 15 and older was chosen to align with other studies, such as (Vanessa K. Noonan et al., 2012) and (Bickenbach, 2013)).

Patients were excluded if they had less than one year of entries in the EMR, if their physician had been using the EMR for at less than 2 years before data collection, or if their physician was inactive.

3.2 Stage 1—Keyword Search Strategy Development

Given the variety of terms that can designate a case of spinal cord injury, it was necessary to develop a comprehensive list of search terms that could be used to identify potential cases in EMRALD and, through an automated keyword search, to select charts for full review. The overriding aim was to ensure maximal sensitivity of the keyword search (to minimize false negatives). Microsoft Structured Query Language was used to search the EMRALD database for inclusion and exclusion terms in the database (the patient’s problem list, medical history, demographic information, laboratory test results, medication list, and consultation reports).

29 3.2.1 Preliminary Chart Review: Round 1

Through consultation with Dr. Jamie Milligan of the Centre for Family Medicine in Kitchener- Waterloo (a primary care clinic with a specialized program for patients with SCI and other mobility impairments), a basic search strategy was identified comprising terms designating SCI- related impairment (paraplegia, quadriplegia), an alternate term (tetraplegia), a common misspelling (quadraplegia), and the phrase “spinal cord injury.” A keyword search using these 5 terms was conducted, looking in the cumulative patient profile (CPP), progress notes (PN) and consult letters (CONS) fields of the EMRALD database. From the charts that were identified using this search strategy, 80 were randomly selected for review. This work was undertaken in collaboration with a Senior Health Information Analyst at ICES, supported by other analysts and programmers as required, and the procedures followed were in line with previous case- identification studies in other conditions (as documented in Table 4). As the full chart review method (described below) was still in development, an abbreviated review method was used, based on the chart review criteria for NTSCI documented by Dr. Chester Ho (Appendix A); chart entries with keyword occurrences were examined for evidence of motor and/or sensory impairment and neurogenic bladder and/or bowel. In addition, each chart was read in its entirety to determine the presence of other evidence that could be useful in case identification (such as references to mobility devices or episodes of incontinence). Charts were classified as “possible” cases if they appeared to meet the criteria as described below in Table 7; otherwise, they were classified as “non-cases.”

3.2.2 Preliminary Chart Review: Round 2

For the next round of preliminary chart review, two lists of keyword search terms were used to identify charts in EMRALD: the first list included the 5 terms used in the previous round with the addition of the phrase “neurogenic b*” (the “*” indicates a “wild card” character, meaning that the search would return all words with the letters before this character regardless of which letters occur after, so that this search term would return instances of both “neurogenic bladder” and “neurogenic bowel”); the second list included 16 terms related to etiologies of NTSCI, taken from an article describing the validation of an algorithm to identify cases of NTSCI in administrative data (Jaglal (2017): Appendix Table A1.b). For a complete list of terms in both groups, see Table 12, below.

30 Based on the work undertaken to this point, an initial version of a final keyword search strategy was drafted; this strategy would be implemented within the total study cohort to identify potential cases of SCI for manual review. This search strategy (detailed in Appendix B) included a list of 12 terms based on the initial 5 suggested by Dr. Milligan, as well as a list of impairment- related terms (such as parapare*, to capture paraparesis, paraparetic, etc.) and “neurogenic bladder”/“neurogenic bowel”. The search using this list would be applied to the CPP, PN, and CONS fields of the charts in EMRALD. In addition, 2 terms (“cauda equina,” “spina bifida”) would be searched for in the CPP only (this is intended to find cases while avoiding spurious hits from the keyword search; for example, looking for “cauda equina” in PNs returned many mentions from warnings to patients re: cauda equina symptoms, while looking for “spina bifida” in CONS and PNs returned many references to spina bifida screening for pregnant women. This final keyword search strategy was modified as further exploratory work was completed, through 2 more versions (documented below).

3.2.3 Preliminary Chart Review: Round 3

In order to ensure the comprehensiveness of the keyword search strategy, a third round of preliminary chart reviews were undertaken. In this round, five separate groups of keywords were examined (for a list of the keywords in this group, see Table 13 below); the keywords were selected in consultation with a senior clinical expert, Dr. Cathy Craven. The particular concern of this round was to find possible cases that would not be identified by the basic search strategy used in Round 1, in order to answer the questions: a) which cases do we risk missing if we perform only the minimal (5-term) search strategy? and b) which other terms would be most useful to help identify these cases (which otherwise would be false negatives)?. The 5 terms from the Round 1 (minimal) strategy (i.e. paraplegia, quadriplegia, tetraplegia, quadraplegia, “spinal cord injury”) were used as exclusion terms in Round 3, so that the only cases considered in this round were ones that would not be identified using the minimal strategy; in this way, the incremental value (i.e. beyond the minimal strategy) for case identification of the terms examined in this round could be evaluated.

31 Table 6. Conclusions from Preliminary Chart Review Round 3 Group Conclusions i Include “cauda equina” in CPP only, omit “conus medullaris” ii Include “spina bifida” in CPP only; also include “spinal bifida” iii Omit “hemiplegia” iv Include “neurogenic bladder,” “neurogenic bowel” v Omit “spasticity” to limit false positives

Following the 3rd round of preliminary chart review, and based on the findings documented above, the final keyword search strategy was altered from version 1 to version 2, including 11 terms in CPP, PNs & CONS (the term spasticity is removed), and one additional term (“spinal bifida”) in CPP only. Furthermore, to address the high number of false positives in the keyword searches (for terms such as “neurogenic bladder”) due to other neurological conditions, 4 exclusion terms were added (in the CPP only): “cerebral palsy,” CP, MS, “multiple sclerosis” (see Appendix B, version 2).

3.2.4 Addition of NTSCI-Related Keywords from Literature

The multiple rounds of preliminary chart review completed up to this point involved examination of 780 primary care charts to evaluate the usefulness of various keywords to identify potential cases of SCI. Examining this relatively large sample yielded numerous possible cases of both TSCI and NTSCI, and it was noted that the possible NTSCI cases were likely to be more difficult to identify, because they tended to have fewer instances of SCI-related keywords in their charts. Given the well-curated nature of the CPP, it was felt that a keyword match in this field would be significant and that even a long list of potentially NTSCI-related diagnostic terms could be applied as part of the keyword search strategy.

In order to maximize the comprehensiveness of the keyword search strategy with respect to potential NTSCI cases, an exhaustive list of NTSCI-related terms was compiled by consulting two sources. New (2012) documents a search strategy designed for use in identifying NTSCI in medical literature, and 22 CPP search terms were taken from the suggested “gold standard” MEDLINE search strategy documented in the article’s Table 4 (these include general terms like “spinal cord compression”). In another article, New (2014) provides a comprehensive classification of the possible etiologies of NTSCI; 104 terms were taken from this list and added to the CPP search. Accordingly, version 3 of the final keyword search strategy contained a list of 143 terms in the CPP as well as the 4 exclusion terms from version 2. A group of 5 terms related

32 to common syndromes of SCI (“anterior cord syndrome,” “Brown-Sequard”) was added to the list of 12 terms from version 2 to be searched for in CPP, PNs, and CONS (this included the rare term “hemiparaplegia,” sometimes used to designate Brown-Sequard syndrome). The final search strategy is summarized in Figure 5 and documented in Appendix B.

Specific terms (17) Non-trauma general terms (22) In CPP, PNs, CONS -spinal cord injury In CPP only Non-trauma etiologies (104) -paraplegia/-paresis, Source: New (2012) tetraplegia/-paresis, In CPP only quadriplegia/-paresis -myelitis Source: New (2014) (+ variants) -spinal cord… -hemiparaplegia -achondroplasia …atrophy, compression, cyst, -acromegaly -syndromes (anterior, central, disease, haemorrhage, -actinomycosis posterior, conus medullaris, hemorrhage, infection, -aneurysmal bone cyst brown sequard) ischaemia, ischemia, lesion, -arachnoiditis -neurogenic bladder/bowel malformation, neoplasm, tumor, -arnold chiari vascular disease -astrocytoma -spinal paralysis -atheromatous embol% -atlanto axial dislocation -syringomyelia -atlanto axial instability -transverse myelitis -b12 deficiency […] complete list in Appendix C

Figure 5: Final keyword search strategy

In addition to yielding a comprehensive list of search terms to identify potential cases of SCI, the preliminary chart review process generated several useful observations to help guide the development of the chart review process. The variable reliability of different types of chart entries was noted: a mention in the CPP was much more likely to identify a case accurately than mention in PNs, which contain much more varied information and are therefore more likely to generate spurious keyword hits. Furthermore, use of diagnostic terms in CONS tended to be quite careful, and CONS from certain specialties (such as neurosurgery, neurology, and physiatry) often contained additional evidence to help validate diagnosis, including reports of neurological examinations. CONS from urology often clearly documented neurogenic bladder. The chart review process was designed to take these observations into account in the methods used to examine charts and the hierarchy of evidence used to evaluate evidence of SCI caseness.

33 3.3 Stage 2—Chart Review and Case Identification

When applied to the EMRALD database, the keyword search strategy yielded a cohort of potential cases of SCI. In the next stage, a random sample of these records was carefully reviewed using a structured multi-step process involving three reviewers. The chart review process was informed by the literature on the reliability of EMRs and took into consideration the evidentiary value of different portions of the electronic records.

3.3.1 Considerations for Identification of Chronic SCI Cases

Identifying cases of a particular health condition requires a clear definition of that condition. The definition can be based on specific values of a diagnostic test, physical examination imaging, reported symptoms, all in various combinations. Defining a disease or health condition and identifying cases of it can be subject to several forms of uncertainty.

In clinical practice, the assignment of a diagnosis can be subject to varying levels of uncertainty, depending on the particularities of the patient and the nature of the condition in question. Some cases may be obvious, where others may be subtle, borderline or questionable. Once a diagnosis is assigned by a clinician, the manner in which it is recorded in clinical records can introduce further uncertainty. Nomenclature is an issue, whether the diagnosis is recorded using words or diagnostic codes. Clinical records may not include every diagnosis for every patient, so there is always the danger of missing information.

3.3.2 Case Definition of Chronic SCI

Based on prior work to characterize the definition of TSCI and the diagnostic criteria for NTSCI (reviewed above), and with the aim of being as comprehensive as reasonably possible when identifying cases of chronic SCI, the following definitions were established for use in the chart review:

34 Table 7. Case Definition: Criteria for Chronic TSCI and NTSCI Case Identification NTSCI Criterion TSCI NTSCI (due to degeneration) Any of motor, Motor or sensory Only motor impairment Type of sensory, bowel or impairment AND (subject to review) neurological bladder impairment bowel or bladder impairment impairment

Duration of Any duration impairment

Injury site Spinal cord and cauda equina

External physical Etiologies as identified Etiologies as identified by force by New (2013): New (2013): acquired Cause/etiology congenital, genetic, and degenerative acquired (non- degenerative) Typically immediately Variable Often immediately Point of maximal following surgery/ preceding impairment hospitalization surgery/hospitalization

When there is traumatic injury to the spinal cord, any type of impairment is accepted as evidence of TSCI whereas evidence of NTSCI requires the patterns of impairment specified by Ho et. al. (Ho et al., 2017) as detailed in Appendix A. This difference reflects the degree of uncertainty attendant on diagnoses of NTSCI, where the symptomatic presentation may not be clearly related to the cause, compared with TSCI, where the traumatic insult is typically well-recognized.

The point of maximal impairment is used to make the determination whether a case meets the criteria, and typically differs according to the type of injury. In TSCI, the greatest impairment typically occurs immediately following the traumatic insult and is documented in diagnostic imaging and reports of surgery or hospitalization. In NTSCI, maximal impairment varies depending on the cause, whether congenital, like spina bifida, versus a disease process, like cancer or vascular conditions. With NTSCI due to degeneration, like stenosis, the maximal impairment typically precedes hospitalization. This is noteworthy because the chart review process followed a reverse temporal sequence (according to the order in which entries were recorded in the EMR).

This definition has been designed to be maximally comprehensive with respect to the divergences noted above. It is also comprehensive with respect to the inclusion of spina bifida

35 (which is sometimes treated as a separate entity, as discussed in the WHO International Survey (Bickenbach, 2013). However, it should be noted that all cases of multiple sclerosis have been excluded (even those which may be related to spinal lesions).

3.3.3 Chart Entries Used for Case Identification

The charts contained in EMRALD are comprehensive in that they contain all available entries in a patient’s EMR as described in Table 5, above. Upon examination during the preliminary review process, it became clear that various types of entries contain information relevant to determining if a patient has chronic SCI. Entries that report on a patient’s medical history or that document a relevant examination (such as a neurological examination) are most likely to contain good evidence of SCI caseness. Within the CPP, the first two sections were most useful: the problem list and past medical history; in PNs, the assessment section was most useful; and in CONS, the history, examination, and impression sections were most useful for determining caseness.

Table 8. EMRALD Chart Entries Used for Case Identification Typical Format Type of Entry (Bold/Italic: most useful for case determination) Cumulative patient • problem list (ongoing issues/diagnoses) profile (CPP) • past medical history • personal traits (risk factors, family, employment) • family history • allergies, treatments, immunizations Progress notes (PN) SOAP: • subjective • objective • assessment • plan Consult letters • history (CONS) • examination • impression/assessment • plan

3.3.4 Hierarchy of Evidence

Classification of patients according to whether they are cases of a given condition is not the same task as making a diagnosis of such a condition. The task of the chart reviewer is to determine

36 whether evidence contained in the chart justifies classification as a case; it is not to interpret examination findings or diagnostic reports. The diagnosis is made by clinicians and reported in the chart. The reviewer may look for evidence to support case identification, particularly when information about the diagnosis is ambiguous or fragmented. Given the heterogeneity of terms that can potentially refer to SCI (as shown in the final keyword search strategy, Appendix B), it may be necessary to verify that a particular diagnosis meets the case-finding criteria for chronic SCI.

During the preliminary chart review process (detailed above), it became clear that the SCI- related keywords used to search the charts have different evidentiary value depending on their meaning in the context in which they occur, and depending on the expertise of the author of the document where the keyword is found. An occurrence in the problem list (initial) portion of the CPP carries very high value, as these entries are made carefully and deliberately by the primary care physician (as documented in section 2.4.3, above); they have therefore an unambiguous meaning and reflect relatively certain knowledge and a conclusive diagnostic verdict. An occurrence of the same term in a progress note, however, can be much more ambiguous in meaning, carrying therefore much lower value: it can reflect uncertainty or even negation (often preceded by terms like, respectively, “consider” and “rule out”); it can also contain an irrelevant reference, often taking one of the following forms:

• Reference to family history or other related circumstances (e.g. “son has spinal cord injury”); • Reference in documentation of informed consent (e.g. “patient warned of risk of spinal cord injury” related to surgical intervention); • Reference to clinic/facility name (e.g. “spinal cord injury unit” in physician’s title/address).

It became clear that inclusion of a keyword term in the CPP was meaningfully different from inclusion in PNs with respect to evidentiary value for case ascertainment. At the same time, keyword occurrences tend to be more frequent in PNs than in the CPP (particularly so for terms other than diagnostic labels). The construction of the final 143-word keyword search strategy was premised on this, with a focused list of 17 keywords to search for in the PNs as well as CPP, and a much more expansive list of another 126 to search for in the CPP only. Likewise, the quantitative analysis of keyword occurrences (described below) took into account the difference between CPP and PN/CONS occurrences.

37 In addition to these clear differences based on context (meaning), the value of keyword terms varied according to the expertise of the medical professional providing the evidence (author). Primary care physicians are the authors of the CPP and PNs; although they are not experts in assigning a diagnosis of SCI, inclusion in the CPP or unambiguous reference in a PN can be taken to represent their conclusion based on the evidence available to them (including potentially an expert diagnosis not otherwise available in the chart). Keyword occurrences in CONS from specialists in disciplines whose domain includes assessment and diagnosis of SCI (i.e., neurosurgery, neurology, physiatry) are considered to have a higher evidentiary value due to this specialist expertise, although the meaning of the keyword occurrence in its context needs to be carefully examined. An added advantage of keyword occurrences in CONS is that they often document a physical/neurological examination and provide a detailed diagnostic assessment/impression. In addition to the three disciplines mentioned previously, CONS from urology were considered authoritative in documenting neurogenic bladder.

Taking these considerations together, a hierarchy of evidence (see Figure 6) was used during the review process to assess the value of different kinds of evidence gleaned from keyword searches in EMRALD charts:

Highest value • Clear diagnostic impression in CONS from:

• Neurosurgery, neurology, physiatry (SCI)

• Urology (neurogenic bladder)

• CPP diagnosis

• Mention in PNs Lowest value

Figure 6: Hierarchy of evidence

3.3.5 Developing a Reference Standard: Chart Review Process

In order to assess the usefulness of potential algorithms for identifying chronic SCI patients, it was necessary to create a reference standard cohort of validated cases via a manual chart review process. This reference standard would then provide a “gold standard” cohort that could be used to test the performance (i.e., sensitivity, PPV, and F-score) of potential algorithms. The manual

38 chart review included five steps and involved two trained chart abstractors and a senior clinical authority.

Table 9. Chart Review Process Steps Step Description 1 Initial review by first reviewer: classification as definite case (DEF), possible case (POSS), or non-case (NO) 2 Review by second reviewer: validation of 10% sample of DEF, NO, review & reclassification of all POSS 3 Consultation between 1st & 2nd reviewers 4 Review of all remaining POSS by senior clinical authority 5 Final review of uncertain cases with senior clinical authority

Potential cases of SCI were identified using the final keyword search strategy described above. When applied to the study cohort of 213,887 patients, this search strategy yielded 3,457 potential SCI cases. In order to select a suitable number of charts for manual review, a subset of 48,000 patients was randomly sampled from the study cohort. Within this sample there were 803 potential SCI cases identified by the keyword search strategy.

Two trained abstractors and a senior clinical expert reviewed the charts of the 803 potential SCI cases to identify a reference standard of verified SCI cases. A standardized review process (described below) was used to examine charts for evidence of SCI. Demographic information and the type, cause, and date of injury (where available) were recorded.

The chart review process was designed to ensure rigorous application of the diagnostic criteria (from Table 7), with validation by a second reviewer and determination by a senior clinical expert who consulted on case determination for difficult or uncertain cases. All 803 charts were initially reviewed by the first reviewer. The second reviewer (an MD with extensive clinical experience treating SCI patients) then reviewed a 10% sample of all charts classified “definite cases” and “non-cases” by the first reviewer. In addition, the second reviewer also reviewed all cases classified “possible” in order to determine if their status could be ascertained. The second reviewer was given the same data to review using the same platform as the first reviewer; however, the first reviewer’s ratings were not provided. In total, the second reviewer manually reviewed 157 charts.

In order to record relevant details, a chart abstraction template was created, with columns to capture information in each of the following categories:

39 • CPP details (relevant to SCI caseness) • Other diagnoses (MS, CP, Parkinsons, stroke, diabetes) • Evidence of impairment • Motor • Sensory • Bladder • Bowel • Injury characteristics (for cases) • Trauma/non-trauma • Cause/etiology • Level, completeness, ASIA (if available)

The CPP details relate to evidence of SCI, as well as evidence of other diagnoses that can be mistaken for SCI given their similar pattern of impairment (MS, stroke) or neurogenic bladder (diabetes). The evidence of impairment was recorded separately for each of the four categories of potential impairment: motor, sensory, bladder, and bowel. Finally, for charts identified as cases of SCI, details concerning the SCI were recorded where available.

Chart Review Process

Read entire CPP

Read all entries with keyword occurrences

Read all consult letters from: physiatry, neurosurgery, neurology, urology, physical therapy, occupational therapy

Y Case-finding Partially Definite Conduct targeted keyword search criteria met?

N Y Case-finding Partially No Definite criteria met?

N Possible: forward No nd to 2 reviewer

Figure 7: Chart review process

The first step in the manual chart review was to read the CPP carefully, focusing on the problem list and medical history sections. Primary care physicians make copious use of abbreviations and acronyms, and this is particularly the case in the CPP, which is meant to be reviewed rapidly on a

40 frequent basis. These acronyms can be ambiguous, with one condition potentially represented several different ways (e.g., adult-onset diabetes can be DM, DM2, or NIDDM), or with one acronym “overloaded” with several different meanings (e.g. “MS” can signify “multiple sclerosis,” “mitral stenosis,” or “mental status”); therefore, care was taken to interpret acronyms correctly, by referring to standard reference sources and, where necessary, consulting with a senior clinical expert.

After reviewing the CPP and noting any relevant details (particularly keyword occurrences), the reviewer would then read, in reverse chronological order, all chart entries with an occurrence of any of the 143 keyword search terms, noting relevant details such as evidence of impairment. Next, the reviewer would read all consult letters from neurosurgery, neurology, and physiatry looking for a diagnosis of SCI, as well as consult letters from urology looking for a diagnosis of neurogenic bladder. Additionally, consult letters from physical therapy and occupational therapy were also read in case they contained evidence of SCI.

Having completed these three initial steps, the reviewer would then assess whether the case identification criteria were clearly and unambiguously met; if so, the chart would be classified as a definite case or “DEF.” If there was no evidence whatsoever of SCI (in cases where the keyword occurrences were due to irrelevant references such as those reviewed above), the chart was classified as a non-case or “NO.” If there was partial evidence of SCI but not enough to fully meet the case-finding criteria, a targeted keyword search was undertaken within the chart to see if additional evidence could be found; for example, if the initial review produced evidence of sensory and bladder impairment but none of motor impairment, the reviewer would search in the chart using keywords such as “wheelchair,” “walker,” and “cane” to find evidence of motor impairment.

If, following this targeted keyword search, the evidence fully met the case-finding criteria, the chart would be classified as “DEF;” otherwise, it would be classified as a possible case for further review or “POSS.”

3.4 Stage 3—Algorithm Development

Having established a reference standard of chart abstracted identified cases of chronic SCI within the 48,000-patient random sample, the next objective was to develop an algorithm that would

41 permit identification of chronic SCI cases within the entire EMRALD database. Based on the approach used in similar studies (see Table 4, above), several types of information contained in the chart were considered for use in algorithm development: free-text keywords, medication names and, diagnostic (ICD-9) codes.

EMRALD case identification algorithms were generated in an iterative fashion and tested against the reference standard. Discordance analysis was performed to examine false positive and false negative results, and to determine whether there were inaccuracies in the selected algorithm.

3.4.1 Keyword Analysis and Keyword Lists

Given the different significance and evidentiary value of keywords depending on where they appear in the chart (particularly in the CPP vs elsewhere), free-text keywords were considered separately in the CPP and in the CONS/PNs. In both cases, to determine the value of each keyword for case identification, keyword occurrences in the reference standard charts were analyzed.

For each keyword term from the CPP, the confusion matrix values in the EMRALD sample were tabulated (true positive, false positive, true negative, false negative), and the sensitivity and positive predictive value (PPV) (see definitions below) of each CPP keyword were calculated.

Using this analysis, 3 lists of CPP keywords were developed (Table 10): CPP1, containing the 9 most precise terms (all with zero false positives, hence PPV of 100%); CPP2, containing the terms in CPP1 as well as the 4 next most precise terms (PPV of 71-92%); CPP3, containing the terms in CPP2 as well as the 12 next most precise terms (PPV of 24-67%). CPP1 and CPP2 also comprised a list of 4 exclusion terms, and CPP3 comprised the same 4 exclusion terms with two others in addition. Grouping the terms in this way facilitated the evaluation of different algorithm permutations with respect to the tradeoff between sensitivity and PPV.

42 Table 10. CPP Keyword Lists

Keyword List Terms included Terms excluded CPP1 brown sequard cerebral palsy central cord syndrome CP meningocele guillain barre neurogenic bowel multiple sclerosis (cases identified quadrapare* using EMRALD MS algorithm) SCI spinal cord injury tetrapare* tetraplegi* CPP2 CPP1+ cerebral palsy cervical myelopathy CP paraplegi* guillain barre quadraplegi* multiple sclerosis (cases identified quadriplegi* using EMRALD MS algorithm) CPP3 CPP2+ cerebral palsy arnold chiari CP cauda equina guillain barre epidural abscess multiple sclerosis (cases identified myelitis using EMRALD MS algorithm) myelopathy occulta neurogenic bladder osteomyelitis quadripare* spina bifida syringomyelia tethered cord transverse myelitis

A similar analysis was performed on keyword occurrences in the CONS and PNs (considered together). Unlike the CPP, it was possible to have multiple keyword occurrences in CONS/PNs, therefore analysis in terms of sensitivity and PPV was not meaningful; rather, for each term, the number of occurrences each in charts rated “DEF,” “POSS,” and “NO” was tabulated, and the proportion of occurrences in charts rated “DEF” and “NO” was calculated. Based on this analysis, the keywords were ranked in descending order of the proportion of “Def” occurrences to total occurrences, and 13 of the 14 terms used in the CONS/PN keyword search were retained for use in the CONS/PN algorithm component; the term “neurogenic bladder” was omitted because of the large number of occurrences of this term in charts rated “NO” (due, for example, to diabetes or MS).

43 CONS/PN Keyword List • brown sequard • central cord syndrome • hemiparaplegia • neurogenic bowel • parapare* • paraplegi* • quadrapare* • quadraplegi* • quadripare* • quadriplegi* • spinal cord injury • tetrapare* • tetraplegi*

Figure 8: CONS/PN keyword list

This algorithm component could specify either a certain number of occurrences of a unique term from the list (e.g. at least three occurrences of any of the terms in the list) or a certain number of occurrences of any of the terms from the list (e.g. at least three occurrences in any combination of any of the terms in the list). As a general rule, when a smaller number of terms and unique occurrences are specified this algorithm would have higher PPV and lower sensitivity; conversely, a larger number of terms in any combination would result in higher sensitivity and a lower PPV.

3.4.2 Other Algorithm Components Considered

Prescriptions for medications commonly used by SCI patients were used as another component in algorithm development. Prescriptions can be useful for case identification in populations (for example MS) where there are medications that are used by a substantial portion of the total population and only by that population (i.e. for that indication alone). In order to identify medications commonly used in the SCI population, we consulted the available literature on pharmacological use in SCI (Sara J.T. Guilcher et al., 2018; Høgholen et al., 2017; Hwang, Zebracki, & Vogel, 2015; Patel, Milligan, & Lee, 2017; Rouleau & Guertin, 2011). This showed that many of the most widely used classes of medications in the SCI population—bowel agents, analgesics, antidepressants and antibiotics—were not useful for case-finding because of their widespread use across many populations. A database of indication-medication pairs (W.-Q. Wei et al., 2013) was also consulted to determine which drugs were most commonly prescribed for SCI. Of these, the reverse pairs (medication-indication) were then examined to determine which

44 of these were prescribed more commonly for SCI than for other indications. From this analysis, three medications were included as an algorithm component: oxybutynin (Ditropan), baclofen (Lioresal), and dantrolnene sodium (Dantrium).

Physician billing codes related to SCI were identified in the literature and were used as another algorithm component. EMRALD uses ICD-9 codes, and a list of codes based on the literature reviewed above, in 2.4.5 (codes 344, 806, and 952) was used as another algorithm component. The EMRALD database did not support searching for more specific codes (i.e. using more than three digits).

3.4.3 Evaluating Algorithms

Algorithm performance was calculated using three metrics. Comprehensiveness, or the ability of an algorithm to identify as many cases as possible, was measured using sensitivity (also called recall), defined as: True Positives True Positives + False Negatives

The ability of an algorithm to identify only true cases was measured using positive predictive value (PPV, also called precision), defined as: True Positives True Positives + False Positives

The optimal case finding algorithm would have both high sensitivity and high PPV, but in practice there tends to be a tradeoff between the two; the metric used may depend on the task at hand, and which is more important: recall or precision. If a single overall metric is desired, however, the F-score (also called F1-score or F-measure) can be used; this is the harmonic mean of sensitivity and PPV, defined as:

2 x (True Positives) 2 x (True Positives) + False Positives + False Negatives

For the purpose of this study, the single case identification algorithm with the highest F-score will be chosen as the optimal algorithm for identification of cases of chronic SCI in EMRALD. However, other algorithms could be chosen for other purposes. In particular, if it were imperative to select a cohort with, as much as possible, only true cases of chronic SCI (i.e.

45 minimize false positives), the algorithm with the highest PPV could be selected. Conversely, if it were imperative to assemble as complete a cohort as possible, without omissions (i.e. minimize false negatives), the algorithm with the highest sensitivity could be selected.

Chapter 4 Results Results 4.1 Stage 1—Keyword Search Strategy Development

In each of the three rounds of preliminary chart review, potential terms for the keyword search strategy were evaluated by examining records in which each term occurs. Terms occurring in records that were determined to be potential cases were retained for the keyword search strategy, and terms which only occurred in non-cases were omitted.

4.1.1 Round 1

In the initial round, the search strategy proposed by the Centre for Family Medicine was used and a random sample of 80 charts was reviewed; this was the first use of an early version of the chart review process and, given the uncertainty about the time required for this process, it was hypothesized that this sample size would be large enough to yield several possible cases while remaining feasible. As the preliminary review process proved manageable with respect to time required (approximately 6 minutes per chart on average), subsequent rounds were expanded to include 100 charts for each search strategy examined.

Table 11. Summary of Preliminary Chart Review Round 1 Keywords Charts Possible Non- Source Inclusion Exclusion Reviewed cases cases Consultation paraplegi*, tetraplegi*, None 80 9 71 with primary quadriplegi*, quadraplegi*, care clinic “spinal cord injury”

Note: “*” indicates a wild-card character, i.e. can be replaced by any number (including zero) of other characters

From the 80 charts reviewed in this round (Table 11), 9 possible cases of SCI were identified: 6 TSCI and 3 NTSCI. Of the 6 possible TSCI cases, 2 had evidence of ongoing impairment whereas 4 showed evidence of impairment limited to a brief period of time. In addition to the 3 possible NTSCI cases (which met the criteria of both motor/sensory and bladder/bowel impairment), there were another 26 charts that mentioned a potential etiology of NTSCI (e.g.

47 myelopathy, spinal stenosis, spondylolisthesis) without evidence of impairment sufficient to meet the case identification criteria.

Among the 71 charts that did not appear to be cases of SCI, there were 4 that had related diagnoses: cerebral palsy (2) and multiple sclerosis (2). Of the remaining charts, some included the keyword search terms in the context of a consultation letter warning of the risks of a treatment (which could include paraplegia or quadriplegia), and some included a mention in a progress note of a family member with a history of SCI.

This first round of preliminary review afforded an initial opportunity to work with the EMRALD abstraction platform and examine the charts in detail. Several observations helped direct subsequent work: examination of the possible cases identified yielded additional search terms (in particular, it was observed that “neurogenic bladder” and “neurogenic bowel” were well documented in progress notes and in consult letters from specialties such as urology; the presence of alternate terms such as paraparesis to identify SCI-related impairment was also noted); the small number of possible NTSCI cases identified, compared with the larger number of charts mentioning a potential etiology of NTSCI, suggested the need to further explore the use of NTSCI etiologies as keywords in the search strategy; and, finally, the identification of possible TSCI cases in which symptoms of impairment were temporally limited and ultimately resolved underlined the importance of clarity in the definition and criteria for case identification in this respect. In order to validate and further explore these findings, another round of preliminary chart review was undertaken.

4.1.2 Round 2

In the second round, two search strategies were used. The incremental value of the terms “neurogenic bladder” and “neurogenic bowel” was examined by adding “neurogenic b*” (where “*” is a wild-card character) to the search from round 1, and the value of terms relating to potential etiologies of NTSCI was examined by using a search strategy with 16 such terms (Table 12).

48 Table 12. Summary of Preliminary Chart Review Round 2 Keywords Charts Group Possible Non- Source Inclusion Identified Reviewed cases cases i Same as round paraplegi*, tetraplegi*, 2,457 100 13 87 1, plus quadriplegi*, “neurogenic quadraplegi*, “spinal b*” cord injury,” “neurogenic b*” ii Article by “intraspinal abscess,” 35,583 100 5 95 Jaglal (2017) “spinal muscular atrophy,” “SMA,” “syringomyelia,” “syringobulbia,” “vascular myelopathy,” “cord compression,” “transverse myelitis,” “ankylosing spondylitis,” “spondylosis,” “discitis,” “spinal stenosis,” “osteomyelitis of vertebra,” “spondylolysis,” “spondylolisthesis,” “spina bifida”

The first search strategy (Group i) identified 2,457 potential cases and the second strategy (Group ii) identified 35,583 potential cases. From the 100 cases randomly selected for review in Group i, 13 possible cases were identified (10 TSCI, 3 NTSCI); from the 100 cases selected in Group ii, 5 possible cases were identified (all NTSCI).

This round of preliminary review highlighted several issues. It demonstrated that terms related to NTSCI etiology (Group ii) occurred much more frequently in the EMRALD database than the initial list of impairment-related terms (Group i). At the same time, Group i yielded more possible cases than Group ii. This suggested that the use of NTSCI-related search terms would be likely to return a greater number of charts, but with a smaller number of possible cases, leading to a trade-off between comprehensiveness of case detection and the feasibility of the manual review process. Examination of non-cases illustrated several reasons why charts of non-cases were identified by the keyword searches: neurogenic bladder from another cause such as diabetes, related diagnoses leading to similar impairment such as MS or stroke, and mention of SCI in a different context (family history, or warning of the risks of a procedure).

49 4.1.3 Round 3

The results of the first two rounds of preliminary chart review were examined with a senior clinical expert and a third round was undertaken, comprising five separate search strategies. Unlike the previous rounds, these strategies also specified exclusion terms; the search strategy (inclusion terms) from round 1 was excluded in this round to determine the incremental value of the five strategies examined (Table 13).

Table 13. Summary of Preliminary Chart Review Round 3 Keywords Charts Group Possible Non- Source Inclusion Exclusion Identified Reviewed cases cases i Consultation “conus 6,050 100 8 92 with clinical medullaris,” expert “cauda equina” ii “spina bifida” 4,363 100 5 95 iii parapare*, “spinal cord 871 100 2 98 tetrapare*, injury,” quadripare*, paraplegi*, quadrapare*, tetraplegi*, hemiplegi* quadriplegi*, iv “neurogenic quadraplegi* 740 100 70 30 bladder,” “neurogenic bowel” v spasticity 2,092 100 3 97

Group i included the terms “cauda equina” and “conus medullaris,” and identified 6,050 charts. From the 100 charts reviewed, 8 possible cases were identified. All 8 included the term “cauda equina” and in 5 of these, the term appeared in the CPP. Given the greater predictive value of a mention in the CPP than in PNs, it was concluded that the term “cauda equina” could be included in the CPP only, and that the term “conus medullaris” could be omitted.

Group ii included the term “spina bifida” and identified 4,363 charts. From the 100 charts reviewed, 5 possible cases were identified. Four of these included the term in the CPP; in the fifth, the term was misspelled in the CPP as “spinal bifida.” Accordingly, it was concluded that the term “spina bifida” could be included in the CPP only, and that the misspelling “spinal bifida” should also be included.

Group iii included 5 impairment-related terms (parapare*, tetrapare*, quadripare*, quadrapare*, hemiplegi*) and identified 871 charts. From the 100 charts reviewed, 2 possible cases were

50 identified. One case mentioned “hemiplegia post-MVA,” although the chart in question also documented a stroke leading to uncertainty concerning the origin of the impairment; the other case mentioned paraparesis as well as “cauda equina.” Based on the low prevalence of possible cases in this group and the relatively small number of charts identified by the 5 keywords, it was concluded that the search terms could be omitted. It was further observed that the term “hemiplegia” occurred more frequently in relation to non-SCI impairments (especially stroke) and was particularly unlikely to be useful. It should be noted that it was subsequently decided to include these terms, other than hemiplegia, in the final keyword search strategy.

Group iv included the terms “neurogenic bladder” and “neurogenic bowel,” and identified 740 charts. From the 100 charts reviewed, 70 possible cases were identified. Of the 30 non-cases, most (25) mentioned neurogenic bladder due to MS. Based on this result, it was concluded that both search terms should be included in the CPP as well as PNs and CONS.

Group v included the term spasticity and identified 2,092 charts. From the 100 charts reviewed, 3 possible cases were identified. It was noteworthy that all 3 also included another keyword term (“neurogenic bladder” or paraparesis). Based on the low prevalence of possible cases, the co- occurrence of other keywords, and the desire to minimize false positives (from mentions of spasticity due to stroke, or other kinds of spasticity) from the keyword search, it was concluded that the term spasticity could be omitted from the keyword search strategy.

4.2 Stage 2—Chart Review and Case Identification

4.2.1 Overview

Through the manual review of electronic medical records, this study identified a cohort of 126 validated cases of spinal cord injury. Using this cohort as a reference standard, case identification algorithms were developed and evaluated. The overall process is depicted below, and key results summarized in Figure 7. The 126 SCI cases in the shaded box became the “reference standard cohort” or RSC.

51 Flowchart for SCI case identification and algorithm development

Figure 9: Flowchart for SCI case identification and algorithm development

From the total eligible EMRALD population of 443,038 individuals as of mid-2018, inclusion and exclusion criteria (see Section 3.1.6) were applied, resulting in a study cohort of 213,887 patients. A random sample of 48,000 patients was taken, and a keyword search strategy consisting of 143 terms was used to identify possible cases of spinal cord injury within this study cohort. The keyword search strategy yielded 803 possible cases, which were manually reviewed. After multiple rounds of manual review involving two reviewers and a senior clinical authority, 126 of these were determined to be cases of SCI; 57 were identified as cases of TSCI, 67 were identified as cases of NTSCI, and it was impossible to identify the remaining two cases as either TSCI or NTSCI.

52 For the 677 charts reviewed and determined not to be cases of SCI, there were several reasons why these charts were identified as possible cases by the keyword search: for example, some conditions had similar keywords associated with them (such as multiple sclerosis or cerebral palsy), the chart could contain a mention of spinal cord injury that is irrelevant to the patient’s caseness (for example, family history of SCI), or it could have insufficient information to justify a diagnosis (see Appendix C). Using the 126 validated cases of SCI as RSC allowed development of case identification algorithms using different components of the EMR. The optimal algorithm was identified and, if applied to the overall EMRALD database, would be expected to yield approximately 500 cases of SCI.

4.2.2 Chart Review Ratings by Round

The keyword search strategy applied to the 48,000 patient sample identified 803 charts as possible cases of SCI. These were manually reviewed by two trained chart reviewers and a senior clinical expert using the 5-step process described above (in 3.3.5). Following the initial review by the first reviewer, 78 charts were identified as definite cases and 83 as possible cases, leaving 642 non-cases (Table 14).

Table 14. Chart Review Process Summary: Ratings by Process Round Step Description DEF NO POSS 1 Initial review by first 78 642 83 reviewer 2 Review by second 98 676 29 reviewer 3 Consultation between 1st 106 672 25 and 2nd reviewers 4 Review of “Possible” 117 674 12 cases by senior clinical authority 5 Final review with senior 126 667 10 clinical authority

During the initial round, the most easily adjudicated cases (those with abundant and unambiguous documentation) were categorized, leaving 83 identified as possible cases requiring further review. In the next round, the second reviewer adjudicated as many as possible of these cases classified as “Possible” by the first reviewer, reducing their number to 29. In addition, the second reviewer examined a randomly selected 10% sample of cases classified as “Definite” or “No” by the first reviewer. For these 79 dual-reviewed cases, agreement between the two

53 reviewers as measured by Cohen’s kappa was 87.37%, (% of agreement 97.30), indicating strong agreement.

Of the 29 remaining possible cases following the second round, a further four were categorized in the 3rd round via consultation between the first and second reviewers. In the next (4th) round, the senior clinical authority reviewed all remaining “Possible” cases, leaving 12. In the final (5th) round, 15 borderline or particularly complex cases were reviewed again with the senior clinical authority, resulting in the final classification of 126 definite cases, 667 non-cases, and 10 remaining possible cases (which, since they did not meet the criteria for case identification, became non-cases).

As Figure 10 depicts, the review process demonstrated a progressive reduction of uncertainty, with a gradual winnowing and adjudication of difficult/marginal cases; therefore, an increase in the number of definite cases and non-cases, and a shrinkage in the number of possible cases.

0 100 200 300 400 500 600 700 800

Round 1

Round 2

DEFDefinite Round 3 POSSPossible NONo Round 4

Round 5

Figure 10: Chart review ratings by round

4.2.3 Demographic Characteristics

The demographic characteristics of the RSC were distinct from the overall eligible EMRALD population in a number of respects (see Table 15). Patients in the RSC were more likely to be male (39.7% female), whereas EMRALD patients were more likely to be female (58.3% female). Patients in the RSC were also older, with a mean age of 55.1 years, compared with 50.8 years for

54 EMRALD patients. Finally, patients in the RSC were slightly less likely to be urban-dwelling (73.8% compared with 80.8% urban for EMRALD).

Table 15. Demographic Characteristics of RSC Compared with EMRALD

RSC EMRALD n=126 n=213,887 Sex Female, n (%) 50 (39.7) 124,776 (58.3) Male, n (%) 76 (60.3) 89,111 (41.7) Age Mean age, years (sd) 55.1 (17.0) 50.8 (19.1) Age 14-34, n (%) 16 (12.7) 50,357 (23.5) Age 35-64, n (%) 74 (58.7) 108,946 (50.9) Age >=65, n (%) 36 (28.6) 54,584 (25.5) Location Urban, n (%) 93 (73.8) 172,769 (80.8) Rural, n (%) 29 (23.0) 38,885 (18.2)

When the RSC is separated into cases of TSCI and NTSC, a different picture emerges (Table 16). Cases of TSCI were much more likely to be male (28.1% female compared with 50.7% female for NTSCI cases). The mean age was similar between the two groups, although with greater dispersion in the case of NTSCI, due to a greater proportion of NTSCI in the youngest age group: 14-34 years (12.78% compared with 8.8% for TSCI), most likely due to the inclusion of congenital causes such as spina bifida, as well as a greater proportion in the oldest group, 65 and over (28.6% compared with 22.8% for TSCI), most likely due to the inclusion of degenerative and other age-related causes. The TSCI group was more concentrated in the middle age group, 35-64 (68.4% compared with 58.7% for NTSCI).

Table 16. Demographic Characteristics of RSC: TSCI and NTSCI Cases

TSCI NTSCI All n=57 n=67 n=124 Sex Female, n (%) 16 (28.1) 34 (50.7) 50 (39.7) Male, n (%) 41 (71.9) 33 (49.3) 76 (60.3) Age Mean age, years (sd) 55.6 (13.8) 54.3 (19.2) 55.1 (17.0) Age 14-34, n (%) 5 (8.8) 11 (16.4) 16 (12.7) Age 35-64, n (%) 39 (68.4) 34 (50.7) 74 (58.7) Age >=65, n (%) 13 (22.8) 22 (32.8) 36 (28.6)

55 4.2.4 EMR Documentation

The EMRALD database contains entries that are generated by health care transactions of various kinds, including among others clinical encounters, prescriptions, and diagnostic procedures. Although EMRALD is not explicitly designed to track and measure health care utilization in a broad sense (beyond the primary care practice which maintains the EMR), this association between health care transactions and records in the EMR means that the volume of EMR documentation (i.e. number of entries, of different types) can be considered a proxy indicator of health care utilization, and can be used to examine differences in utilization between populations.

Overall, patients with SCI as identified in the RSC had approximately 2.5 times as many entries in their EMR as the eligible EMRALD population (both groups had a similar mean time in the EMR). This ratio is similar across all types of entries except referrals (where it is 1.6 times). This observation is in line with the fact that people with SCI are known to be high health care users (S J T Guilcher et al., 2010).

Table 17. EMR Documentation of RSC Compared with EMRALD Mean number of RSC EMRALD EMR entries n=126 n=213,887 All entries 453.9 (394.9) 180.3 (193.0) PNs (sd) 120.3 (126.7) 51.1 (50.5) CONS (sd) 79.3 (75.6) 31.1 (33.5) Prescriptions (sd) 158.0 (197.2) 58.9 (90.8) Referrals (sd) 17.3 (16.1) 10.8 (12.3) Diagnostics (sd) 79.3 (75.6) 31.1 (33.5) Mean time in EMR, years (sd) 7.2 (3.8) 6.7 (3.6)

4.2.5 Primary Care Utilization

Primary care encounters are recorded in EMRALD, providing a basic measure of primary health care utilization. The mean number of primary care encounters per year was calculated separately for TSCI and NTSCI cases in the RSC, and for the EMRALD study cohort (see Figure 11). Both groups in the RSC had a much higher mean number of annual primary care visits, with TSCI patients slightly higher than NTSCI patients.

Primary Care Utilization: encounters per year 80 75 Cohorts Encounters/yr 70 n Mean S. D. 63.2 EMRald 213,887 2.8 2.5 60 56.7 TSCI 57 5.1 4.3 NTSCI 67 4.0 3.5 50

40 29.9 30 24.6 % of total cohort 20 15.1 13.4 10.5 10 7.5 0 0 2.1 0.1 1.8 0 0 0 1-4 5-9 10-19 >19 Encounters per year

Figure 11: Primary care utilization: encounters per year

4.3 Stage 3—Algorithm Development

4.3.1 Keyword Analysis

The case identification algorithms developed made use of different types of information in the EMR database. The CPP component returned charts with a keyword match in the CPP (specifically in the problem list and medical history portions of the CPP). Similarly, the CONS/PN component returned charts with a keyword match in the CONS or PN entries.

In order to determine which keywords to use in these algorithm components, the occurrence of keywords in the RSC was analyzed.

Because a given keyword (or keyword phrase) can occur at most once in the CPP, it was possible to calculate the sensitivity and PPV of each CPP keyword. This information, shown in Table 18, was used to construct keyword lists for use in case identification algorithms. The first list of CPP keyword terms, called CPP1, contains eight keywords all of which have a PPV of 100%, meaning that these keywords appeared only in the CPP of true positive cases (zero false positives). The next list, CPP2, added the next four terms in descending order of PPV, with PPVs between 71% and 92%. The third list, CPP3, added another 11 terms with PPVs between 24% and 67%.

57 Table 18. CPP Keyword Occurrences Term TP FP TN FN Sens PPV List spinal cord injury 21 0 677 105 17% 100% neurogenic bowel 4 0 677 122 3% 100% tetraplegi* 4 0 677 122 3% 100% central cord syndrome 3 0 677 123 2% 100% CPP1 brown sequard 3 0 677 123 2% 100% meningocele 2 0 677 124 2% 100% tetrapare* 1 0 677 125 1% 100% quadrapare* 1 0 677 125 1% 100% paraplegi* 22 2 675 104 17% 92% quadriplegi* 10 2 675 116 8% 83% CPP2 cervical myelopathy 4 1 676 122 3% 80% quadraplegi* 5 2 675 121 4% 71% myelopathy 14 7 670 112 11% 67% quadripara* 1 1 676 125 1% 50% cauda equina 7 8 669 119 6% 47% neurogenic bladder 25 29 648 101 20% 46% transverse myelitis 5 6 671 121 4% 45% tethered cord 2 3 674 124 2% 40% CPP3 myelitis NOT osteomyelitis 11 17 660 115 9% 39% arnold chiari 2 4 673 124 2% 33% epidural abscess 1 2 675 125 1% 33% syringomyelia 1 3 674 125 1% 25% spina bifida 17 54 623 109 13% 24% Abbreviations: TP, true positive; FP, false positive; TN, true negative; FN, false negative

In the case of keywords occurring in the CONS and PN entries, it was possible for a given keyword to occur multiple times in the same chart, making it infeasible to use the same analysis (sensitivity and PPV of keywords) as with CPP keywords. Rather, the total numbers of keyword occurrences were tabulated, and the occurrences in cases were compared with those in non-cases to determine the value of the keyword for case identification (Table 19).

58 Table 19. CONS/PN Keyword Occurrences

Term Def No Poss Total Def/Total No/Total tetraplegi* 77 1 6 84 92% 1% neurogenic bowel 59 2 6 67 88% 3% spinal cord injury 477 59 63 599 80% 10% central cord syndrome 15 0 4 19 79% 0% quadraplegi* 123 32 0 155 79% 21% tetrapare* 3 1 0 4 75% 25% paraplegi* 572 159 43 774 74% 21% quadriplegi* 273 92 29 394 69% 23% neurogenic bladder 242 455 108 805 30% 57% quadripare* 11 24 6 41 27% 59% parapare* 6 22 8 36 17% 61% brown sequard 4 2 22 28 14% 7% quadrapare* 0 0 5 5 0% 0% hemiparaplegia 0 0 0 0 n/a n/a Mean keyword occurrences 23.87 1.32 3.61 3.75 Mean unique keywords 2.87 0.47 1.10 0.77

4.3.2 Algorithm Component Evaluation

The performance of case identification algorithm components was evaluated by applying the algorithm component to the 803 possible cases that were identified by the keyword search strategy and manually reviewed. Using the RSC as the criterion, the confusion matrix values (true positives, true negatives, false positives, false negatives) were tabulated for each algorithm component, and the sensitivity, PPV, and F-score were calculated.

Algorithms using the three CPP word lists (Table 20) illustrate the tradeoff between sensitivity and PPV: the CPP1 algorithm, which uses the 8 CPP terms with 100% PPV, also has 100% PPV and 0 false positives but sensitivity of only 31.7%; whereas CPP3, which includes 31 terms, captures 79 false positives lowering PPV to 54.6% but also captures 55 additional true positives increasing sensitivity to 75.4%. The best overall CPP algorithm is CPP2 with an F-score of 67.3%. This component maintains an excellent PPV of 94.3% but the sensitivity of 52.4% is inadequate. However, by combining this component with others a synergistic improvement in performance may be obtained, increasing true positives and hence sensitivity without commensurately increasing false negatives and diminishing PPV.

59 Table 20. CPP Algorithm Component Performance Component TP TN FN FP Sensitivity PPV F-score CPP1 40 677 86 0 31.7 100 48.1 CPP2 66 673 60 4 52.4 94.3 67.4 CPP3 95 598 31 79 75.4 54.6 63.3

Algorithm components using keywords found in CONS and PN (Table 21) demonstrated a similar tradeoff between sensitivity and PPV. Overall, the best-performing of these achieved F- scores comparable to the best-performing CPP algorithm components, which is a significant result since this method (using keywords in CONS and PNs in case identification algorithms) has not been documented in previous studies. Two overall patterns are observable in these results: as the number of keyword occurrences required diminishes, the sensitivity increases and the PPV diminishes; and for a given minimum number of keyword “occurrences” (e.g. 3), the algorithm component specifying “unique” occurrences (unique>=’3’) will have lower sensitivity and higher PPV than the component specifying “any” keyword occurrence (any>=’3’). The two highest-performing components both use any keyword occurrence, and the highest-performing one is also the least restrictive, requiring at least two occurrences of any of the CONS/PN keywords. These components could be combined with others to further improve performance.

Table 21. CONS/PN Algorithm Component Performance Component TP TN FN FP Sensitivity PPV F-score any >='2' 78 652 48 25 61.9 75.7 68.1 any >='3' 68 669 58 8 54.0 89.5 67.4 unique >='2' 63 668 63 9 50.0 87.5 63.6 any >='4' 58 672 68 5 46.0 92.1 61.4 any >='5' 48 674 78 3 38.1 94.1 54.2 unique >='3' 28 675 98 2 22.2 93.3 35.9 unique >='4' 14 677 112 0 11.1 100 20.0 unique >='5' 6 677 120 0 4.8 100 9.2

In addition to CPP keywords, previous EMR case identification studies have used medication names in prescriptions and disease classification codes (such as ICD-9 or ICD-10) to identify cases of a condition of interest. Unlike conditions such as diabetes or multiple sclerosis, where there exist medications that are widely and exclusively used within that population, there are no medications that are known to be good indicators of SCI caseness; with respect to codes, prior work (see section 2.4.5) has shown that they are problematic, particularly ICD-9 (the version used in EMRALD), when used to identify cases of TSCI, and work using codes to identify cases of NTSCI is in early stages and has only been done using ICD-10.

60 Nonetheless, the performance of algorithm components using medication names and ICD-9 codes was evaluated to determine if these could be of use in combination with other components (Table 22). The medication name algorithm component performed poorly, as expected, with sensitivity of only 29.4% and PPV of 47.4%. Occurrences of ICD-9 codes were analyzed using the same method as in other studies: looking for single or multiple (“x2,” “x3,” etc) occurrences, and also looking for multiple occurrences in a single year (“x2in1;” this approach is important when identifying conditions that can change or where repeated documentation can increase certainty, but may be less relevant in a condition like SCI). The only ICD-9 code that identified a significant number of cases was 806 (“fracture of vertebral column with spinal cord injury”), with 20 true positives and 0 false positives. Since there were no false positives, there was no improvement in performance from requiring multiple code occurrences. The other code that identified SCI cases was 344, with 3 true positives and 0 false positives; however, all 3 of these cases also used the 806 code so there was no incremental value in using both over using 806 alone. For the sake of parsimony ICD-9 code 344 was dismissed in further analysis but it should be noted that it would be prudent to re-introduce code 344 when working with a different sample. ICD-9 code 952 did not identify any true positive cases and was also dismissed in further analysis.

Table 22. Medication and ICD-9 Code Algorithm Component Performance Component TP TN FN FP Sensitivity PPV F-score Medication name 37 636 89 41 29.4 47.4 36.3 344 3 677 123 0 2.4 100 4.7 344x2 1 677 125 0 0.8 100 1.6 806 20 677 106 0 15.9 100 27.4 806x2 16 677 110 0 12.7 100 22.5 806x3 12 677 114 0 9.5 100 17.4 806x2in1 11 677 115 0 8.7 100 16.0 806x3in1 8 677 118 0 6.3 100 11.9 952 0 677 126 0 0.0 0.0 0.0

4.3.3 Case Identification Algorithm Evaluation

Using various permutations of algorithm components that search for keywords in the CPP and CONS/PNs, medication names, and ICD-9 codes, a range of 125 candidate algorithms was created and their performance evaluated (see Appendix E for a list of all algorithms).

61 Using F-score as an overall measure of algorithm performance, the highest performing algorithm identifies cases that have a CPP keyword on the CPP2 list, or an occurrence of ICD-9 code 806, or 3 or more keywords in CONS/PN (Table 23). This algorithm has an F-score of 79.1%, sensitivity of 70.6%, and PPV of 89.9% (there is no definitive cutoff for performance on these metrics, but performance >90% can be considered excellent, >80% good, and >70% adequate). Of note, although the CONS/PN algorithm component specifying “any>=2” outperforms “any>=3” with respect to F-score when considered in isolation, this relationship is reversed when other components are added. The synergistic effect that is possible when combining components can be seen by comparing the CONS/PN component “any>=3,” which has sensitivity of 54.0% and 89.5%, with the optimal algorithm which combines this component with CPP2 and code 806. The combined algorithm increases the sensitivity to 70.6% (16.6 points) without diminishing the PPV at all; in fact, the latter increases slightly, to 89.9%.

Table 23. Algorithms with Optimal F-score Algorithm Sensitivity % PPV % F-score % CPP2 or 806 or any>='3' 70.6 89.9 79.1 CPP2 or 806 or any>='4' 67.5 92.4 78.0 CPP2 or any>='3' 69.0 89.7 78.0 CPP2 or 806 or unique>='2' 68.3 88.7 77.2 CPP2 or any>='4' 65.9 92.2 76.9 CPP1 or 806 or any>='3' 65.9 91.2 76.5 CPP2 or unique>='2' 66.7 88.4 76.0 CPP1 or any>='3' 64.3 91.0 75.4 CPP2 or 806 or any>='2' 73.0 77.3 75.1 CPP1 or 806 or any>='4' 61.9 94.0 74.6 CPP2 or any>='2' 71.4 76.9 74.0 CPP2 or 806 or any>='5' 61.1 92.8 73.7

Algorithms can be ranked by F-score as an overall measure of performance; however, the way in which an algorithm’s performance is measured needs to take into account the purpose for which the algorithm is used. In a situation where it is crucial to ensure those selected are true cases (e.g. a clinical trial), an algorithm with optimal PPV may be chosen (Table 24). In order to have absolute certainty in case identification (i.e. 0 false positives and 100% PPV) the maximum sensitivity is 41.3%. However, by relaxing the PPV criterion to 94%, a sensitivity of 61.9% can be achieved. These algorithms with higher PPV have more exacting criteria, using the more restrictive CPP1 word list and requiring a greater number of CONS/PN keyword occurrences.

62 Table 24. Algorithms with Optimal PPV Algorithm Sensitivity % PPV % F-score % CPP1 or 806 or unique>='4' 41.3 100.0 58.5 CPP1 or 806 or unique>='3' 47.6 96.8 63.8 CPP1 or 806 or any>='5' 54.8 95.8 69.7 CPP2 or 806 or unique>='4' 57.9 94.8 71.9 CPP1 or 806 or any>='4' 61.9 94.0 74.6

Conversely, in a situation where it is important to maximize inclusiveness, an algorithm with optimal sensitivity may be chosen (Table 25). The highest sensitivity attainable is 88.1%, with a PPV of 52.6%, meaning that a given patient identified by this algorithm has a slightly better than 50% chance of being a true case. By relaxing the sensitivity criterion to 76.2%, a PPV of 65.8% can be achieved. These algorithms with higher sensitivity have more inclusive criteria, using the less restrictive CPP3 word list and, in some cases, medication names, and requiring a smaller number of CONS/PN keyword occurrences.

Table 25. Algorithms with Optimal Sensitivity Algorithm Sensitivity % PPV % F-score % CPP3 or 806 or any>='2' 88.1 52.6 65.9 CPP3 or 806 or any>='3' 86.5 56.2 68.1 CPP3 or 806 or any>='4' 84.9 56.6 67.9 CPP2 or Medication or 806 or any>='2' 77.8 59.8 67.6 CPP2 or Medication or 806 or any>='3' 76.2 65.8 70.6

Figure 12 illustrates the tradeoff between sensitivity and PPV, showing a number of algorithms ranked in decreasing order of sensitivity. The overall optimal algorithm is identified with a box and is placed in the bottom half in this ranking.

Figure 12: Algorithms ranked by sensitivity

The tradeoff between sensitivity (or recall) and PPV (or precision) for different classifiers (or algorithms) can be shown graphically by a precision-recall curve, which, for each algorithm, shows sensitivity/recall on the x-axis and PPV/precision on the y-axis (Figure 13).

100

90 Optimal algorithm: CPP2+806+any>=3 80

50 PPV (%)

0 0 10 20 30 40 50 60 70 80 90 100 Sensitivity (%)

Figure 13: Precision-recall curve for case identification algorithms

Develop Rounds 2-3: Round 1: Iterations to Final keyword keyword search Preliminary chart identify & validate search strategy review (143 terms) strategy potential keywords

3,457 potential cases, 803 randomly sampled for review

First review & Conduct chart classification, Second review & Third review, final review capture of relevant validation adjudication chart details

Reference standard: 126 definite cases

Develop within- Other search Keyword criteria Testing of EMR search analysis (billing codes, alternatives algorithm prescriptions)

Optimal algorithm: CPP2 + 806 + any>=3 F-score: 0.791, Sensitivity: 0.706, PPV: 0.899

Figure 14: Results overview: results by process step

Chapter 5 Discussion

Discussion 5.1 Statement of Principal Findings

5.1.1 Summary This study has demonstrated that it is possible to reliably identify cases of SCI in a primary care electronic medical record database using a detailed case definition (1) a comprehensive keyword search strategy, (2), and a rigorous manual chart review process (3). The reference standard cohort (RSC) of validated cases of SCI identified in the EMRALD database using this method was similar demographically to other cohort studies combining both TSCI and NTSCI populations (4). With respect to the development of algorithms for use in the automated identification of cases of SCI in EMRALD (and potentially in other similar EMR databases), the precise methods used in previous studies proved infeasible given the specific impairments associated with SCI and types of evidence able to identify SCI (5). However, it was possible to construct satisfactory case identification algorithms by making more extensive use of free-text keyword searching (text mining) in the unstructured portions of the EMRALD database (6). The best algorithms constructed using this method perform comparably to those documented in previous case-finding studies using EMRALD (7). Comparing the size of the RSC to the reference population provides an estimate of the cohort prevalence of SCI (8). This estimate contributes to the relatively few documented estimates of SCI prevalence in Canada (9).

5.1.2 Similarities and Differences with Respect to Other Studies

1. Case Definition The reliable identification of SCI cases required a detailed case definition, based on the literature and operationalized in a chart review protocol. This study examined the literature providing definitions of SCI and developed a comprehensive definition of TSCI and NTSCI, specifying type and duration of impairment and lesion site. This case definition is maximally comprehensive compared to other definitions reviewed and, unlike previous case definitions, includes NTSCI.

67 2. Keyword Search Strategy A comprehensive search strategy of 143 terms was required to flag all possible cases of SCI for manual review. Given the low prevalence of SCI in the population, manual review of a random sample of charts was not feasible (with a goal of >100 validated cases to form a reference standard and using prevalence estimates from the literature, this would require review of at least 5,000 charts). Rather, a group of possible cases was identified using a keyword search strategy and a randomly selected group of these possible cases was then manually reviewed. The incremental value of the work required to develop a comprehensive search strategy can be measured by the sensitivity of a naïve keyword search within the RSC using the phrase “spinal cord injury,” which is 16.7% (CI: 10.6%-24.3%), indicating that over 80% of the SCI cases identified in this study would be missed by such a strategy. Furthermore, even the use of a multiple-term search strategy, as used in practice by a family physician with a leading clinic specialized in serving the SCI community, would only improve sensitivity to 42.1% (CI: 33.3%- 51.2%), meaning that more than half of the cases of SCI would be missed. In particular, it should be noted that use of either of these naïve search strategies would particularly under-represent NTSCI (since they include terms that are traditionally used to designate TSCI and omit the etiologies by which cases of NTSCI are often designated). The methods used in this study are similar to those previously documented in other studies that use EMRALD for case identification in a number of chronic conditions (see section 2.4.4 and Table 4, above). Within these studies, there were two different overall approaches to case identification. In conditions that are relatively more prevalent, random sampling is a feasible approach because a sufficient number of cases can be gleaned from a sample size that can be manually reviewed (references). In less prevalent conditions, the sample size required for a random sample to yield a sufficiently large RSC (at least 100 definite cases) would be too large for manual review. In these conditions, case identification involved a preliminary keyword search using a comprehensive search strategy designed to identify all possible cases of the condition of interest, creating a smaller set of potential cases which can be manually reviewed.

When compared with other studies that followed a two-stage case identification process (Table 26), this study was similar (i.e. in the same order of magnitude) with respect to the size of the study cohort, the number of charts manually reviewed, and the number of cases identified.

68 Table 26. Two-Stage Case Finding Studies Using EMRALD

Reference Population Study cohort Reviewed Cases Identified Ivers (2011) Ischemic heart disease 19,376 969 87 Butt (2014) Parkinsonism 73,003 2,319 231 def/294 poss Breiner (2015) Myasthenia gravis 123,997 645 49 def/256 poss Krysko (2015) Multiple sclerosis 73,003 943 247 This study SCI 213,887 803 126

3. Chart Review Process The case definition was translated into a chart review process by specifying the types of evidence required to fulfill the criteria of the case definition, and by implementing this process with a small team to review charts of potential cases and find definitive evidence of SCI. This method follows other EMRALD case-finding studies in validating the chart review process with a second reviewer, who reviewed a 10% sample of cases reviewed by the first reviewer in addition to reviewing all possible cases not classified by the first reviewer. Particularly complicated and borderline cases not classified by the first two reviewers were referred to a senior clinical authority for adjudication.

4. Reference Standard Cohort of SCI Cases The RSC identified in this study, when compared with other studies that include a cross-sectional cohort of both TSCI and NTSCI (see Table 27), is similarly composed with respect to gender. Only 28.1% of TSCI cases in the RSC are female, compared with 50.7% of NTSCI cases. This compares with an average across 9 studies of 23.2% female for TSCI and 46.8% female for NTSCI. With respect to age, the RSC identified in this study showed a similar average age overall (55.1 years for the RSC vs 52.3 years across 8 studies) but a slightly different pattern within subgroups: the pattern observed across other studies is that the TSCI cohort is younger on average than the NTSCI cohort (43.7 years for TSCI vs 53.6 years for NTSCI, across 8 studies); whereas in the RSC the TSCI cohort is slightly older than the NTSCI cohort (55.6 years vs 54.3 years). Of note, the age of the NTSCI group in the RSC was more widely dispersed (SD of 19.2 years vs 13.8 years for TSCI, reflecting a bimodal distribution with a younger group of congenital cases (particularly spina bifida) and an older group of degenerative cases.

69 Table 27. Studies including TSCI and NTSCI Cohorts

TSCI NTSCI Age Female Age Female Study Country N N (mean, SD) % (mean, SD) % Gedde (2019) Norway 102 49, 19.8 28 72 52, 18.6 42 Halvorsen (2019) Norway 349 47, 19 24 225 55, 17 41 Guilcher (2015)* Canada 1,537 51.6, 21.3 25.8 2,830 57, 19.6 43.3 Milicevic (2012) Serbia 279 40.2, 16 17.6 162 55.5, 13.8 43.2 New (2011) Australia 1,361 46** 28.4 2,241 67** 47.5 Scivoletto (2011) Italy 144 19.4 236 45.8 Guilcher (2010) Canada 560 46.9, 17.3 24.6 1,002 61.6, 15.8 47.8 Gupta (2008) India 38 32.9, 8.0 10.5 38 31.1, 14.4 57.9 Ones (2007) Turkey 131 35.8, 15.4 30.5 63 49.9, 16.4 52.4

Overall, the similarity in demographic composition between the RSC and similar cohorts in the literature lends credence to the results of the case identification process documented in this study.

5. Algorithm Development Although the overall process used in this study follows closely the methods used in previous studies, the particular characteristics of SCI necessitated a more extensive use of certain types of information. Unlike many other chronic conditions where diagnosis or billing codes (using a standard terminology like ICD-9) can be used, the relevant codes for SCI were found to be unreliable (with a sensitivity of only 15.9%). Furthermore, prescriptions and lab test values were similarly unhelpful, whereas they have been shown to be reliable in identifying cases of MS and diabetes, the latter even in the absence of diagnosis codes (Krysko et al., 2015; Tu et al., 2011).

Previous studies have shown that algorithm performance improves with the inclusion of additional data elements (W. Q. Wei et al., 2016). In a systematic review of case identification algorithms used in primary care EMR databases, diagnostic codes were the most commonly used data element across all conditions. However, previous studies have shown the unreliability of diagnostic codes for the identification of TSCI (Hagen et al., 2008; St. Germaine-Smith et al., 2012) and this is particularly the case with ICD-9 codes (the version used in EMRALD). Furthermore, use of ICD-9 codes by physicians using EMRALD was low, in that only 15.9% (20 of 126) of the RSC included one of the ICD-9 codes used to identify SCI.

In certain conditions such as diabetes where several case identification algorithms have been described in the literature, the inclusion of additional elements, such as medications or laboratory

70 values, was shown to improve algorithm performance; in a study of case identification algorithm performance for multiple sclerosis in the EMRALD database, medications were successfully used in addition to codes. However, neither of these data elements are useful for case identification in SCI. Among medications that are widely used in the SCI population, most are very unspecific (e.g. laxatives). With respect to the three medications identified as being both relatively widely used within and specific to SCI (Ditropan, Baclofen, Dantrium), only 29.4% of the RSC cases included a prescription for one of these medications; furthermore, 89 non-cases were identified in the 803-patient REVIEW COHORT (false positives) using these medications, for a PPV of 47.4%. With respect to laboratory tests, there is no test value that is indicative of SCI (only an imaging study appropriately interpreted by a clinical expert can serve to diagnose TSCI, and imaging is of limited use in the diagnosis of NTSCI). Unlike diabetes and multiple sclerosis, therefore, case identification in SCI cannot be improved by using medications and laboratory results.

Notwithstanding the inadequacy of codes, medications, and laboratory results to identify cases of SCI in EMRALD, preliminary review showed that physicians using EMRALD tended to capture information concerning diagnosis of SCI in the problem list field of the CPP using one of a number of potential terms. These terms were identified and their reliability (sensitivity and PPV) was analyzed in order to evaluate which should be used as free-text keyword search terms in a case identification algorithm. In effect, these keyword terms were used in the CPP in place of codes. Performance was adequate, illustrating a typical tradeoff between sensitivity and PPV (see Table 20, above).

Although the PPV of the CPP2 algorithm was excellent at 94.3%, sensitivity was poor at 52.4%, meaning that this algorithm would miss almost half of true cases. However, increasing the sensitivity to a more acceptable 75.4% reduced PPV to 54.6%, meaning that almost half of the cases identified by the algorithm would be false positives.

In order to improve algorithm performance, additional unstructured (free text) elements of the EMRALD database were used, in particular the progress notes entered by the primary care physician and the consult letters (including reports from specialists and admission and discharge summaries from hospitals). Occurrences in these fields of highly specific terms such as “tetraplegia,” “neurogenic bowel,” and “spinal cord injury” proved to be a valuable addition to the contents of the CPP for the purpose of case identification. This approach leveraged the

71 unique inclusion in EMRALD of these substantial free-text elements, unlike other EMR databases (Jaakkimainen et al., 2018).

6. Use of Text Mining for Automated Case Identification

Due to the unsuitability of ICD-9 codes, medications, and laboratory values to identify cases of SCI, it proved necessary to make more extensive use of free text searching within the extensive unstructured content of physician notes, reports, and consultation letters. The process used in this study to construct case identification algorithms is similar to that used in other studies using EMRALD for case identification, with one significant innovation. Due to the extensive use made of free text searching within unstructured notes, these results were included in an algorithm component. This component, which specified a minimum number of keyword hits from a list of 13 keywords, permitted an improvement in sensitivity of 16.6% (from 52.4% to 69%). With respect to previous case identification studies in SCI (see section 2.4.5), this is the first to use EMRs as a data source, and the first to use information other than diagnostic codes (such as ICD-9 or ICD-10).

7. Algorithm Performance The algorithms developed in this study performed similarly to those developed to identify other disease populations using the EMRALD database. The sensitivity of the optimal algorithm (70.6%) was lower than that seen in some other studies, and this can be attributed to the difficulty of capturing every instance of a diagnosis that can be designated in many different ways, particularly when NTSCI is included. The PPV of the optimal algorithm (89.9%), however, compared very favorably with previous EMRALD studies. When compared with a broader sample of 40 case-finding studies using a variety of primary care databases (McBrien et al., 2018), the sensitivity is well within the range observed and the PPV is at the higher end of the range of those previously reported.

8. Prevalence Estimate of SCI Although this study was not designed to measure the prevalence of SCI in the Ontario population, there is a prevalence estimate implicit in the size of the RSC compared with the reference population (in this case the EMRALD database). With 126 cases of SCI identified from a sample of 48,000 charts, this implies a cohort prevalence of 262.5 per 100,000. Although the literature shows a wide range of prevalence estimates for both TSCI and NTSCI (and a dearth

72 overall of reliable estimates, particularly for NTSCI), the most relevant study is the Canadian estimate coving both TSCI and NTSCI published by Noonan in 2012, which gives a prevalence of 252.5 per 100,000. The agreement of these estimates is mutually corroborating, however a larger sample of good quality prevalence estimates is required to validate this finding.

9. Prevalence Estimate: Contribution of this Study Because the EMRALD database only includes persons who receive care from a family physician, it necessarily omits a portion of the population (in this sense it is actually measuring “contact prevalence,” the prevalence of those who interact with the health care system via a family physician). People living with SCI could be assumed to be more likely than average to seek the care of a family physician, in which case the contact prevalence would be greater than the true prevalence. Set against this drawback of using EMRALD to estimate prevalence is a considerable advantage in that EMRALD offers a sample that is much closer to being population-based than many other studies of SCI prevalence. Unlike studies that use hospital discharge records to identify cases or those that use incidence figures to calculate prevalence (such as the Noonan study), EMRALD covers a large sample that has shown to be fairly representative of the overall population. The improved capture of this method vs hospital records (especially of milder cases that are managed in primary care) may account for the fact that the EMRALD-derived estimate is substantially greater than the average of hospital-based estimates (118.8 per 100,000 for TSCI in EMRALD vs 38.7 per 100,000 across 6 studies). At the same time, EMRALD offers a crucial advantage over most other population-based prevalence estimates of SCI, which have used surveys and relied on self-report; EMRALD uses clinical records documenting an authoritative diagnosis and is therefore more reliable than self-report. The improved reliability of this method vs surveys may account for the fact that the EMRALD- derived estimate is lower than the average of population-based estimates (118.8 per 100,000 for TSCI in EMRALD vs 207.0 per 100,000 across 8 studies). Furthermore, it is reasonable to assume that a meaningful portion of the total SCI population does not have a family physician (particularly marginalized persons, homeless, indigenous, comorbid TBI) and would therefore not be adequately represented in EMRALD.

5.2 Strengths and Limitations

An important strength of this study is the use of an EMR database (EMRALD) that includes a large number of patients and has been shown to be fairly representative of the overall population.

73 Furthermore, EMRALD is unique among EMR databases in that it includes a large amount of unstructured free-text information (from progress notes, reports, consultation letters, etc.), and this information proved particularly valuable for case identification in SCI; given the findings of this study, it is doubtful that cases of SCI could be comprehensively identified without this free- text information, using only diagnostic codes. Another strength is the inclusive and evidence- based case definition of SCI, which is based on a detailed review of the literature over time on definitions of SCI, and which includes both TSCI and NTSCI. Finally, the rigorous and detailed chart review method helped ensure that cases were identified comprehensively and reliably.

There are several important limitations to this thesis. First, the emphasis on comprehensiveness in the case definition may result in the inclusion of some patients (for example those involving only temporary impairment lasting >48 hours) that would not be considered cases of chronic SCI according to some definitions. The inclusion of these cases would tend to increase the reported prevalence of SCI. (It is therefore the lifetime prevalence of chronic SCI that is being measured in this study). Second, the use of only 2 reviewers (one principal reviewer and one secondary reviewer who validated a 10% sample of the first reviewer’s work and consulted on difficult cases) imposed a limitation on the number of charts that could feasibly be manually reviewed. Third, the preliminary chart review (stage 1) used to develop the keyword search was done while the review process itself was still in development; however the risk of omitting potentially valuable keyword terms was mitigated by including a large number of terms in the final keyword search (choosing to err on the side of comprehensiveness). Fourth, there is insufficient evidence in EMR to reliably determine level, severity (ASIA), and completeness of SCI. Fifth, the date of onset of SCI is not clearly recorded in EMR, making identification of incident cases infeasible.

Finally, comparison of the population in the EMRALD database to the overall Ontario population has shown some differences (the EMRALD population is more female and more urban than the overall population) and these may result in an RSC that is not representative of the Ontario SCI population in these respects, and in a prevalence estimate that is inaccurate. Although family physicians in EMRALD come from across Ontario, rural physicians and Canadian-trained physicians are over-represented (Jaakkimainen et al., 2018). This work was only performed on a convenience sample of family physicians in Ontario. Findings may not be generalizable to the rest of Canada, let alone the world, but can be used as a point of comparison for other studies.

74 5.3 Meaning and Implications

This study has demonstrated the particular challenges involved in identifying cases of spinal cord injury. In the first place, the definition of spinal cord injury is problematic and needs to be clarified in order to serve as a basis for case identification. Furthermore, the types of information (diagnostic codes, medications, and test values) typically used for case identification are not effective in SCI. Therefore, this study made more extensive use of text mining (Ford, Carroll, Smith, Scott, & Cassell, 2016b), searching for keywords, in the free text contained in the unstructured portions of the EMR. Several different kinds of keywords or keyword phrases proved useful in case identification: terms that designate impairment (“paraplegia”), SCI subtypes or syndromes (“Brown-Sequard”), related etiologies (“meningocele”), and secondary consequences (“neurogenic bowel”). It should be noted that the diagnostic term “spinal cord injury” occurred in only a small proportion (<20%) of the SCI cases identified in this study. The novel work using free text keywords documented in this study may be useful to other researchers looking to identify cases of SCI in free text or unstructured data sources including EMRs.

SCI is known to have large impact on patients, who become high-volume users of the health care system. However, the SCI literature is focused on acute and rehabilitative management, and there is a need to describe the chronic SCI population living in the community, with basic epidemiological and health services research (Rowan et al., 2019). Lack of longitudinal understanding means that health outcomes and needs of this population are not well-understood and, consequently, it is difficult to evaluate and optimize their health care. This need for better understanding applies in particular to NTSCI population, and is vital given the evolving picture of the demographics of SCI.

The prevalence estimate implicit in the size of the RSC provides a useful contribution to the literature on SCI prevalence. Although subject to limitations with respect to the use of an EMR database to calculate prevalence (the database may not be representative of the overall population, and inclusion is biased by use/nonuse of primary care), this estimate is nonetheless population-based and as such superior to many prior estimates of SCI prevalence that use hospital discharge records and registries or models that combine incidence and disease duration. And compared with the relatively few population-based estimates of SCI prevalence, which use a self-report survey method for data collection, this study uses information gathered and reported by medical experts. As such, it is significant that this study’s prevalence estimate is very close to

75 the best comparator, the model-based estimate of Noonan (2012). Given the very wide range of estimates found in the literature, this closeness is reassuring and mutually corroborating.

The demographic characteristics of the RSC confirm the picture of an evolving SCI population that can be seen in a number of recent studies. Once limited to traumatic injuries primarily affecting young males, SCI now includes roughly equal numbers of non-traumatic and traumatic injuries and affects an older population that is closer to being evenly split between females and males (although traumatic injuries still disproportionately impact males). An understanding of these trends is essential for planning service delivery and developing health policy in SCI.

5.4 Future Research Directions

Going forward, this work can help address the longstanding research gap with respect to the understanding of longitudinal patterns of SCI (as reviewed in 2.3, above). The cohort identified in EMRALD can be linked with other databases maintained by IC/ES including administrative databases containing information on physician billings, hospital discharges, and prescriptions in order to develop an understanding of the health care utilization of the SCI population across the entire healthcare system (not just primary care). Furthermore, the EMRALD cohort can be used to validate an administrative data algorithm which could then be used to identify SCI patients within the total Ontario population (not just those with records in EMRALD). Machine learning and artificial intelligence methods including natural language processing could be employed to evaluate their effectiveness for case identification. Comparison between databases (using capture-recapture methods, for example (Cameron, Coppell, Fletcher, & Sharples, 2012)) can validate the case-finding method of this study and the cohort prevalence estimate. Further, linkage with other databases containing information about, for example, social assistance payments, can elucidate the socioeconomic status of this population and the importance of the social determinants of health. Finally, the rich longitudinal data afforded by combining multiple databases can be used to study episodes of care, treatment patterns and outcomes over time in the Ontario SCI population.

(Acton et al., 1993; Thurman et al., 1994). (Hagen et al., 2008; Johnson et al., 1997). (Hagen et al., 2008; Kraus et al., 1975).

76 References

Acton, P., Farley, T., Freni, L., Ilegbodu, V., Sniezek, J., Wohlleb, J., … J.E., S. (1993). Traumatic spinal cord injury in Arkansas, 1980 to 1989. Archives of Physical Medicine and Rehabilitation, 74(10), 1035–1040. Bárbara-Bataller, E., Méndez-Suárez, J. L., Alemán-Sánchez, C., Sánchez-Enríquez, J., & Sosa- Henríquez, M. (2018). Change in the profile of traumatic spinal cord injury over 15 years in Spain. Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine, 26(1), 1– 8. https://doi.org/10.1186/s13049-018-0491-4 Barnado, A., Casey, C., Carroll, R. J., Wheless, L., Denny, J. C., & Crofford, L. J. (2017). Developing Electronic Health Record Algorithms that Accurately Identify Patients with Systemic Lupus Erythematosus. Arthritis Care Res, 69(5), 687–693. https://doi.org/10.1002/acr.22989 Bickenbach, J. E. (Ed.). (2013). International perspectives on spinal cord injury. Geneva. https://doi.org/10.1007/978-1-4939-7880-9_11 Biermans, M. C. J., Spreeuwenberg, P., Verheij, R. A., de Bakker, D. H., de Vries Robbe, P. F., & Zielhuis, G. A. (2009). Striking trends in the incidence of health problems in the Netherlands (2002-05). Findings from a new strategy for surveillance in general practice. The European Journal of Public Health, 19(3), 290–296. https://doi.org/10.1093/eurpub/ckn130 Birtwhistle, R., & Williamson, T. (2015). Primary care electronic medical records: a new data source for research in Canada. CMAJ: Canadian Medical Association Journal = Journal De L’association Medicale Canadienne, 187(4), 239–240. https://doi.org/10.1503/cmaj.140473 Bjørnshave Noe, B., Mikkelsen, E. M., Hansen, R. M., Thygesen, M., & Hagen, E. M. (2015). Incidence of traumatic spinal cord injury in Denmark, 1990-2012: A hospital-based study. Spinal Cord. https://doi.org/10.1038/sc.2014.181 Cameron, C. M., Coppell, K. J., Fletcher, D. J., & Sharples, K. J. (2012). Capture-recapture using multiple data sources: Estimating the prevalence of diabetes. Australian and New Zealand Journal of Public Health, 36(3), 223–228. https://doi.org/10.1111/j.1753- 6405.2012.00868.x Chen, Y., He, Y., & DeVivo, M. J. (2016). Changing Demographics and Injury Profile of New Traumatic Spinal Cord Injuries in the United States, 1972-2014. Archives of Physical Medicine and Rehabilitation, 97(10), 1610–1619. https://doi.org/10.1016/j.apmr.2016.03.017 Cheng, C. L., Plashkes, T., Shen, T., Fallah, N., Humphreys, S., Connell, C. O., … Noonan, V. K. (2017). Abstracts and Workshops 7th National Spinal Cord Injury Conference November 9 – 11 , 2017 Fallsview Casino Resort Niagara Falls , Ontario , Management of obesity after spinal cord injury : a systematic review people with traumatic spinal cord injury, 40(6), 813–869. Chhabra, H. S. (ed). (2015). ISCoS Textbook on Comprehensive Management of Spinal Cord Injuries. New Dehli. Cobb, J., Dumont, F. S., Leblond, J., Park, S. E., Noonan, V. K., & Noreau, L. (2014). An Exploratory Analysis of the Potential Association Between SCI Secondary Health Conditions and Daily Activities. Topics in Spinal Cord Injury Rehabilitation, 20(4), 277–

77 288. https://doi.org/10.1310/sci2003-277 Coleman, N., Halas, G., Peeler, W., Casaclang, N., Williamson, T., & Katz, A. (2015). From patient care to research: A validation study examining the factors contributing to data quality in a primary care electronic medical record database. BMC Family Practice, 16(1), 1–8. https://doi.org/10.1186/s12875-015-0223-z Cripps, R. A., Lee, B. B., Wing, P., Weerts, E., MacKay, J., & Brown, D. (2011). A global map for traumatic spinal cord injury epidemiology: Towards a living data repository for injury prevention. Spinal Cord, 49(4), 493–501. https://doi.org/10.1038/sc.2010.146 Devivo, M. J. (2012). Epidemiology of traumatic spinal cord injury: trends and future implications. Spinal Cord, 50(5), 365–372. https://doi.org/10.1038/sc.2011.178 DeVivo, M J. (2007). Trends in spinal cord injury rehabilitation outcomes from model systems in the United States: 1973-2006. Spinal Cord, 45(11), 713–721. https://doi.org/10.1038/sj.sc.3102026 DeVivo, M J, Go, B. K., & Jackson, A. B. (2002). Overview of the National Spinal Cord Injury Statistical Center Database. J Spinal Cord Med, 25(4), 335–338. DeVivo, Michael J, Maetz, H. M., Fine, P. R., & Stover, S. L. (1980). Prevalence of Spinal Cord Injury: A Reestimation Employing Life Table Techniques. Archives of Neurology, 37(11), 707–708. Dick, T. B. . (1969). Traumatic paraplegia pre-Guttmann. Paraplegia, 7, 173–178. Donnelly, C., McColl, M. a, Charlifue, S., Glass, C., O’Brien, P., Savic, G., & Smith, K. (2007). Utilization, access and satisfaction with primary care among people with spinal cord injuries: a comparison of three countries. Spinal Cord, 45, 25–36. https://doi.org/10.1038/sj.sc.3101933 Dryden, D. M., Saunders, L. D., Rowe, B. H., May, L. A., Yiannakoulias, N., Svenson, L. W., … Voaklander, D. C. (2004). Utilization of health services following spinal cord injury: a 6- year follow-up study. Spinal Cord, 42(9), 513–525. https://doi.org/DOI 10.1038/sj.sc.3101629 Dvorak, M. F., Cheng, C. L., Fallah, N., Santos, A., Atkins, D., Humphreys, S., … Noonan, V. K. (2017). Spinal Cord Injury Clinical Registries: Improving Care across the SCI Care Continuum by Identifying Knowledge Gaps. Journal of Neurotrauma, 34(20), 2924–2933. https://doi.org/10.1089/neu.2016.4937 Fitzharris, M., Cripps, R. A., & Lee, B. B. (2014). Estimating the global incidence of traumatic spinal cord injury. Spinal Cord, 52(2), 117–122. https://doi.org/10.1038/sc.2013.135 Ford, E., Carroll, J. A., Smith, H. E., Scott, D., & Cassell, J. A. (2016a). Extracting information from the text of electronic medical records to improve case detection: A systematic review. Journal of the American Medical Informatics Association, 23(5), 1007–1015. https://doi.org/10.1093/jamia/ocv180 Ford, E., Carroll, J. A., Smith, H. E., Scott, D., & Cassell, J. A. (2016b). Extracting information from the text of electronic medical records to improve case detection: A systematic review. Journal of the American Medical Informatics Association, 23(5), 1007–1015. https://doi.org/10.1093/jamia/ocv180 Furlan, J. C., Sakakibara, B. M., Miller, W. C., & Krassioukov, A. V. (2013). Global Prevalence and incidence of traumatic spinal cord injury. Can J Neurol Sci, 40, 456–464.

78 https://doi.org/10.2147/CLEP.S68889 Garies, S., Birtwhistle, R., Drummond, N., Queenan, J., & Williamson, T. (2017). Data Resource Profile: National electronic medical record data from the Canadian Primary Care Sentinel Surveillance Network (CPCSSN). International Journal of Epidemiology, 46(4), 1091- 1092f. https://doi.org/10.1093/ije/dyw248 Gentil, M.-L., Cuggia, M., Fiquet, L., Hagenbourger, C., Le Berre, T., Banâtre, A., … Chapron, A. (2017). Factors influencing the development of primary care data collection projects from electronic health records: a systematic review of the literature. BMC Medical Informatics and Decision Making. https://doi.org/10.1186/s12911-017-0538-x Griffin, M. R., O’Fallon, W. M., Opitz, J. L., & Kurland, L. T. (1985). Mortality, survival and prevalence: traumatic spinal cord injury in Olmsted County, Minnesota, 1935-1981. Journal of Chronic Diseases, 38(8), 643–653. Guilcher, S J T, Craven, B. C., Calzavara, a, McColl, M. a, & Jaglal, S. B. (2013). Is the emergency department an appropriate substitute for primary care for persons with traumatic spinal cord injury? Spinal Cord, 51(March), 202–208. https://doi.org/10.1038/sc.2012.123 Guilcher, S J T, Munce, S. E. P., Couris, C. M., Fung, K., Craven, B. C., Verrier, M., & Jaglal, S. B. (2010). Health care utilization in non-traumatic and traumatic spinal cord injury: a population-based study. Spinal Cord, 48(December 2008), 45–50. https://doi.org/10.1038/sc.2009.78 Guilcher, Sara J. T. (2009). Physician utilization among adults with traumatic spinal cord injury in Ontario. Spinal Cord, 47, 470–476. Guilcher, Sara J. T., Craven, B. C., Lemieux-Charles, L., Casciaro, T., McColl, M. A., & Jaglal, S. B. (2012). Secondary health conditions and spinal cord injury: an uphill battle in the journey of care. Disability and Rehabilitation, 35(August 2012), 1–13. https://doi.org/10.3109/09638288.2012.721048 Guilcher, Sara J.T., Bronskill, S. E., Guan, J., & Wodchis, W. P. (2016). Who are the high-cost users? A method for person-centred attribution of health care spending. PLoS ONE, 11(3), 1–15. https://doi.org/10.1371/journal.pone.0149179 Guilcher, Sara J.T., Hogan, M. E., Calzavara, A., Hitzig, S. L., Patel, T., Packer, T., & Lofters, A. K. (2018). Prescription drug claims following a traumatic spinal cord injury for older adults: a retrospective population-based study in Ontario, Canada. Spinal Cord, 56(11), 1059–1068. https://doi.org/10.1038/s41393-018-0174-z Hagen, E., Rekand, T., Gilhus, N., & Gronning, M. (2008). Diagnostic coding accuracy for traumatic spinal cord injuries. Spinal Cord, 47, 367–371. https://doi.org/10.1038/sc.2008.118 Haroon, S. M., Jordan, R. E., O’beirne-Elliman, J., & Adab, P. (2015). Effectiveness of case finding strategies for COPD in primary care: A systematic review and meta-analysis. Npj Primary Care Respiratory Medicine, 25(1), 15056. https://doi.org/10.1038/npjpcrm.2015.56 Häyrinen, K., Saranto, K., & Nykänen, P. (2008). Definition, structure, content, use and impacts of electronic health records: A review of the research literature. International Journal of Medical Informatics. https://doi.org/10.1016/j.ijmedinf.2007.09.001 Hitzig, S. L., Tonack, M., Campbell, K. a, McGillivray, C. F., Boschen, K. a, Richards, K., & Craven, B. C. (2008). Secondary health complications in an aging Canadian spinal cord

79 injury sample. American Journal of Physical Medicine & Rehabilitation / Association of Academic Physiatrists, 87(July), 545–555. https://doi.org/10.1097/PHM.0b013e31817c16d6 Ho, C., Guilcher, S. J. T., McKenzie, N., Mouneimne, M., Williams, A., Voth, J., … Jaglal, S. B. (2017). Validation of Algorithm to Identify Persons with Non-traumatic Spinal Cord Dysfunction in Canada Using Administrative Health Data. Topics in Spinal Cord Injury Rehabilitation, 23(4), 333–342. https://doi.org/10.1310/sci2304-333 Høgholen, H., Storhaug, A., Kvernrød, K., Kostovski, E., Viktil, K. K., & Mathiesen, L. (2017). Use of medicines, adherence and attitudes to medicines among persons with chronic spinal cord injury. Spinal Cord, 56, 35–40. https://doi.org/10.1038/sc.2017.95 Holmes, C., Brown, M., Hilaire, D. S., & Wright, A. (2012). Healthcare provider attitudes towards the problem list in an electronic health record: a mixed-methods qualitative study. BMC Medical Informatics and Decision Making, 12, 1. https://doi.org/10.1186/1472-6947- 12-127 Hripcsak, G., & Albers, D. J. (2013). Next-generation phenotyping of electronic health records. Journal of the American Medical Informatics Association, 20(1), 117–121. https://doi.org/10.1136/amiajnl-2012-001145 Hwang, M., Zebracki, K., & Vogel, L. C. (2015). Medication profile and polypharmacy in adults with pediatric-onset spinal cord injury. Spinal Cord, 53(9), 673–678. https://doi.org/10.1038/sc.2015.62 Ivers, N., Pylypenko, B., & Tu, K. (2011). Identifying Patients With Ischemic Heart Disease in an Electronic Medical Record. Journal of Primary Care & Community Health, 2(1), 49–53. https://doi.org/10.1177/2150131910382251 Jaakkimainen, L., Crampton, N., Pinzaru, V. B., DelGiudice, L., & Tu, K. (2018). Using family physician Electronic Medical Record data to measure the pathways of cancer care. Toronto, ON. https://doi.org/10.23889/ijpds.v3i4.782 Jaglal, S B, Munce, S. E. P., Guilcher, S. J., Couris, C. M., Fung, K., Craven, B. C., & Verrier, M. (2009). Health system factors associated with rehospitalizations after traumatic spinal cord injury: a population-based study. Spinal Cord, 47(8), 604–609. https://doi.org/10.1038/sc.2009.9 Jaglal, Susan B. (2009). Health system factors associated with rehospitalizations after traumatic spinal cord injury. Spinal Cord, 47, 604–609. Jain, N. B., Ayers, G. D., Peterson, E. N., Harris, M. B., Morse, L., O’Connor, K. C., & Garshick, E. (2015). Traumatic Spinal Cord Injury in the United States, 1993-2012. JAMA. https://doi.org/10.1001/jama.2015.6250 Jensen, M. P., Truitt, a R., Schomer, K. G., Yorkston, K. M., Baylor, C., & Molton, I. R. (2013). Frequency and age effects of secondary health conditions in individuals with spinal cord injury: a scoping review. Spinal Cord, 51(12), 882–892. https://doi.org/10.1038/sc.2013.112 Johnson, R. L., Gabella, B. A., Gerhart, K. A., McCray, J., Menconi, J. C., & Whiteneck, G. G. (1997). Evaluating sources of traumatic spinal cord injury surveillance data in Colorado. Am J Epidemiol, 146(0002–9262), 266–272. Kennedy, P., & Chessell, Z. J. (2013). Traumatic versus non-traumatic spinal cord injuries: Are there differential rehabilitation outcomes¿. Spinal Cord, 51(7), 579–583. https://doi.org/10.1038/sc.2013.27

80 Kennedy, P., & Hasson, L. (2016). An audit of demographics and rehabilitation outcomes in non-traumatic spinal cord injury. Spinal Cord, 54(11), 1020–1024. https://doi.org/10.1038/sc.2016.12 Kirshblum, S. C., Waring, W., Biering-Sorensen, F., Burns, S. P., Johansen, M., Schmidt-Read, M., … Krassioukov, A. (2011). International Standards for Neurological Classification of Spinal Cord Injury. Journal of Spinal Cord Medicine, 34(6), 547–554. https://doi.org/10.1179/107902611X13186000420242 Koskinen, E. A., Alen, M., Väärälä, E. M., Rellman, J., Kallinen, M., & Vainionpää, A. (2014). Centralized spinal cord injury care in Finland: unveiling the hidden incidence of traumatic injuries. Spinal Cord. https://doi.org/10.1038/sc.2014.131 Kraus, J. F., Franti, C. E., Riggins, R. S., Richards, D., & Borhani, N. O. (1975). Incidence of traumatic spinal cord lesions. Journal of Chronic Diseases, 28(9), 471–492. https://doi.org/10.1016/0021-9681(75)90057-0 Krause, J. S., Clark, J. M. R., & Saunders, L. L. (2015). SCI Longitudinal Aging Study: 40 Years of Research. Topics in Spinal Cord Injury Rehabilitation, 21(3), 189–200. https://doi.org/10.1310/sci2103-189 Krause, J. S., & Saunders, L. L. (2011). Health, secondary conditions, and life expectancy after spinal cord injury. Archives of Physical Medicine and Rehabilitation, 92(11), 1770–1775. https://doi.org/10.1016/j.apmr.2011.05.024 Krauss, J. C., Boonstra, P. S., Vantsevich, A. V, & Friedman, C. P. (2016). Is the problem list in the eye of the beholder? An exploration of consistency across physicians. Journal of the American Medical Informatics Association, 23(5), 859–865. https://doi.org/10.1093/jamia/ocv211 Krysko, K. M., Ivers, N. M., Young, J., O’Connor, P., & Tu, K. (2015). Identifying individuals with multiple sclerosis in an electronic medical record. Multiple Sclerosis Journal, 21(2), 217–224. https://doi.org/10.1177/1352458514538334 Lee, B. B., Cripps, R. A., Fitzharris, M., & Wing, P. C. (2014). The global map for traumatic spinal cord injury epidemiology: Update 2011, global incidence rate. Spinal Cord, 52(2), 110–116. https://doi.org/10.1038/sc.2012.158 Luna, D., Franco, M., Plaza, C., Otero, C., Wassermann, S., Gambarte, M. L., … González Bernaldo De Quirós, F. (2013). Accuracy of an electronic problem list from primary care providers and specialists. Studies in Health Technology and Informatics, 192(1–2), 417– 421. https://doi.org/10.3233/978-1-61499-289-9-417 Majdan, M., Brazinova, A., & Mauritz, W. (2016). Epidemiology of traumatic spinal cord injuries in Austria 2002–2012. European Spine Journal, 25(1), 62–73. https://doi.org/10.1007/s00586-015-3985-z McBrien, K. A., Souri, S., Symonds, N. E., Rouhi, A., Lethebe, B. C., Williamson, T. S., … Ronksley, P. E. (2018). Identification of validated case definitions for medical conditions used in primary care electronic medical record databases: a systematic review. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocy094 McCaughey, E. J., Purcell, M., Mclean, A. N., Fraser, M. H., Bewick, A., Borotkanics, R. J., & Allan, D. B. (2016). Changing demographics of spinal cord injury over a 20-year period: A longitudinal population-based study in Scotland. Spinal Cord, 54(4), 270–276.

81 https://doi.org/10.1038/sc.2015.167 Montoto-Marqués, A., Ferreiro-Velasco, M. E., Salvador-De La Barrera, S., Balboa-Barreiro, V., Rodriguez-Sotillo, A., & Meijide-Failde, R. (2017). Epidemiology of traumatic spinal cord injury in Galicia, Spain: Trends over a 20-year period. Spinal Cord, 55(6), 588–594. https://doi.org/10.1038/sc.2017.13 Munce, S. E. P., Guilcher, S. J. T., Couris, C. M., Fung, K., Craven, B. C., Verrier, M., & Jaglal, S. B. (2009). Physician utilization among adults with traumatic spinal cord injury in Ontario: A population-based study. Spinal Cord, 47(6), 470–476. https://doi.org/10.1038/sc.2008.173 Munce, S. E. P., Wodchis, W. P., Guilcher, S. J. T., Couris, C. M., Verrier, M., Fung, K., … Jaglal, S. B. (2013). Direct costs of adult traumatic spinal cord injury in Ontario. Spinal Cord, 51(1), 64–69. https://doi.org/10.1038/sc.2012.81 New, P. W., Farry, A., Baxter, D., & Noonan, V. K. (2013). Prevalence of non-traumatic spinal cord injury in Victoria, Australia. Spinal Cord. https://doi.org/10.1038/sc.2012.61 New, P. W., & Marshall, R. (2014a). International Spinal Cord Injury Data Sets for non- traumatic spinal cord injury. Spinal Cord. https://doi.org/10.1038/sc.2012.160 New, P. W., & Marshall, R. (2014b). International Spinal Cord Injury Data Sets for non- traumatic spinal cord injury. Spinal Cord, 52(2), 123–132. https://doi.org/10.1038/sc.2012.160 New, P. W., Simmonds, F., & Stevermuer, T. (2011). A population-based study comparing traumatic spinal cord injury and non-traumatic spinal cord injury using a national rehabilitation database. Spinal Cord, 49(3), 397–403. https://doi.org/10.1038/sc.2010.77 New, Peter W, & Delafosse, V. (2012). What to call spinal cord damage not due to trauma? Implications for literature searching. The Journal of Spinal Cord Medicine, 35(2), 89–95. https://doi.org/10.1179/2045772311Y.0000000053 Noonan, V., Thorogood, N. P., Fingas, M., Batke, J., Belanger, L. M., Kwon, B. K., & Dvorak, M. F. (2012). The Validity of Administrative Data to Classify Patients with Spinal Column and Cord injuries. Journal of Neurotrauma, 180, 120924081652009. https://doi.org/10.1089/neu.2012.2441 Noonan, V. V. K., Fallah, N., Park, S. S. E., Dumont, F. F. S., Leblond, J., Cobb, J., & Noreau, L. (2014). Health Care Utilization in Persons with Traumatic Spinal Cord Injury: The Importance of Multimorbidity and the Impact on Patient Outcomes. Topics in Spinal Cord Injury Rehabilitation, 20(4), 289–301. https://doi.org/10.1310/sci2003-289 Noonan, V K, Kwon, B. K., Soril, L., Fehlings, M. G., Hurlbert, R. J., Townson, A., … Dvorak, M. F. (2012). The Rick Hansen Spinal Cord Injury Registry (RHSCIR): a national patient- registry. Spinal Cord, 50(1), 22–27. https://doi.org/10.1038/sc.2011.109 Noonan, Vanessa K., Fingas, M., Farry, A., Baxter, D., Singh, A., Fehlings, M. G., & Dvorak, M. F. (2012). Incidence and prevalence of spinal cord injury in Canada: A national perspective. Neuroepidemiology, 38(4), 219–226. https://doi.org/10.1159/000336014 Patel, T., Milligan, J., & Lee, J. (2017). Medication-related problems in individuals with spinal cord injury in a primary care-based clinic. Journal of Spinal Cord Medicine, 40(1), 54–61. https://doi.org/10.1179/2045772315Y.0000000055 Post, M. W. M., Brinkhof, M. W. G., von Elm, E., Boldt, C., Brach, M., Fekete, C., … Stucki, G.

82 (2011). Design of the Swiss Spinal Cord Injury Cohort Study. American Journal of Physical Medicine & Rehabilitation, 90, S5–S16. https://doi.org/10.1097/PHM.0b013e318230fd41 Rick Hansen Institute. (2017). Rick Hansen Spinal Cord Injury Registry A look at traumatic spinal cord injury in Canada in 2016. Vancouver. Rouleau, P., & Guertin, P. A. (2011). Traumatic and nontraumatic spinal-cord-injured patients in Quebec, Canada. Part 3: Pharmacological characteristics. Spinal Cord, 49(2), 186–195. https://doi.org/10.1038/sc.2010.70 Rowan, C. P., Chan, B. C. F., Jaglal, S. B., & Catharine Craven, B. (2019). Describing the current state of post-rehabilitation health system surveillance in Ontario–an invited review. Journal of Spinal Cord Medicine, 42(sup1), 21–33. https://doi.org/10.1080/10790268.2019.1605724 Rusanov, A., Weiskopf, N. G., Wang, S., & Weng, C. (2014). Hidden in plain sight: Bias towards sick patients when sampling patients with sufficient electronic health record data for research. BMC Medical Informatics and Decision Making, 14(1), 51. https://doi.org/10.1186/1472-6947-14-51 Singh, A., Tetreault, L., Kalsi-Ryan, S., Nouri, A., & Fehlings, M. G. (2014). Global prevalence and incidence of traumatic spinal cord injury. Clinical Epidemiology, 6, 309–331. https://doi.org/10.2147/CLEP.S68889 Smith, E., Fitzpatrick, P., Murtagh, J., Lyons, F., Morris, S., & Synnott, K. (2018). Epidemiology of Traumatic Spinal Cord Injury in Ireland, 2010-2015. Neuroepidemiology, 51(1–2), 19– 24. https://doi.org/10.1159/000488146 St. Germaine-Smith, C., Metcalfe, A., Pringsheim, T., Roberts, J. I., Beck, C. A., Hemmelgarn, B. R., … Jette, N. (2012). Recommendations for optimal ICD codes to study neurologic conditions a systematic review. Neurology. https://doi.org/10.1212/WNL.0b013e3182684707 Stover, S. L., DeVivo, M. J., & Go, B. K. (1999). History, implementation, and current status of the National Spinal Cord Injury Database. Archives of Physical Medicine and Rehabilitation, 80(11), 1365–1371. https://doi.org/10.1016/S0003-9993(99)90246-0 Strauss, D. J., Devivo, M. J., Paculdo, D. R., & Shavelle, R. M. (2006). Trends in life expectancy after spinal cord injury. Archives of Physical Medicine and Rehabilitation, 87(8), 1079– 1085. https://doi.org/10.1016/j.apmr.2006.04.022 Szeto, H. C., Coleman, R. K., Gholami, P., Hoffman, B. B., & Goldstein, M. K. (2002). Accuracy of computerized outpatient diagnoses in a Veterans Affairs General Medicine Clinic. American Journal of Managed Care, 8(1), 37–43. https://doi.org/370 [pii] Teixeira, P. L., Wei, W. Q., Cronin, R. M., Mo, H., VanHouten, J. P., Carroll, R. J., … Denny, J. C. (2017). Evaluating electronic health record data sources and algorithmic approaches to identify hypertensive individuals. Journal of the American Medical Informatics Association, 24(1), 162–171. https://doi.org/10.1093/jamia/ocw071 Thietje, R., & Hirschfeld, S. (2017). Epidemiology of spinal cord injury. In N. Weidner & Et al. (Eds.), Neurological Aspects of Spinal Cord Injury (Vol. 3, pp. 3–17). Springer International Publishing Switzerland. https://doi.org/10.1007/978-3-319-46293-6_1 Thurman, D. J., Burnett, C. L., Jeppson, L., Beaudoin, D. E., & Sniezek, J. E. (1994). Surveillance of spinal cord injuries in UTAH, USA. Paraplegia, 32(10), 665–669.

83 https://doi.org/10.1038/sc.1994.107 Tu, K., Manuel, D., Lam, K., Kavanagh, D., Mitiku, T. F., & Guo, H. (2011). Diabetics can be identified in an electronic medical record using laboratory tests and prescriptions. Journal of Clinical Epidemiology, 64(4), 431–435. https://doi.org/10.1016/j.jclinepi.2010.04.007 Tu, K., Mitiku, T. F., Ivers, N. M., Guo, H., Lu, H., Jaakkimainen, L., … Tu, J. V. (2014). Evaluation of Electronic Medical Record Administrative data Linked Database (EMRALD). American Journal of Managed Care, 20(1), 15–21. Tu, K., Msc, M., Fcfp, C., Mitiku, T., Msc, B., Lee, D. S., … Tu, J. V. (2010). Validation of physician billing and hospitalization data to identify patients with ischemic heart disease using data from the Electronic Medical Record Administrative data Linked Database (EMRALD). Canadian Journal of Cardiology, 26(7), e225–e228. https://doi.org/10.1016/S0828-282X(10)70412-8 Tu, K., Wang, M., Jaakkimainen, R. L., Butt, D., Ivers, N. M., Young, J., … Jetté, N. (2014). Assessing the validity of using administrative data to identify patients with epilepsy. Epilepsia, 55(2), 335–343. https://doi.org/10.1111/epi.12506 Tu, K., Widdifield, J., Young, J., Oud, W., Ivers, N. M., Butt, D. A., … Jaakkimainen, L. (2015). Are family physicians comprehensively using electronic medical records such that the data can be used for secondary purposes? A Canadian perspective. BMC Medical Informatics and Decision Making, 15(1), 67. https://doi.org/10.1186/s12911-015-0195-x Van Den Berg, M. E. E., Castellote, J. M., Mahillo-Fernandez, I., & De Pedro-Cuesta, J. (2012). Incidence of nontraumatic spinal cord injury: A Spanish cohort study (1972-2008). Archives of Physical Medicine and Rehabilitation. https://doi.org/10.1016/j.apmr.2011.08.027 Wei, W.-Q., Cronin, R. M., Xu, H., Lasko, T. A., Bastarache, L., & Denny, J. C. (2013). Development and evaluation of an ensemble resource linking medications to their indications. Journal of the American Medical Informatics Association : JAMIA, 20(5), 954– 961. https://doi.org/10.1136/amiajnl-2012-001431 Wei, W. Q., Teixeira, P. L., Mo, H., Cronin, R. M., Warner, J. L., & Denny, J. C. (2016). Combining billing codes, clinical notes, and medications from electronic health records provides superior phenotyping performance. Journal of the American Medical Informatics Association, 23(e1), 20–27. https://doi.org/10.1093/jamia/ocv130 Welk, B., Loh, E., Shariff, S. Z., Liu, K., & Siddiqi, F. (2014). An administrative data algorithm to identify traumatic spinal cord injured patients: a validation study. Spinal Cord, 52(1), 34– 38. https://doi.org/10.1038/sc.2013.134 Wright, A., Feblowitz, J., Maloney, F. L., Henkin, S., & Bates, D. W. (2012). Use of an electronic problem list by primary care providers and specialists. Journal of General Internal Medicine, 27(8), 968–973. https://doi.org/10.1007/s11606-012-2033-5 Wright, A., Pang, J., Feblowitz, J. C., Maloney, F. L., Wilcox, A. R., Ramelson, H. Z., … Wright, A. (2011). A method and knowledge base for automated inference of patient problems from structured data in an electronic medical record. Journal of the American Medical Informatics Association, 18, 859–867. https://doi.org/10.1136/amiajnl-2011- 000121 Xu, J., Rasmussen, L. V, Shaw, P. L., Jiang, G., Kiefer, R. C., Mo, H., … Montague, E. (2015). Review and evaluation of electronic health records-driven phenotype algorithm authoring

84 tools for clinical and translational research. Journal of the American Medical Informatics Association, 22(6), 1251–1260. https://doi.org/10.1093/jamia/ocv070

Appendices

Appendix A: NTSCI Chart Abstraction Case-Finding Criteria (From Ho (2017))

85 86 Appendix B: Final Keyword Search Strategy (3 versions)

Keyword search strategy, version 1: In CPP, PN, Cons (12): • neurogenic bladder • neurogenic bowel • parapare* • paraplegi* • quadrapare* • quadraplegi* • quadripare* • quadriplegi* • spasticity • spinal cord injury • tetrapare* • tetraplegi* In CPP only (2): • cauda equina • spina bifida

Keyword search strategy, version 2: In CPP, PN, Cons (11): • neurogenic bladder • neurogenic bowel • parapare* • paraplegi* • quadrapare* • quadraplegi* • quadripare* • quadriplegi* • spinal cord injury • tetrapare* • tetraplegi* In CPP only (3): • cauda equina • spina bifida • spinal bifida Exclusions in CPP only (4): • cerebral palsy • CP • MS • multiple sclerosis

87 Keyword search strategy, version 3 (final version): In CPP, PN, Cons (17): • anterior cord syndrome • brown sequard • central cord syndrome • conus medullaris syndrome • hemiparaplegi* • neurogenic bladder • neurogenic bowel • parapare* • paraplegi* • posterior cord syndrome • quadrapare* • quadraplegi* • quadripara* • quadriplegi* • spinal cord injury • tetrapare* • tetraplegi* In CPP only (126): • achondroplasia • acromegaly • actinomycosis • aneurysmal bone cyst • anterior spinal artery syndrome • arachnoiditis • arnold chiari • astrocytoma • atheromatous embol • atlanto axial dislocation • atlanto axial instability • basilar arachnoiditis • borreliosis • brucella spondylitis • cauda equina • cauda equine • cavernoma • cervical myelopathy • cervical stenosis • chordoma • copper deficiency • coxsackievirus • CVB • cryptococc • cysticercosis • cytomegalovirus

88 • diastometamyelia • dural arteriovenous • dural AV • ependymoma • epidural abscess • epidural haematoma • epidural hematoma • epidural neoplasm • extradural abscess • extradural disease • extradural lipoma • fibrocartilaginous embol • fluorosis • folate deficiency • friedreich • hereditary spastic paraparesis • hydatid • hydromyelia • hypertrophied filum terminale • hypoplastic dens • intramedullary tuberculoma • john cunningham • klippel feil • konzo • lathyrism • laxity of transverse atlantal ligament • leptomeninge • leukodystrophy • ligamentum flavum hypertrophy • lipomatosis • lumbar stenosis • lumbosacral agenesis • melioidosis • meningioma • meningocele • meningocoele • meningomyelitis • motor neuron disease • motor neurone disease • muco polysaccharididosis • myelitis • myeloneuropathy • myelopathy • neurobrucellosis • neurofibroma • neuromyelitis optica

89 • oligodendroglioma • os odontoideum • osteogenesis imperfecta • osteoma • osteomalacia • osteosarcoma • polyomavirus • primary lateral sclerosis • progressive muscular atrophy • pyogenic spodylitis • radiation myelitis • rickets • schwannoma • SCIWORA • septic discitis • spina bifida • spinal arachnoiditis • spinal artery syndrome • spinal bifida • spinal cord atrophy • spinal cord compression • spinal cord cyst • spinal cord disease • spinal cord haemorrhage • spinal cord hemorrhage • spinal cord infarct • spinal cord infection • spinal cord ischaemia • spinal cord ischemia • spinal cord lesion • spinal cord malformation • spinal cord neoplasm • spinal cord tumor • spinal cord vascular disease • spinal dysraphism • spinal infarct • spinal muscular atrophy • spinal osteophytosis • spinal paralysis • spinal TB • spinal tuberculosis • spino cerebellar ataxia • spondylolisthesis • spondylosis • subacute combined degeneration of spinal cord • syphilitic myelopathy

90 • syringomyelia • syrinx • tabes dorsalis • takayasu • tethered cord • thoracic stenosis • transverse myelitis • vertebral osteomyelitis Exclusions in CPP only (4): • cerebral palsy • CP • MS • multiple sclerosis

91 Appendix C: Charts Rated Non-cases

Appendix C, Table 1. Charts rated “No,” reason for capture by keyword search Reason for capture by keyword search n Warning mentions SCI 58 • warning re epidural block 36 • warning re risks of surgery 16 • warning re risks of anesthesia 2 • warning re cauda equina symptoms 1 • warning re failure to wear cervical collar 1 • warning re risk of SCI in case of fall/motor vehicle accident 1 • warning re risks of lumbar puncture 1 Spina bifida occulta 33 Patient report of SCI 32 • family history of SCI 24 • friend history of SCI 2 • patient concern re SCI 2 • patient reports near-SCI 2 • patient employment involves SCI 2 SCI mentioned, non-diagnosis 27 • SCI ruled out 12 • SCI mentioned 10 • SCI in name of clinic 3 • SCI considered 2 Other diagnosis (see detail) 161 Other (some evidence but did not meet criteria for diagnosis of SCI) 356 Total 667

Appendix C, Table 2. Other Diagnoses Diagnosis n Diabetes 42 Multiple sclerosis 28 Cerebral palsy 17 Irritable bowel syndrome 17 Cerebrovascular accident 10 Fibromyalgia 7 Arthritis 7 Benign prostatic hyperplasia 6 Colitis 4 Diverticulitis 4 Transient ischemic attack 4 Parkinsons 3 Amyotrophic lateral sclerosis 2 Diverticulosis 2

Prostate cancer 2 Alzheimers 1 Freidrich's ataxia 1 Prostatitis 1 Recurrent conversion disorder 1 Seizure disorder 1 Traumatic brain injury 1

93 Appendix D: Case-Finding Algorithm Components

Appendix D, Table 1. Case-Finding Algorithm Components: Free Text Keywords

Algorithm Search criteria component Type Terms included Terms excluded CPP1 Keywords in CPP brown sequard cerebral palsy central cord syndrome CP meningocele guillain barre neurogenic bowel MS quadrapare% (cases identified using SCI EMRALD MS spinal cord injury algorithm) tetrapare% tetraplegi%

CPP2 Keywords in CPP CPP1+ cerebral palsy cervical myelopathy CP paraplegi% guillain barre quadraplegi% MS quadriplegi% (cases identified using EMRALD MS algorithm) CPP3 Keywords in CPP CPP2+ cerebral palsy arnold chiari CP cauda equina guillain barre cervical myelopathy multiple sclerosis epidural abscess (cases identified using myelitis EMRALD MS myelopathy algorithm) neurogenic bladder occulta quadripare% osteomyelitis spina bifida syringomyelia tethered cord transverse myelitis

Consult/PN Keywords/phrases in brown sequard n/a progress notes/consult central cord syndrome letters hemiparaplegia neurogenic bowel parapare% paraplegi% quadrapare% quadraplegi% quadripare% quadriplegi% spinal cord injury tetrapare% tetraplegi%

95 Appendix E: Complete List of Algorithms Considered

Appendix E, Table 1. Case-Finding Algorithm Performance

Algorithm Description TP TN FN FP Sens. Spec. PPV NPV F-score CPPa 21 677 105 0 16.7 100 100 86.6 28.62 CPPb 53 671 73 6 42.1 99.1 89.8 90.2 57.32 CPP1 40 677 86 0 31.7 100 100 88.7 48.14 CPP2 66 673 60 4 52.4 99.4 94.3 91.8 67.37 CPP3 95 598 31 79 75.4 88.3 54.6 95.1 63.34 Drug 37 636 89 41 29.4 93.9 47.4 87.7 36.29 unique word count >='5' 6 677 120 0 4.8 100 100 84.9 9.16 any word count>='5' 48 674 78 3 38.1 99.6 94.1 89.6 54.24 unique word count>='4' 14 677 112 0 11.1 100 100 85.8 19.98 any word count>='4' 58 672 68 5 46 99.3 92.1 90.8 61.36 unique word count>='3' 28 675 98 2 22.2 99.7 93.3 87.3 35.87 any word count>='3' 68 669 58 8 54 98.8 89.5 92 67.36 unique word count>='2' 63 668 63 9 50 98.7 87.5 91.4 63.64 any word count>='2' 78 652 48 25 61.9 96.3 75.7 93.1 68.11 anyCode344 3 677 123 0 2.4 100 100 84.6 4.69 344x2 1 677 125 0 0.8 100 100 84.4 1.59 344x3 0 677 126 0 0 100 0 84.3 - 344x2in1 0 677 126 0 0 100 0 84.3 - 344x3in1 0 677 126 0 0 100 0 84.3 - anyCode806 20 677 106 0 15.9 100 100 86.5 27.44 806x2 16 677 110 0 12.7 100 100 86 22.54 806x3 12 677 114 0 9.5 100 100 85.6 17.35 806x2in1 11 677 115 0 8.7 100 100 85.5 16.01 806x3in1 8 677 118 0 6.3 100 100 85.2 11.85 anyCode952 0 677 126 0 0 100 0 84.3 - 952x2 0 677 126 0 0 100 0 84.3 - 952x3 0 677 126 0 0 100 0 84.3 - 952x2in1 0 677 126 0 0 100 0 84.3 - 952x3in1 0 677 126 0 0 100 0 84.3 - Drug or anyCode806 51 636 75 41 40.5 93.9 55.4 89.5 46.79 CPP1 or anycode806 45 677 81 0 35.7 100 100 89.3 52.62 CPP1 or 806x2 43 677 83 0 34.1 100 100 89.1 50.86 CPP1 or 806x3 42 677 84 0 33.3 100 100 89 49.96 CPP1 or 806x2in1 40 677 86 0 31.7 100 100 88.7 48.14 CPP1 or 806x3in1 40 677 86 0 31.7 100 100 88.7 48.14 CPP1 or Drug 66 636 60 41 52.4 93.9 61.7 91.4 56.67 CPP1 or Drug or anycode806 71 636 55 41 56.3 93.9 63.4 92 59.64 CPP1 or (Drug and anyCode806) 40 677 86 0 31.7 100 100 88.7 48.14 CPP1 or uniqueWord_count >='5' 43 677 83 0 34.1 100 100 89.1 50.86 CPP1 or anyWord_count >='5' 66 674 60 3 52.4 99.6 95.7 91.8 67.72 CPP1 or anycode806 or uniqueWord_count >='5' 48 677 78 0 38.1 100 100 89.7 55.18 CPP1 or anycode806 or anyWord_count >='5' 69 674 57 3 54.8 99.6 95.8 92.2 69.72 CPP1 or Drug or anycode806 or uniqueWord_count >='5' 71 636 55 41 56.3 93.9 63.4 92 59.64 CPP1 or Drug or anycode806 or anyWord_count >='5' 84 633 42 44 66.7 93.5 65.6 93.8 66.15 CPP1 or uniqueWord_count >='4' 48 677 78 0 38.1 100 100 89.7 55.18 CPP1 or anyWord_count >='4' 75 672 51 5 59.5 99.3 93.8 92.9 72.81 CPP1 or anycode806 or uniqueWord_count >='4' 52 677 74 0 41.3 100 100 90.1 58.46 CPP1 or anycode806 or anyWord_count >='4' 78 672 48 5 61.9 99.3 94 93.3 74.65 CPP1 or Drug or anycode806 or uniqueWord_count >='4' 74 636 52 41 58.7 93.9 64.3 92.4 61.37 CPP1 or Drug or anycode806 or anyWord_count >='4' 90 631 36 46 71.4 93.2 66.2 94.6 68.70 CPP1 or uniqueWord_count >='3' 56 675 70 2 44.4 99.7 96.6 90.6 60.84 CPP1 or anyWord_count >='3' 81 669 45 8 64.3 98.8 91 93.7 75.35

CPP1 or anycode806 or uniqueWord_count >='3' 60 675 66 2 47.6 99.7 96.8 91.1 63.82 CPP1 or anycode806 or anyWord_count >='3' 83 669 43 8 65.9 98.8 91.2 94 76.51 CPP1 or Drug or anycode806 or uniqueWord_count >='3' 79 634 47 43 62.7 93.6 64.8 93.1 63.73 CPP1 or Drug or anycode806 or anyWord_count >='3' 92 629 34 48 73 92.9 65.7 94.9 69.16 CPP1 or uniqueWord_count >='2' 76 668 50 9 60.3 98.7 89.4 93 72.02 CPP1 or anyWord_count >='2' 84 652 42 25 66.7 96.3 77.1 93.9 71.52 CPP1 or anycode806 or uniqueWord_count >='2' 78 668 48 9 61.9 98.7 89.7 93.3 73.25 CPP1 or anycode806 or anyWord_count >='2' 86 652 40 25 68.3 96.3 77.5 94.2 72.61 CPP1 or Drug or anycode806 or uniqueWord_count >='2' 89 628 37 49 70.6 92.8 64.5 94.4 67.41 CPP1 or Drug or anycode806 or anyWord_count >='2' 94 613 32 64 74.6 90.5 59.5 95 66.20 CPP2 or anycode806 70 673 56 4 55.6 99.4 94.6 92.3 70.04 CPP2 or 806x2 69 673 57 4 54.8 99.4 94.5 92.2 69.37 CPP2 or 806x3 68 673 58 4 54 99.4 94.4 92.1 68.70 CPP2 or 806x2in1 66 673 60 4 52.4 99.4 94.3 91.8 67.37 CPP2 or 806x3in1 66 673 60 4 52.4 99.4 94.3 91.8 67.37 CPP2 or Drug 80 632 46 45 63.5 93.4 64 93.2 63.75 CPP2 or Drug or anycode806 84 632 42 45 66.7 93.4 65.1 93.8 65.89 CPP2 or uniqueWord_count >='5' 68 673 58 4 54 99.4 94.4 92.1 68.70 CPP2 or anyWord_count >='5' 75 671 51 6 59.5 99.1 92.6 92.9 72.45 CPP2 or anycode806 or uniqueWord_count >='5' 72 673 54 4 57.1 99.4 94.7 92.6 71.24 CPP2 or anycode806 or anyWord_count >='5' 77 671 49 6 61.1 99.1 92.8 93.2 73.69 CPP2 or Drug or anycode806 or uniqueWord_count >='5' 84 632 42 45 66.7 93.4 65.1 93.8 65.89 CPP2 or Drug or anycode806 or anyWord_count >='5' 89 630 37 47 70.6 93.1 65.4 94.5 67.90 CPP2 or uniqueWord_count >='4' 70 673 56 4 55.6 99.4 94.6 92.3 70.04 CPP2 or anyWord_count >='4' 83 670 43 7 65.9 99 92.2 94 76.86 CPP2 or anycode806 or uniqueWord_count >='4' 73 673 53 4 57.9 99.4 94.8 92.7 71.89 CPP2 or anycode806 or anyWord_count >='4' 85 670 41 7 67.5 99 92.4 94.2 78.01 CPP2 or Drug or anycode806 or uniqueWord_count >='4' 85 632 41 45 67.5 93.4 65.4 93.9 66.43 CPP2 or Drug or anycode806 or anyWord_count >='4' 94 629 32 48 74.6 92.9 66.2 95.2 70.15 CPP2 or uniqueWord_count >='3' 72 671 54 6 57.1 99.1 92.3 92.6 70.55 CPP2 or anyWord_count >='3' 87 667 39 10 69 98.5 89.7 94.5 78.00 CPP2 or anycode806 or uniqueWord_count >='3' 75 671 51 6 59.5 99.1 92.6 92.9 72.45 CPP2 or anycode806 or anyWord_count >='3' 89 667 37 10 70.6 98.5 89.9 94.7 79.09 CPP2 or Drug or anycode806 or uniqueWord_count >='3' 87 630 39 47 69 93.1 64.9 94.2 66.89 CPP2 or Drug or anycode806 or anyWord_count >='3' 96 627 30 50 76.2 92.6 65.8 95.4 70.62 CPP2 or uniqueWord_count >='2' 84 666 42 11 66.7 98.4 88.4 94.1 76.03 CPP2 or anyWord_count >='2' 90 650 36 27 71.4 96 76.9 94.8 74.05 CPP2 or anycode806 or uniqueWord_count >='2' 86 666 40 11 68.3 98.4 88.7 94.3 77.17 CPP2 or anycode806 or anyWord_count >='2' 92 650 34 27 73 96 77.3 95 75.09 CPP2 or Drug or anycode806 or uniqueWord_count >='2' 95 626 31 51 75.4 92.5 65.1 95.3 69.87 CPP2 or Drug or anycode806 or anyWord_count >='2' 98 611 28 66 77.8 90.3 59.8 95.6 67.62 CPP3 or anycode806 97 598 29 79 77 88.3 55.1 95.4 64.23 CPP3 or 806x2 97 598 29 79 77 88.3 55.1 95.4 64.23 CPP3 or 806x3 97 598 29 79 77 88.3 55.1 95.4 64.23 CPP3 or 806x2in1 95 598 31 79 75.4 88.3 54.6 95.1 63.34 CPP3 or 806x3in1 95 598 31 79 75.4 88.3 54.6 95.1 63.34 CPP3 or Drug 99 562 27 115 78.6 83 46.3 95.4 58.27 CPP3 or Drug or anycode806 101 562 25 115 80.2 83 46.8 95.7 59.11 CPP3 or (Drug and anyCode806) 95 598 31 79 75.4 88.3 54.6 95.1 63.34 CPP3 or uniqueWord_count >='5' 97 598 29 79 77 88.3 55.1 95.4 64.23 CPP3 or anyWord_count >='5' 102 596 24 81 81 88 55.7 96.1 66.01

CPP3 or anycode806 or uniqueWord_count >='5' 99 598 27 79 78.6 88.3 55.6 95.7 65.13 CPP3 or anycode806 or anyWord_count >='5' 103 596 23 81 81.7 88 56 96.3 66.45 CPP3 or Drug or anycode806 or uniqueWord_count >='5' 102 562 24 115 81 83 47 95.9 59.48 CPP3 or Drug or anycode806 or anyWord_count >='5' 106 560 20 117 84.1 82.7 47.5 96.6 60.71 CPP3 or uniqueWord_count >='4' 100 598 26 79 79.4 88.3 55.9 95.8 65.61 CPP3 or anyWord_count >='4' 106 595 20 82 84.1 87.9 56.4 96.7 67.52 CPP3 or anycode806 or uniqueWord_count >='4' 101 598 25 79 80.2 88.3 56.1 96 66.02 CPP3 or anycode806 or anyWord_count >='4' 107 595 19 82 84.9 87.9 56.6 96.9 67.92 CPP3 or Drug or anycode806 or uniqueWord_count >='4' 104 562 22 115 82.5 83 47.5 96.2 60.29 CPP3 or Drug or anycode806 or anyWord_count >='4' 109 559 17 118 86.5 82.6 48 97 61.74 CPP3 or uniqueWord_count >='3' 101 596 25 81 80.2 88 55.5 96 65.60 CPP3 or anyWord_count >='3' 108 592 18 85 85.7 87.4 56 97 67.74 CPP3 or anycode806 or uniqueWord_count >='3' 102 596 24 81 81 88 55.7 96.1 66.01 CPP3 or anycode806 or anyWord_count >='3' 109 592 17 85 86.5 87.4 56.2 97.2 68.13 CPP3 or Drug or anycode806 or uniqueWord_count >='3' 105 560 21 117 83.3 82.7 47.3 96.4 60.34 CPP3 or Drug or anycode806 or anyWord_count >='3' 110 557 16 120 87.3 82.3 47.8 97.2 61.78 CPP3 or uniqueWord_count >='2' 106 592 20 85 84.1 87.4 55.5 96.7 66.87 CPP3 or anyWord_count >='2' 110 577 16 100 87.3 85.2 52.4 97.3 65.49 CPP3 or anycode806 or uniqueWord_count >='2' 107 592 19 85 84.9 87.4 55.7 96.9 67.27 CPP3 or anycode806 or anyWord_count >='2' 111 577 15 100 88.1 85.2 52.6 97.5 65.87 CPP3 or Drug or anycode806 or uniqueWord_count >='2' 109 557 17 120 86.5 82.3 47.6 97 61.41 CPP3 or Drug or anycode806 or anyWord_count >='2' 111 543 15 134 88.1 80.2 45.3 97.3 59.83

98 Appendix F: ICES Dataset Creation Plan

Dataset Creation Plan

Project Initiation This Section must be Completed Prior to Project Dataset(s) Creation Project Title: Developing a case-finding algorithm to identify patients with spinal cord injury in Ontario primary care electronic health records Project TRIM number: 2018 0904 616 000 Research Program: CDP Site: ICES Central Project Objectives: Insert Project Objectives as listed in the approved ICES Project PIA Develop and validate a case-finding algorithm that can be used to identify patients with spinal cord injury (SCI) in the electronic medical records of primary care practices. A reference standard will be developed using EMRALD data and will be used to measure the predictive value of possible algorithms. ICES Project PIA Initial Approval The ICES Employee or agent who is responsible for creating the Project Dataset(s) is responsible for ensuring Date: there is an approved ICES Project PIA and verifying the date of approval prior to creating the Project Dataset(s) 2017-Oct-25 Principal Investigator (PI): John Shepherd Check the applicable box if the PI ☒ ICES Student ☐ ICES Fellow ☐ ICES Post-Doctoral Trainee ☐ Visiting Scholar is an ICES Student/Trainee

Responsible EMRALD ICES Liisa Jaakkimainen Scientist: Liisa Jaakkimainen

Responsible ICES Scientist: Name the Responsible ICES Scientist if the PI is not a Full Status ICES Scientist Susan Jaglal Project Team Member(s) All person(s) (ICES Analyst, Appointed Analyst, Analytic Epidemiologist, PI, and/or Student) responsible for Responsible for Project Dataset creating the Project Dataset(s) and/or statistical analysis on the Research Analytics Environment (RAE) and the date they joined the project must be recorded Creation and/or Statistical Analysis and date joined (list all): Jacqueline Young, Bogdan Pinzaru 2017-Dec-06 Other ICES Project Team Members All other Research Project Team Members (e.g., Research Administrative Assistants, Research Assistants, and date joined (list all): Project Managers, Epidemiologists) and the date they joined the project must be recorded Erika Yates 2017-Dec-06 Confirmation that DCP is The following individuals must confirm that the ICES Data provided for in this DCP is relevant (e.g., with respect consistent with Project to cohort, timeframe, and variables) and required to achieve the Project Objectives stated in the ICES Project PIA prior to initial Project Dataset creation: 1) PI; 2) Responsible ICES Scientist if the PI is not a Full Status ICES Objectives: Scientist, or a second ICES Scientist or the Scientific Program Lead if the PI is creating both the DCP and the Project Dataset[s]; 3) ICES Research and Analysis Staff creating the DCP; and 4) ICES Analytic Staff (ICES Employee or agent responsible for creating the Project Dataset[s]). This may be delegated either verbally or via e-mail. Principal Investigator ☒ 2017-Dec-08 Responsible ICES Scientist or Second ICES Scientist/Lead ☐ yyyy-mon-dd ICES Research and Analysis Staff Creating the DCP ☐ yyyy-mon-dd ICES Analytic Staff ☐ yyyy-mon-dd Designated ICES Research and The person named (ICES staff) is accountable for ensuring that the approved ICES Project PIA, ICES Project PIA Amendments, and DCP are saved on the T Drive, ensuring ICES Project PIA Amendments are submitted as

ICES DCP Template v. 1.5 (04/11/16) Page 1

Dataset Creation Plan

Project Initiation This Section must be Completed Prior to Project Dataset(s) Creation Analysis Staff accountable for required, ensuring DCP Amendments are documented, and sharing the final DCP with the PI/Responsible ICES Project Documentation: Scientist at project completion

DCP Creation Date and Author: Date DCP was finalized prior to Project Dataset(s) creation Name of person who created the DCP Date Name yyyy-mon-dd John Shepherd ICES Data This Section must be Completed Prior to Project Dataset(s) Creation The ICES Employee or agent who is responsible for creating the Project Dataset(s) must ensure that this list includes only data listed in the ICES Project PIA Mandatory for all datasets that are available by Changes to this list after initial ICES Project PIA approval require an ICES Project PIA Amendment individual year General Use Datasets – Health Services Years (where applicable)

CIHI DAD 2015-2016 NACRS 2015-2016 ODB 2015-2016 OHIP 2015-2016 OMHRS 2015-2016 General Use Datasets – Care Providers

IPDB 2015-2016 See list General Use Datasets – Population

RPDB 2015-2016 See list General Use Datasets – Coding/Geography

See list See list General Use Datasets - Facilities

See list General Use Datasets - Other

ASTHMA 2015-2016 CHF 2015-2016 COPD 2015-2016 HYPER 2015-2016 ODD 2015-2016 Controlled Use Datasets See list ICES DCP Template v. 1.5 (04/11/16) Page 2

100

Dataset Creation Plan

Project Initiation This Section must be Completed Prior to Project Dataset(s) Creation See list Other Datasets EMRALD Up to 2016

ICES DCP Template v. 1.5 (04/11/16) Page 3

101

Dataset Creation Plan

Project Amendments and Reconciliation Privacy approval Person who submitted Note that any changes to the list of ICES Data or Project date amendment Objectives require an ICES Project PIA Amendment ICES Project PIA Amendment History (add additional rows as Date Name Amendment needed): 2017-Dec-06 John Shepherd Added data to Schedule 1, added collaborating researcher Note that any DCP amendments involving changes to the list Date DCP Person who made the DCP of ICES Data or Project Objectives require an ICES Project PIA DCP Amendment History (add amended amendment Amendment additional rows as needed): Date Name Amendment yyyy-mon-dd The person(s) creating the dataset and/or analyzing the data are responsible for ensuring that the final DCP Date Programs/DCP reconciled reflects the final program(s) when the project is completed yyyy-mon-dd

EMRALD Project Cohort

☐ Cohort study ☐ Matched cohort study ☐ Case-control study Study Design ☐ Cross-sectional study ☒ Other (specify): EMR All patients in EMRALD whose EMR start date was 2 years prior to their load date into Index Event/Date/Load Date EMRALD

Load # 5 Load Date Range Jan 1, 2015-Dec 31, 2016 Step Description Non DSA and non-active during load period (use d_DSA_status = 0 and Load #/Physician Exclusions 1 (in order) d_active = 0 for the load) 2 … # Final Physician Cohort Physicians N=385 Clinics N=43 Load/Physician Notes Step Description Belong to a non DSA/non active physician 1 (PS_DW.dbo.tb_basic_info.d_primary_entry = 0) 2 No valid Health Card Number (PS_DW.dbo.tb_basic_info.d_valid = 0)

Patient Exclusions (in order) 3 No valid date of birth (PS_DW.dbo.tb_basic_info.d_clean = 0) 4 Not rostered (PS_DW.dbo.tb_profile_info.d_roster = 0) 5 Less than (<) [15] years of age as of load date/index date Less than (<) [1] year on the EMR 6 (PS_DW.dbo.tb_basic_info.d_start_onEMR)

ICES DCP Template v. 1.5 (04/11/16) Page 4

102

Dataset Creation Plan

EMRALD Project Cohort No visit in the past [1] year since the load date/index date 7 (PS_DW.dbo.tb_progress_notes.d_has_bill) Patient Notes # Physicians in N= 376 cohort Clinics in cohort N=43 EMRALD Cohort Eligible Patients N=443,038 Cohort Patients N=213,887 Disease Patients N=3,547 potential SCI patients (if applicable) Random Sample N=48,000 (803 potential SCI patients) Patients Admin Project Cohort

Index Event / Inclusion If different than EMR date… Criteria Estimated Size of Cohort

(if known) Step Description 1 Exclusions (in order) 2 3 EMRALD Admin Linked Cohort Reason(s) for exclusions during linkage #

Physicians N = Clinics N = EMRALD Admin Eligible Patients N = Linked Cohort Sample Patients N = (if applicable) Disease Patients N = (if applicable)

ICES DCP Template v. 1.5 (04/11/16) Page 5

103

Dataset Creation Plan

Project Time Frame Definitions

Max Follow-up Date Accrual Window

Look-back Window Observation Window (in which to look for outcomes) Index Event Date Accrual Start/End Dates Max Follow-up Date When does observation window terminate? Lookback Window(s) Variable Definitions (add additional rows as needed) Main Exposure or Risk Factor Spinal cord injury Primary Outcome Definition Sensitivity, specificity, PPV, NPV of various within EMR data algorithms for identifying patients with the condition Secondary Outcome Definition(s) Baseline Characteristics Other Variables Analysis Plan and Dummy Tables (expand/modify as needed) Descriptive Tables (insert or append dummy tables), e.g.: Table 1. Baseline characteristics according to primary/secondary exposure Table 2. Outcomes according to primary/secondary exposure Table 3. Covariates (baseline characteristics) according to outcomes Statistical Model(s) Type of model Primary independent variable Dependent variable Covariates Sensitivity Analyses Type of model Primary independent variable Dependent variable Covariates

Quality Assurance Activities EMR Table(s)/Databases EMR_abstractin_spinal_injury.dbo.April27_cohort (n=443,038) EMR_abstraction_spinal_injury.dbo.April27_sample (n=48,000) EMR_abstraction_spinal_injury.dbo.April27_results (search terms and frequency) EMR_abstraction_spinal_injury.dbo.SCI_results (1st abstractor scores and evidence) EMR_abstraction_spinal_injury.dbo.SCI_results_final (validated scores + trauma categorization) EMR_within_EMR.dbo.SCI_n98 (within EMR results) EMR Shared Drive Directory

ICES DCP Template v. 1.5 (04/11/16) Page 6

104

Dataset Creation Plan

Quality Assurance Activities Date of EMR Analysis/ Date Cohort Created

RAE Directory of SAS Programs RAE Directory of Final Dataset(s) The final analytic dataset for each cohort includes all the data required to create the baseline tables and run all the models. It should include all covariates for all models such as patient risk factors, hospital characteristics, physician characteristics, exposure measures (continuous, categorical) and outcomes. It should include covariates that were considered but didn’t make the final cut. This would permit an analyst to easily re-run the models in the future.

RAE README file available: ☐Yes ☐No Date results of quality assurance tools for final dataset shared with project team (where applicable): %assign yyyy-mon-dd %evolution yyyy-mon-dd %dinexplore yyyy-mon-dd %track / %exclude yyyy-mon-dd %codebook yyyy-mon-dd Additional comments:

Analysis Plan

A reference standard population will be established using text mining (automated key work searching) within EMRALD, as well as complete chart review of potential cases. [The search terms used to identify spinal cord injury in a patient’s chart are listed in Appendix A.]

Charts will be labelled “Definite” if there is an unequivocal mention of spinal cord injury by the family physician or a specialist. “Possible” cases will refer to patients under investigation for spinal cord injury, due to evidence of related secondary complications including (but not limited to) neurogewnic bladder, neurogenic bowel, pressure ulcers, autonomic dysreflexia, and spasticity. Patients will receive a score of “No” if spinal cord injury was excluded/rulled out, or an alternative explanation for symptoms was discovered.

EMRALD data algorithms will be generated in an iterative fashion and tested against the reference standard. The collected data will be used to calculate cohort prevalence rates. An optimized algorithm will be selected and tested against the reference standard to maximize sensitivity, specificity, positive predictive value, and negative predictive value. Discordance analysis will be performed to examine false positive and false negative results, and to determine whether there are inaccuracies in the selected algorithm.

Appendix A – EMRALD Search Terms for Spinal cord injury

Spinal Cord Injury: final EMRald search strategy (revised April 27, 2018) Total keywords: 143 Inclusion: patients >14 years as of load date 1. Search for occurrences in CP, PN, Cons [17 terms]: anterior cord syndrome brown sequard central cord syndrome

ICES DCP Template v. 1.5 (04/11/16) Page 7

105

Dataset Creation Plan conus medullaris syndrome hemiparaplegi% neurogenic bladder neurogenic bowel parapare% paraplegi% posterior cord syndrome quadrapare% quadraplegi% quadripare% quadriplegi% spinal cord injury tetrapare% tetraplegi%

2. Search for occurrences in CPP only [136 126 terms]: achondroplasia acromegaly actinomycosis aneurysmal bone cyst anterior spinal artery syndrome arachnoiditis arnold chiari astrocytoma atheromatous embol% atlanto axial dislocation atlanto axial instability b12 deficiency basilar arachnoiditis beck% borreliosis brucella spondylitis cauda equina cauda equine cavernoma cervical myelopathy cervical stenosis chordoma copper deficiency coxsackievirus CVB cryptococc% cysticercosis Cytomegalovirus

ICES DCP Template v. 1.5 (04/11/16) Page 8

106

Dataset Creation Plan

CMV devic diastometamyelia dural arteriovenous dural AV ependymoma epidural abscess epidural haematoma epidural hematoma epidural neoplasm extradural abscess extradural disease extradural lipoma fibrocartilaginous embol% fluorosis folate deficiency friedreich% hereditary spastic paraparesis hydatid hydromyelia hypertrophied filum terminale hypoplastic dens intramedullary tuberculoma john cunningham klippel feil konzo lathyrism laxity of transverse atlantal ligament leptomeninge leukodystrophy ligamentum flavum hypertrophy lipomatosis lumbar stenosis lumbosacral agenesis melioidosis meningioma meningocele meningocoele meningomyelitis motor neuron disease motor neurone disease muco polysaccharididosis myelitis AND NOT osteomyelitis myeloneuropathy ICES DCP Template v. 1.5 (04/11/16) Page 9

107

Dataset Creation Plan myelopathy neurobrucellosis neurofibroma neuromyelitis optica oligodendroglioma os odontoideum osteochondroma osteogenesis imperfecta osteoma osteomalacia osteosarcoma paget% PD polyomavirus primary lateral sclerosis progressive muscular atrophy pyogenic spodylitis radiation myelitis rickets sarcoidosis schwannoma SCIWORA septic discitis sjogren% SS spina bifida SB spinal arachnoiditis spinal artery syndrome spinal bifida spinal cord atrophy spinal cord compression spinal cord cyst spinal cord disease spinal cord haemorrhage spinal cord hemorrhage spinal cord infarct% spinal cord infection spinal cord ischaemia spinal cord ischemia spinal cord lesion spinal cord malformation spinal cord neoplasm spinal cord tumor ICES DCP Template v. 1.5 (04/11/16) Page 10

108

Dataset Creation Plan spinal cord vascular disease spinal dysraphism spinal infarct% spinal muscular atrophy spinal osteophytosis spinal paralysis spinal stenosis spinal TB spinal tuberculosis spino cerebellar ataxia spondylolisthesis spondylosis subacute combined degeneration of spinal cord syphilitic myelopathy syringomyelia syrinx tabes dorsalis takayasu% tethered cord thoracic stenosis transverse myelitis vertebral osteomyelitis

The above search terms include misspellings.

ICES DCP Template v. 1.5 (04/11/16) Page 11