arXiv:1703.10090v1 [cs.CL] 29 Mar 2017 hl h eet fciia L n aamin- data and NLP clinical of benefits the while providers healthcare by written notes of use The eo n pl opttoa ehd o lin- for methods computational apply and velop ifrn rus n itesaigo implemen- of sharing little from and groups, researchers data, different between to access collaboration difficult limited notably most identified, has slow techniques been NLP clinical of development the in acknowledged, universally But been have ing 2008). al., et Meystre 2009; al., 1995; et (Demner-Fushman al., et Hripcsak reports text from knowledge free of extraction de- and analysis to guistic is clini- (NLP) of processing goal language The natural cal 2014; 2013). Goldacre, monitoring Detsky, and 2016; and Murdoch (Agus, treatments, outcomes dis- clinical ef- beneficial of side of detecting causes clinical fects diagnoses, identifying of infor- re- establishing in quantities eases, valuable medical help large rec- may to and of reports Access been practice source long clinical search. a for has be mation settings to clinical ognized the in Introduction 1 { feprclrsac n edt socially to applications. lead harmful and research validity empirical the of compromise can that attention biases less draw to we of Finally, sources data. suggest it sensitive also measures We the the and discuss entails. privacy we for paper, concern this In re- for searchers. policies access primarily data strict progress by caused slow largely to remained due has untapped potential this How- ever, large records. clinical of of advent processing scale the by will revolutionized practice be clinical how in to potential contributing immense an has NLP Clinical simon.suster,stephan.tulkens,walter.daelemans eea otiuigfcoshv been have factors contributing Several . Simon Abstract hr eiwo tia hlegsin Challenges Ethical of Review Short A lnclNtrlLnug Processing Language Natural Clinical Suster ˇ 1,2 1 2 LP,Uiest fAntwerp of University CLiPS, St , nwr nvriyHospital University Antwerp pa Tulkens ephan ´ progress okn aacnito imdclrsac litera- research biomedical of consist data working ihahatcr raiain opr ihtrans- with Corpora organization. healthcare a with tts.Asmn htsc pcfi apeis sample specific a such United that the Assuming in hospital States). single ge- and (a case) location this ographic in clini- care intensive particular only (the a domain is cal from it patients admissions), (cover- of hospital size representative large 58,000 its over often Despite ing 2016; An al., 2011). et al., (Johnson et few. Saeed MIMICII(I) is are corpus but used exist, researchers NLP aetacs oiisthat policies access parent connected not researchers the This for true 2013). especially al., is et relevant Friedman to 2016; access (Barzilay, difficult gain data is to community it NLP data, the clinical for of sensitivity the arising from concerns institutional and legal of Because privacy and data of Sensitivity 2 gen- well. as to science applicable data not are eral part and NLP, large clinical in to are unique The here describe processing. we their emanat- and challenges notes biases clinical and from impact ing social draw the do also but to privacy, we attention to norm, right the the discuss been only not has access data NLP restricted through broader privacy enhancing the Although in community. those about discussion ulate hspprt ahrteehclcalne htare that for challenges relevant ethical especially the in gather aim to We paper 2016). this Spruit, and (Hovy NLP general risk. at privacy patient’s put situations may certain processing in whose data, been of nature has sensitive situation the this more to The factor been contributing has main 2014). Demner-Fushman, progress and (Cohen the rapid and much degree, a to lesser present been have conditions these the ture, where NLP, biomedical in 2011). comparison, al., For et (Chapman models trained and tations h tisdsuso sgiigmmnu in momentum gaining is discussion ethics The 1 n atrDaelemans Walter and lnclNLP clinical } @uantwerpen.be are 1 ihnrahof reach within n ostim- to and , representative of a larger population is an exam- tems suitable for general use. ple of sampling bias (we discuss further sources of bias in section 3). Increasing the size of a sam- 2.1 Protecting the individual ple without recognizing that this sample is atypical Clinical notes contain detailed information about for the general population (e.g. not all patients are patient-clinician encounters in which patients con- critical care patients) could also increase sampling fide not only their health complaints, but also their 1 bias (Kaplan et al., 2014). We need more large lifestyle choices and possibly stigmatizing condi- corpora for various medical specialties, narrative tions. This confidential relationship is legally pro- types, as well as languages and geographic areas. tected in US by the HIPAA privacy rule in the case Related to difficult access to raw clinical data of individuals’ medical data. In EU, the conditions is the lack of available annotated datasets for for scientific usage of health data are set out in the model training and benchmarking. The reality General Data Protection Regulation (GDPR). San- is that annotation projects do take place, but are itization of sensitive data categories and individu- typically constrained to a single healthcare or- als’ informed consent are in the forefront of those ganization. Therefore, much of the effort put legislative acts and bear immediate consequences into annotation is lost afterwards due to impos- for the NLP research. sibility of sharing with the larger research com- The GDPR lists general principles relating to munity (Chapman et al., 2011; Fan et al., 2011). processing of personal data, including that pro- Again, exceptions are either few—e.g. THYME cessing must be lawful (e.g. by means of con- (Styler IV et al., 2014), a corpus annotated with sent), fair and transparent; it must be done for ex- temporal information—or consist of small datasets plicit and legitimate purposes; and the data should resulting from shared tasks like the i2b2 and be kept limited to what is necessary and as long ShARe/CLEF. In addition, stringent access poli- as necessary. This is known as data minimiza- cies hamper reproduction efforts, impede scien- tion, and it includes sanitization. The scientific tific oversight and limit collaboration, not only be- usage of health data concerns “special categories tween institutions but also more broadly between of personal data”. Their processing is only al- the clinical and NLP communities. lowed when the data subject gives explicit consent, There are known cases of datasets that had or the personal data is made public by the data been used in published research (including re- subject. Scientific usage is defined broadly and 2 production) in its full form, like MiPACQ , includes technological development, fundamental Blulab, EMC Dutch Clinical Corpus and 2010 and applied research, as well as privately funded i2b2/VA (Albright et al., 2013; Kim et al., 2015; research. Afzal et al., 2014; Uzuner et al., 2011), but were later trimmed down or made unavailable, likely Sanitization Sanitization techniques are often due to legal issues. Even if these datasets were still seen as the minimum requirement for protect- available in full, their small size is still a concern, ing individuals’ privacy when collecting data and the comments above regarding sampling bias (Berman, 2002; Velupillai et al., 2015). The goal certainly apply. For example, a named entity rec- is to apply a procedure that produces a new ver- ognizer trained on 2010 i2b2/VA data, which con- sion of the dataset that looks like the original for sists of 841 annotated patient records from three the purposes of data analysis, but which maintains different specialty areas, will due to its size only the privacy of those in the dataset to a certain contain a small portion of possible named entities. degree, depending on the technique. Documents Similarly, in linking clinical concepts to an ontol- can be sanitized by replacing, removing or oth- ogy, where the number of output classes is larger erwise manipulating the sensitive mentions such (Pradhan et al., 2013), the small amount of train- as names and geographic locations. A distinc- ing data is a major obstacle to deployment of sys- tion is normally drawn between anonymization, pseudonymization and de-identification. We refer 1Sampling bias could also be called selection bias; it is not inherent to the individual documents, but stems from the the reader to Polonetsky et al. (2016) for an excel- way these are arranged into a single corpus. lent overview of these procedures. 2The access to the MiPACQ corpus will be re-enabled in Although it is a necessary first step in protect- the future within the Health NLP Center for distributing lin- guistic annotations of clinical texts (Guergana Savova, per- ing the privacy of patients, sanitization has been sonal communication). criticized for several reasons. First, it affects the integrity of the data, and as a consequence, by several research teams, without the need to their utility (Duquenoy et al., 2008). Second, ask for consent each time. The success of imple- although sanitization in principle promotes data menting this approach in practice is likely to de- access and sharing, it may often not be suffi- pend on public trust and awareness about possible cient to eliminate the need for consent. This risks and opportunities. We also believe that a dis- is largely due to the well-known fact that orig- tinction between academic research and commer- inal sensitive data can be re-identified through cial use of clinical data should be implemented, deductive disclosure (Amblard et al., 2014; as the public is more willing to allow research De Mazancourt et al., 2015; Hardt et al., 2016; than commercial exploitation (Lawrence, 2016; Malin et al., 2013; Tene, 2011).3 Finally, sanitiza- van Staa et al., 2016). tion focuses on protecting the individual, whereas Yet another possibility is open consent, in which ethical harms are still possible on the group level individuals make their data publicly available. Ini- (O’Doherty et al., 2016; Taylor et al., 2017). In- tiatives like Personal Genome Project may have stead of working towards increasingly restrictive an exemplary role, however, they can only provide sanitization and access measures, another course limited data and they represent a biased population of action could be to work towards heightening sample (Mittelstadt and Floridi, 2016). the perception of scientific work, emphasizing pro- Secure access Since withholding data from re- fessionalism and existence of punitive measures searchers would be a dubious way of ensuring con- for illegal actions (Fairfield and Shtein, 2014; fidentiality (Berman, 2002), the research has long Mittelstadt and Floridi, 2016). been active on secure access and storage of sensi- Consent Clinical NLP typically requires a large tive clinical data, and the balance between the de- amount of clinical records describing cases of pa- gree of privacy loss and the degree of utility. This tients with a particular condition. Although obtain- is a broad topic that is outside the scope of this ar- ing consent is a necessary first step, obtaining ex- ticle. The interested reader can find the relevant plicit informed consent from each patient can also information in Dwork and Pottenger (2013), Ma- compromise the research in several ways. First, lin et al. (2013) and Rindfleisch (1997). obtaining consent is time consuming by itself, and Promotion of knowledge and application of it results in financial and bureaucratic burdens. It best-of-class approaches to health data is seen can also be infeasible due to practical reasons such as one of the ethical duties of researchers as a patient’s death. Next, it can introduce bias as (Duquenoy et al., 2008; Lawrence, 2016). But for those willing to grant consent represent a skewed this to be put in practice, ways need to be guar- population (Nyr´en et al., 2014). Finally, it can be anteed (e.g. with government help) to provide re- difficult to satisfy the informedness criterion: In- searchers with access to the relevant data. Re- formation about the experiment sometimes can not searchers can also go to the data rather than have be communicated in an unambiguous way, or ex- the data sent to them. It is an open question periments happen at speed that makes enacting in- though whether medical institutions—especially formed consent extremely hard (Bird et al., 2016). those with less developed research departments— The alternative might be a default opt-in pol- can provide the infrastructure (e.g. enough CPU icy with a right to withdraw (opt-out). Here, con- and GPU power) needed in statistical NLP. Also, sent can be presumed either in a broad manner— granting access to one healthcare organization at allowing unspecified future research, subject a time does not satisfy interoperability (cross- to ethical restrictions—or a tiered manner— organizational data sharing and research), which allowing certain areas of research but not others can reduce bias by allowing for more complete in- (Mittelstadt and Floridi, 2016; Terry, 2012). Since put data. Interoperability is crucial for epidemiol- the information about the intended use is no longer ogy and rare disease research, where data from one uniquely tied to each research case but is more institution can not yield sufficient statistical power general, this could facilitate the reuse of datasets (Kaplan et al., 2014). 3Additionaly, it may be due to organizational skepticism Are there less sensitive data? One criterion about the effectiveness of sanitization techniques, although which may have influence on data accessibility is it has been shown that automated de-identification systems for English perform on par with manual de-identification whether the data is about living subjects or not. (Deleger et al., 2013). The HIPAA privacy rule under certain conditions allows disclosure of personal health information of 3 Social impact and biases deceased persons, without the need to seek IRB agreement and without the need for sanitization Unlocking knowledge from free text in the health (Huser and Cimino, 2014). It is not entirely clear domain has a tremendous societal value. How- though how often this possibility has been used in ever, discrimination can occur when individuals clinical NLP research or broader. or groups receive unfair treatment as a result of automated processing, which might be a result of biases in the data that were used to train mod- Next, the work on surrogate data has re- els. The question is therefore what the most im- cently seen a surge in activity. Increasingly more portant biases are and how to overcome them, health-related texts are produced in social media not only out of ethical but also legal responsi- (Abbasi et al., 2014), and patient-generated data bility. Related to the question of bias is so- are available online. Admittedly, these may not called algorithm transparency (Goodman, 2016; resemble the clinical discourse, yet they bear to Kamarinou et al., 2016), as this right to explana- the same individuals whose health is documented tion requires that influences of bias in training data in the clinical reports. Indeed, linking individu- are charted. In addition to sampling bias, which als’ health information from online resources to we introduced in section 2, we discuss in this sec- their health records to improve documentation is tion further sources of bias. Unlike sampling bias, an active line of research (Padrez et al., 2015). Al- which is a corpus-level bias, these biases here are though it is generally easier to obtain access to already present in documents, and therefore hard social media data, the use of social media still to account for by introducing larger corpora. requires similar ethical considerations as in the clinical domain. See for example the influen- Data quality Texts produced in the clinical set- tial study on emotional contagion in Facebook tings do not always tell a complete or accurate pa- posts by Kramer et al. (2014), which has been tient story (e.g. due to time constraints or due to criticized for not properly gaining prior consent patient treatment in different hospitals), yet impor- from the users who were involved in the study tant decisions can be based on them.4 As language (Schroeder, 2014). is situated, a lot of information may be implicit, such as the circumstances in which treatment de- cisions are made (Hersh et al., 2013). If we fail Another way of reducing sensitivity of data and to detect a medical concept during automated pro- improving chances for IRB approval is to work cessing, this can not necessarily be a sign of nega- on derived data. Data that can not be used to tive evidence.5 Work on identifying and imputing reconstruct the original text (and when sanitized, missing values holds promise for reducing incom- can not directly re-identify the individual) include pleteness, see Lipton et al. (2016) for an example text fragments, various statistics and trained mod- in sequential modeling applied to diagnosis classi- els. Working on randomized subsets of clinical fication. notes may also improve the chances of obtaining the data. When we only have access to trained Reporting bias Clinical texts may include bias models from disparate sources, we can refine them coming from both patient’s and clinician’s report- through ensembling and creation of silver standard ing. Clinicians apply their subjective judgments corpora, cf. Rebholz-Schuhmann et al. (2011). to what is important during the encounter with pa- tients. In other words, there is separation between, Finally, clinical NLP is also possible on veteri- on the one side, what is observed by the clinician nary texts. Records of companion animals are and communicated by the patient, and on the other, perhaps less likely to involve legal issues, while 4A way to increase data completeness and reduce selec- still amounting to a large pool of data. As an tion bias is the use of nationwide patient registries, as known example, around 40M clinical documents from for example in Scandinavian countries (Schmidt et al., 2015). 5We can take timing-related “censoring” effects as an ex- different veterinary clinics in UK and Australia ample. In event detection, events prior to the start of an obser- are stored centrally in the VetCompass repository. vation may be missed or are uncertain, which means that the First NLP steps in this direction were described in first appearance of a diagnosis in the clinical record may not coincide with the occurrence of the disease. Similarly, key the invited talk at the Clinical NLP 2016 workshop events after the end of the observation may be missing (e.g. (Baldwin, 2016). death, when it occurred in another institution). what is noted down. Cases of more serious illness is present in social media with a known identity. may be more accurately documented as a result of More general examples of dual use are when the clinician’s bias (increased attention) and patient’s NLP tools are used to analyze clinical notes with recall bias. On the other hand, the cases of stig- a goal of determining individuals’ insurability and matized diseases may include suppressed informa- employability. tion. In the case of traffic injuries, documentation may even be distorted to avoid legal consequences 4 Conclusion (Indrayan, 2013). In this paper, we reviewed some challenges that We need to be aware that clinical notes may re- we believe are central to the work in clinical NLP. flect health disparities. These can originate from Difficult access to data due to privacy concerns has prejudices held by healthcare practitioners which been an obstacle to progress in the field. We have may impact patients’ perceptions; they can also discussed how the protection of privacy through originate from communication difficulties in the sanitization measures and the requirement for in- case of ethnic differences (Zestcott et al., 2016). formed consent may affect the work in this do- Finally, societal norms can play a role. Brady et al. main. Perhaps, it is time to rethink the right to pri- (2016) find that obesity is often not documented vacy in health in the light of recent work in ethics equally well for both sexes in weight-addressing of big data, especially its uneasy relationship to clinics. Young males are less likely to be recog- the right to science, i.e. being able to benefit nized as obese, possibly due to societal norms see- from science and participate in it (Tasioulas, 2016; ing them as “stocky” as opposed to obese. Unless Verbeek, 2014). We also touched upon possible we are aware of such bias, we may draw premature sources of bias that can have an effect on the ap- conclusions about the impact of our results. plication of NLP in the health domain, and which It is clear that during processing of clinical texts, can ultimately lead to unfair or harmful treatment. we should strive to avoid reinforcing the biases. It is difficult to give a solution on how to actually Acknowledgments reduce the reporting bias after the fact. One pos- sibility might be to model it. If we see clinical re- We would like to thank Madhumita and the anony- ports as noisy annotations for the patient story in mous reviewers for useful comments. Part of this which information is left-out or altered, we could research was carried out in the framework of the try to decouple the bias from the reports. Inspira- Accumulate IWT SBO project, funded by the gov- tion could be drawn, for example, from the work ernment agency for Innovation by Science and on decoupling reporting bias from annotations in Technology (IWT). visual concept recognition (Misra et al., 2016). Observational bias Although variance in health References outcome is affected by social, environmental and [Abbasi et al.2014] Ahmed Abbasi, Donald Adjeroh, behavioral factors, these are rarely noted in clini- Mark Dredze, Michael J. Paul, Fatemeh Mariam cal reports (Kaplan et al., 2014). The bias of miss- Zahedi, Huimin Zhao, Nitin Walia, Hemant Jain, ing explanatory factors because they can not be Patrick Sanvanson, Reza Shaker, et al. 2014. Social identified within the given experimental setting is media analytics for smart health. IEEE Intelligent Systems, 29(2):60–80. also known as the streetlight effect. In certain cases, we could obtain important prior knowledge [Afzal et al.2014] Zubair Afzal, Ewoud Pons, Ning (e.g. demographic characteristics) from data other Kang, Miriam C.J.M. Sturkenboom, Martijn J. than clinical notes. Schuemie, and Jan A. Kors. 2014. ContextD: an al- gorithm to identify contextual properties of medical Dual use We have already mentioned linking terms in a Dutch clinical corpus. BMC Bioinformat- personal health information from online texts to ics, 15(1):373. clinical records as a motivation for exploring sur- [Agus2016] David B. Agus. 2016. Give Up Your Data rogate data sources. However, this and many other to Cure Disease, The New York Times, February 6. applications also have potential to be applied in https://goo.gl/0REG0n. both beneficial and harmful ways. It is easy to [Albrightetal.2013] D. Albright, A. Lanfranchi, imagine how sensitive information from clinical A. Fredriksen, W. F. Styler, C. Warner, J. D. Hwang, notes can be revealed about an individual who J. D. Choi, D. Dligach, R. D. Nielsen, J. Martin, W. Ward, M. Palmer, and G. K. Savova. 2013. [Demner-Fushman et al.2009] Dina Demner-Fushman, Towards comprehensive syntactic and semantic Wendy W. Chapman, and Clement J. McDonald. annotations of the clinical narrative. Journal of 2009. What can natural language processing do for the American Medical Informatics Association, clinical decision support? Journal of Biomedical 20(5):922–930. Informatics, 42(5):760–772. [Amblard et al.2014] Maxime Amblard, Kar¨en Fort, [Duquenoyet al.2008] Penny Duquenoy, Carlisle Michel Musiol, and Manuel Rebuschi. 2014. George, and Anthony Solomonides. 2008. Con- L’impossibilit´e de l’anonymat dans le cadre de sidering something ELSE: Ethical, legal and l’analyse du discours. In Journee´ ATALA ethique´ socio-economic factors in medical imaging and et TAL. medical informatics. Computer Methods and Programs in Biomedicine, 92(3):227–237. [Baldwin2016] Timothy Baldwin. 2016. Vet- Compass: Clinical Natural Language Processing [Dwork and Pottenger2013] Cynthia Dwork and Re- for Animal Health. Clinical NLP 2016 keynote. becca Pottenger. 2013. Toward practicing privacy. https://goo.gl/ScGFa2. Journal of the American Medical Informatics Asso- ciation, 20(1):102–108. [Barzilay2016] Regina Barzilay. 2016. How NLP can help cure cancer? NAACL’16 keynote. [Fairfield and Shtein2014] Joshua Fairfield and Hannah https://goo.gl/hi5nrq. Shtein. 2014. Big data, big problems: Emerging [Berman2002] Jules J. Berman. 2002. Confidential- issues in the ethics of data science and journalism. ity issues for medical data miners. Artificial Intel- Journal of Mass Media Ethics, 29(1):38–51. ligence in Medicine, 26(1):25–36. [Fanetal.2011] Jung-wei Fan, Rashmi Prasad, [Bird etal.2016] Sarah Bird, Solon Barocas, Kate Romme M. Yabut, Richard M. Loomis, Daniel S. Crawford, Fernando Diaz, and Hanna Wallach. Zisook, John E. Mattison, and Yang Huang. 2011. 2016. Exploring or Exploiting? Social and Ethical Part-of-speech tagging for clinical text: wall or Implications of Autonomous Experimentation in AI. bridge between institutions. In AMIA Annual In Workshop on Fairness, Accountability, and Trans- Symposium Proceedings. parency in Machine Learning. [Friedmanetal.2013] Carol Friedman, Thomas C. [Bradyetal.2016] Cassandra C. Brady, Vidhu V. Rindflesch, and Milton Corn. 2013. Natural Thaker, Todd Lingren, Jessica G. Woo, Stephanie S. language processing: state of the art and prospects Kennebeck, Bahram Namjou-Khales, Ashton for significant progress, a workshop sponsored Roach, Jonathan P. Bickel, Nandan Patibandla, by the National Library of Medicine. Journal of Guergana K. Savova, et al. 2016. Suboptimal Clini- Biomedical Informatics, 46(5):765–773. cal Documentation in Young Children with Severe Obesity at Tertiary Care Centers. International [Goldacre2014] Ben Goldacre. 2014. The NHS plan to Journal of Pediatrics, 2016. share our medical data can save lives but must be done right. https://goo.gl/MH2eC0. [Chapman et al.2011] Wendy W. Chapman, Prakash M. Nadkarni, Lynette Hirschman, Leonard W. [Goodman2016] Bryce W. Goodman. 2016. A Step To- D’Avolio, Guergana K. Savova, and Ozlem¨ Uzuner. wards Accountable Algorithms?: Algorithmic Dis- 2011. Overcoming barriers to NLP for clinical text: crimination and the European Union General Data the role of shared tasks and the need for additional Protection. In NIPS Symposium on Machine Learn- creative solutions. Journal of the American Medical ing and the Law. Informatics Association, 18(5):540–543. [Hardt et al.2016] Moritz Hardt, Eric Price, and Nati [Cohen and Demner-Fushman2014] Kevin Bretonnel Srebro. 2016. Equality of opportunity in supervised Cohen and Dina Demner-Fushman. 2014. Biomed- learning. In NIPS. ical natural language processing. John Benjamins Publishing Company. [Hersh et al.2013] William R. Hersh, Mark G. Weiner, Peter J. Embi, Judith R. Logan, Philip R.O. Payne, [De Mazancourt et al.2015] Hugues De Mazancourt, Elmer V. Bernstam, Harold P. Lehmann, George Alain Couillault, Gilles Adda, and Ga¨elle Recourc´e. Hripcsak, Timothy H. Hartzog, James J. Cimino, 2015. Faire du TAL sur des donn´ees personnelles : and Joel H. Saltz. 2013. Caveats for the use of oper- un oxymore? In TALN 2015. ational electronic health record data in comparative effectiveness research. Medical care, 51(8 0 3). [Deleger et al.2013] Louise Deleger, Katalin Molnar, Guergana Savova, Fei Xia, Todd Lingren, Qi Li, [HovyandSpruit2016] Dirk Hovy and Shannon L. Keith Marsolo, Anil Jegga, Megan Kaiser, Laura Spruit. 2016. The social impact of natural lan- Stoutenborough, and Imre Solti. 2013. Large- guage processing. In Proceedings of the 54th An- scale evaluation of automated clinical note de- nual Meeting of the Association for Computational identification and its impact on information extrac- Linguistics (Volume 2: Short Papers), pages 591– tion. Journal of the American Medical Informatics 598, Berlin, Germany, August. Association for Com- Association, 20(1):84. putational Linguistics. [Hripcsak et al.1995] George Hripcsak, Carol Fried- of recent research. Yearbook of Medical Informatics, man, Philip O. Alderson, William DuMouchel, 35:128–44. StephenB. Johnson, and PaulD. Clayton. 1995. Un- locking clinical data from narrative reports: a study [Misra et al.2016] Ishan Misra, C. Lawrence Zitnick, of natural language processing. Annals of internal Margaret Mitchell, and Ross B. Girshick. 2016. medicine, 122(9):681–688. Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels. In [Huser and Cimino2014] Vojtech Huser and James J. The IEEE Conference on Computer Vision and Pat- Cimino. 2014. Don’t take your EHR to heaven, tern Recognition. donate it to science: legal and research policies for EHR post mortem. Journal of the American Medical [Mittelstadt and Floridi2016] Brent Daniel Mittelstadt Informatics Association, 21(1):8–12. and Luciano Floridi. 2016. The ethics of big data: [Indrayan2013] Abhaya Indrayan. 2013. current and foreseeable issues in biomedical con- Varieties of bias to guard against. texts. Science and engineering ethics, 22(2):303– https://goo.gl/SqnuZY. 341. [Johnson et al.2016] Alistair EW Johnson, Tom J. Pol- [Murdoch and Detsky2013] Travis B. Murdoch and Al- lard, Lu Shen, Li-wei H. Lehman, Mengling Feng, lan S. Detsky. 2013. The inevitable application of Mohammad Ghassemi, Benjamin Moody, Peter big data to health care. JAMA, 309(13):1351–1352. Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016. MIMIC-III, a freely accessible critical care [Nyr´en et al.2014] Olof Nyr´en, Magnus Stenbeck, and database. Scientific data, 3. Henrik Gr¨onberg. 2014. The European Parliament proposal for the new EU General Data Protection [Kamarinou et al.2016] Dimitra Kamarinou, Christo- Regulation may severely restrict European epidemi- pher Millard, and Jatinder Singh. 2016. Machine ological research. European Journal of Epidemiol- Learning with Personal Data. Queen Mary School ogy, 29(4):227–230. of Law Legal Studies Research Paper, (247/2016). [O’Doherty et al.2016] Kieran C. O’Doherty, Emily [Kaplan et al.2014] Robert M. Kaplan, David A. Cham- Christofides, Jeffery Yen, Heidi Beate Bentzen, bers, and Russell E. Glasgow. 2014. Big data Wylie Burke, Nina Hallowell, Barbara A. Koenig, and large sample size: a cautionary note on the po- and Donald J. Willison. 2016. If you build it, tential for bias. Clinical and translational science, they will come: unintended future uses of organised 7(4):342–346. health data collections. BMC Medical Ethics, 17(1). [Kim etal.2015] Youngjun Kim, Ellen Riloff, and John F. Hurdle. 2015. A study of concept extrac- [Padrezetal.2015] Kevin A. Padrez, Lyle Ungar, tion across different types of clinical notes. In AMIA Hansen Andrew Schwartz, Robert J. Smith, Shawn- Annual Symposium Proceedings, volume 2015, page dra Hill, Tadas Antanavicius, Dana M. Brown, 737. American Medical Informatics Association. Patrick Crutchley, David A. Asch, and Raina M. Merchant. 2015. Linking social media and medical [Kramer et al.2014] Adam D.I. Kramer, Jamie E. Guil- record data: a study of adults presenting to an aca- lory, and Jeffrey T. Hancock. 2014. Experimen- demic, urban emergency department. BMJ quality tal evidence of massive-scale emotional contagion & safety. through social networks. Proceedings of the Na- tional Academy of Sciences, 111(24):8788–8790. [Polonetsky et al.2016] Jules Polonetsky, Omer Tene, and Kelsey Finch. 2016. Shades of Gray: [Lawrence2016] Neil Lawrence. 2016. Data Seeing the Full Spectrum of Practical Data De- Analysis, NHS and Industrial Partners. identification. Santa Clara Law Review, 56(3). https://goo.gl/rRIcu5. [Pradhan et al.2013] Sameer Pradhan, Noemie Elhadad, [Lipton et al.2016] Zachary C. Lipton, David Kale, and Brett R. South, David Martinez, Amy Vogel, Hanna Randall Wetzel. 2016. Modeling Missing Data in Suominen, Wendy W. Chapman, and Guergana Clinical Time Series with RNNs. arXiv preprint Savova. 2013. Task 1: Share/clef ehealth evaluation arXiv:1606.04130. lab. In Online Working Notes of CLEF. [Malin et al.2013] Bradley A. Malin, Khaled El Emam, and Christine M. O’Keefe. 2013. Biomedical [Rebholz-Schuhmannet al.2011] Dietrich Rebholz- data privacy: problems, perspectives, and recent ad- Schuhmann, Antonio Jimeno Yepes, Chen Li, et al. vances. Journal of the American Medical Informat- 2011. Assessment of NER solutions against the ics Association, 20(1):2–6. first and second CALBC Silver Standard Corpus. Journal of Biomedical Semantics, 2(5):S11. [Meystre et al.2008] St´ephane M. Meystre, Guergana K. Savova, Karin C. Kipper-Schuler, John F. Hurdle, [Rindfleisch1997] Thomas C. Rindfleisch. 1997. Pri- et al. 2008. Extracting information from textual vacy, information technology, and health care. Com- documents in the electronic health record: a review munications of the ACM, 40(8):92–100. [Saeed et al.2011] Mohammed Saeed, Mauricio Villar- [Zestcott et al.2016] Colin A. Zestcott, Irene V. Blair, roel, Andrew T. Reisner, Gari Clifford, Li-Wei and Jeff Stone. 2016. Examining the presence, con- Lehman, George Moody, Thomas Heldt, Tin H. sequences, and reduction of implicit bias in health Kyaw, Benjamin Moody, and Roger G. Mark. 2011. care: A narrative review. Group Processes & Inter- Multiparameter Intelligent Monitoring in Intensive group Relations. Care II (MIMIC-II): a public-access intensive care unit database. Critical care medicine, 39(5):952.

[Schmidt et al.2015] Morten Schmidt, S.A. Schmidt, Jakob Lynge Sandegaard, Vera Ehrenstein, Lars Ped- ersen, and Henrik Toft Sørensen. 2015. The danish national patient registry: a review of content, data quality, and research potential. Clinical Epidemiol- ogy, 7(449):e490.

[Schroeder2014] Ralph Schroeder. 2014. Big data and the brave new world of social media research. Big Data & Society, 1(2):2053951714563194.

[Styler IV et al.2014] William Styler IV, Steven Bethard, Sean Finan, Martha Palmer, Sameer Prad- han, Piet de Groen, Brad Erickson, Timothy Miller, Chen Lin, Guergana Savova, and James Pustejovsky. 2014. Temporal annotation in the clinical domain. Transactions of the Association for Computational Linguistics, 2:143–154.

[Tasioulas2016] John Tasioulas. 2016. Van Hasselt Lecture 2016: Big Data, Human Rights and the Ethics of Scientific Research. https://goo.gl/QREHUN.

[Taylor et al.2017] Linnet Taylor, Luciano Floridi, and Bart van der Sloot. 2017. Group Privacy: New Challenges of Data Technologies. Springer Interna- tional Publishing.

[Tene2011] Omer Tene. 2011. Privacy: The new gen- erations. International Data Privacy Law, 1(1):15– 27.

[Terry2012] Nicolas P. Terry. 2012. Protecting pa- tient privacy in the age of big data. University of Missouri-Kansas City Law Review, 81(2).

[Uzuner et al.2011] Ozlem¨ Uzuner, Brett R. South, Shuying Shen, and Scott L. DuVall. 2011. 2010 i2b2/va challenge on concepts, assertions, and rela- tions in clinical text. Journal of the American Medi- cal Informatics Association, 18(5):552–556.

[vanStaaetal.2016] Tjeerd-Pieter van Staa, Ben Goldacre, Iain Buchan, and Liam Smeeth. 2016. Big health data: the need to earn public trust. BMJ, 354:i3636.

[Velupillai et al.2015] Sumithra Velupillai, D. Mowery, Brett R. South, Maria Kvist, and Hercules Dalianis. 2015. Recent advances in clinical natural language processing in support of semantic analysis. Year- book of Medical Informatics, 10:183–193.

[Verbeek2014] Peter-Paul Verbeek. 2014. Op de vleugels van Icarus. Lemniscaat.