A Reference Architecture for Secure Medical Devices Steven Harp, Todd Carpenter, and John Hatcliff

FEATURE © Copyright AAMI 2018. Single user license only. Copying, networking, and distribution prohibited. A Reference Architecture for Secure Medical Devices Steven Harp, Todd Carpenter, and John Hatcliff

Abstract and users cannot assume that medical Steven Harp, is a distinguished We propose a reference architecture aimed at devices will operate in a benign security engineer at Adventium Labs in supporting the safety and security of medical environment. Any device that is capable of Minneapolis, MN. Email: steven. devices. The ISOSCELES (Intrinsically connecting to a network or physically [email protected] Secure, Open, and Safe Cyber-Physically exposes any sort of data port is potentially at Enabled, Life-Critical Essential Services) risk. How should we think about and Todd Carpenter, is chief engineer architecture is justified by a collection of design manage risk in this context? at Adventium Labs in Minneapolis, principles that leverage recent advances in This question has been explored exten- MN. Email: todd.carpenter@ software component isolation based on sively.3,4 Special publications from the adventiumlabs.com hypervisor and other separation technologies. National Institute of Standards and Technol- The instantiation of the architecture for ogy (e.g., 800-39) provide a conceptual risk John Hatcliff, PhD, is a particular medical devices is supported by a management framework.5,6 AAMI TIR577 distinguished professor at Kansas development process based on Architecture describes how security risk management State University in Manhattan, KS. Analysis and Design Language. The architec- can be integrated with safety risk manage- Email: [email protected] ture models support safety and security ment (e.g., as addressed in ISO 14971 and analysis as part of a broader risk management specifically for medical devices in IEC framework. The models also can be used to 80001). The UL 2900 series of standards derive skeletons of the device software and to provides cybersecurity requirements for configure the platform’s separation policies and medical devices.8 an extensive set of services. We are developing In brief, a threat source initiates a threat prototypes of the architecture and example event, which may exploit a device vulnera- medical device instantiations on low-cost bility, causing an adverse impact. The boards that can be used in product solutions. impact might affect the mission of the The prototype and supporting development device, the end user, or the organization and assurance artifacts are being released that created or operated the device. In this under an open-source license. framework, risk is based on the likelihood of a threat event occurring and amplified by A reference architecture is a domain-spe- the potential loss (adverse impact) should cific design template for the structure of a the event occur. class of systems as a set of constituent parts Threat sources have capabilities, intents, with special roles and communications and often preferences for particular targets. patterns. The reference architecture Automated threats, such as malware, can be discussed in this article addresses the class indiscriminate in their target selection and of small bedside medical devices (e.g., do not care whether they are exploiting a infusion pumps, electrocardiographs, personal system or a hospital. Threats ventilators). The architecture also aims to targeting particular organizations also are support interoperability interfaces through well documented.9–11 The intent of the threat compatibility with the ASTM F2761 Inte- source is frequently financial, as exemplified grated Clinical Environment (ICE)1 or the by the recent spate of ransomware (e.g., IEEE 11073 service-oriented device connec- WannaCry).12 In addition to direct extortion, tivity standards.2 Any such architecture the medical information that is present in must address a wide range of requirements, some devices can have indirect value, such including safety, cost, maintainability, and as protected health information or personally performance. It also must address an identifiable information being leveraged to unpleasant modern reality: Manufacturers steal funds or illegally acquire drugs.

www.aami.org/bit 357 FEATURE © Copyright AAMI 2018. Single user license only. Copying, networking, and distribution prohibited.

A special type of impact needs to be instantiate it can reduce the attack surface. considered for medical devices: The lives Security controls in the device can further and health of patients may depend on them. reduce an attacker’s ability to exploit certain Security and safety are intertwined in such outstanding vulnerabilities and minimize systems, and failure to manage security the impact of successful exploitation of risks may pose safety risks.7 Beyond being others. an end target of an attack, a medical device The remainder of this article presents may represent a stepping stone for a larger principles driving the design and instantia- campaign. Exploitation of a vulnerable tion of Intrinsically Secure, Open, and Safe device may permit access to other clinical or Cyber-Physically Enabled, Life-Critical financial systems by allowing the attacker to Essential Services (ISOSCELES)—a safe and leverage trust information or to extract secure architecture for medical devices. credentials that allow the device to connect to these other systems. An example of a Architecture successful attack of this sort is documented The ISOSCELES reference architecture is in the MEDJACK report on blood-gas-ana- specified with tiered requirements: lyzers attacked in 2015–16.9 1. Platform core requirements for hard- A noteworthy long-term risk is when ware, system software, and services malware authors or researchers automate supporting the medical application. the attacks. This could result in multiple, 2. Platform design requirements derived nearly simultaneous malfunctions, similar from the platform core requirements Security controls in to what occurred with the WannaCry attack and refined to a specific design. the device can further in May 2017. With automation, botnet 3. Device design requirements specific to a reduce an attacker’s controllers could use the devices to attack particular medical device being devel- ability to exploit other devices on healthcare facility net- oped (e.g., an infusion pump). certain outstanding works. Decades ago, the attackers were The first two platform tiers drive the vulnerabilities and people who actively attacked your computer. reference architecture. These 134 require- minimize the impact of Unfortunately, automated botnets—mil- ments are intended to be suitable for a successful exploitation of lions of already-compromised machines broad family of potential medical devices. others. trying to grow their reach—are now the Figure 1 illustrates various components of prevalent attackers. a hypothetical medical device design based Recent Department of Homeland Security on this reference architecture. The exact advisories documented hard-coded pass- medical applications and services needed words and credentials in medical devices.13,14 may vary according to the device. A hard- Vulnerabilities such as these make it easier ware layer (bottom of figure) contains for malware (e.g., botnets, ransomware) to processors and peripheral devices that gain a foothold on the device. support security mechanisms. A low-level By itself, a medical device architecture separation layer of software uses these cannot address all aspects of risk. Security mechanisms to isolate all other software analyst Sergey Lozhkin noted that although components from each other, except for vulnerable devices were revealed in penetra- specifically allowed interactions. The tion testing, “the problem is not only one of ISOSCELES platform services satisfy typical weak protection of medical equipment, it needs of the top-layer medical applications has a much wider scope. The whole IT that provide medical diagnostics and infrastructure of modern hospitals is not therapies. The design of a specific device properly organized and protected, and the might incorporate only the required ele- problem persists worldwide."10 ments needed for that class of device. Until the day when software can be automatically and completely validated, we Architectural Principles must assume that some vulnerabilities The ISOSCELES reference architecture remain even after formal design and attempts to promote basic principles that extensive testing. However, a suitable provide security and safety. These principles reference architecture and tools to help and their motivations are described below.

358 Biomedical Instrumentation & Technology September/October 2018 FEATURE © Copyright AAMI 2018. Single user license only. Copying, networking, and distribution prohibited.

Use Strong Time and Space Separation Many modern systems provide a basic security perimeter but lack internal security barriers, making it easy for attackers to access arbitrary information within the device, and even to control the device, after they have breached the perimeter. Strong barriers between software components help contain faults and attacks from propagating to safety-critical components. For example, if an infusion pump requires a network connection (e.g., to maintain current drug libraries), the network stack and code associated with retrieving the drug library should be wholly separate from the software that interprets the contents of the Figure 1. The layered ISOSCELES (Intrinsically Secure, Open, and Safe Cyber-Physically Enabled, drug library. In turn, that function also Life-Critical Essential Services) architecture. Abbreviations used: CPU, central processing unit; HMI, should be entirely separate from the function human-machine interface; HSM, hardware security module; ICE, integrated clinical environment; that controls the rate at which the drug is MMU, memory management unit; OEM, original equipment manufacturer; RAM, random-access memory; TPM, trusted platform module. pumped. In safety architectures, each unit of separation is called a partition.15 Separation spatial and temporal separation. It is the one kernels and some real-time operating program on the device that has direct ability systems (OSs) provide strong separation. to control the protection facilities provided Historically, application software on medical by the hardware. It is intentionally as small devices has been built either on minimal and simple as possible to increase its OSs that fail to provide strong separation or trustworthiness. without the benefit of any OS at all. Key hardware features that support this Spatial separation refers to barriers role are hardware memory management between the memory regions and addressa- units (MMUs) and input/output MMUs ble peripherals assigned to different (IOMMUs). While standard on desktop components. A component operating in one systems, these features historically have not partition should not be able to directly been present on inexpensive or low-power reference the memory or input/output (IO) microprocessors. For example, ARM micro- devices assigned exclusively to another controllers prior to the ARMv3 family lacked component. Failure to do so would allow a integral MMUs, and Intel Atom processors defective or compromised component to prior to 2016 lacked IOMMUs. These classes affect the missions of other components. of processors are present in many fielded Temporal separation refers to the inability medical devices, providing opportunities for of one component to affect or measure the attacks that begin in vulnerable processes to time intervals during which another compo- escape and tamper with the resources of nent accesses a resource. For example, a other processes. However these features defective or compromised component have recently appeared in affordable embed- should not be able to dominate a central ded CPUs in both ARM and x86 processor processing unit (CPU) and thereby keep architectures. another component from performing its An MMU is programmable hardware that mission in a timely manner. Programs controls the physical memory that a given running in one partition should be oblivious process may access while providing the to the operation and resource usage of process with the appearance of a well-or- programs in other partitions, except as dered “virtual address space.” It also can explicitly intended. mark a section of memory as read-only. In ISOSCELES addresses these requirements some cases, it may be able to prevent the by specifying a low-level software kernel processor from executing instructions in dedicated to establishing and maintaining certain areas of memory. If a process

www.aami.org/bit 359 FEATURE © Copyright AAMI 2018. Single user license only. Copying, networking, and distribution prohibited.

attempts to address memory outside of its privilege, is a cornerstone of security. In the partition or in ways not permitted by the context of the reference architecture, it programmed policy, then access is blocked means that any given component should be and the CPU will trap to a handler to deal granted only the privileges required to with the fault. perform its mission. Privileges here include Recent Meltdown and Spectre attacks have the ability to access resources such as demonstrated several flaws in super-scalar memory, arbitrary processor instructions, processors that violate this protection; the and peripheral devices. Unfortunately, processors specified by ISOSCELES, such as embedded systems based on commodity OSs the ARM A53, do not currently exhibit these (e.g., Linux, Windows) are all too often issues. The IOMMU performs a similar implemented with “root” access, which function for peripheral devices. Failure to means that as soon as the perimeter is isolate the IO space opens a door for defec- breached, the attacker can easily access tive or compromised software components anything. to breach their partitions and interfere with For our reference architecture, the hard- other components and external devices. For ware layer assists in this goal in ways beyond example, a process designed to check for the separation mechanisms described above. network updates might be Modern microprocessors offer The principle of compromised by a mali- multiple operation modes, minimal privilege, or ciously crafted packet, and including a supervisory mode least privilege, is a injected code could modify with higher privilege and a user cornerstone of security. the GPIO (general-purpose mode with lower privilege. The In the context of the input/output) pins to turn on supervisory mode is allowed to reference architecture, a pump or change the power execute all of the processor it means that any given state of the device. instruction set and is permitted component should Prototypes of ISOSCELES to program the MMU and be granted only the have used both the seL4 IOMMU. The user mode privileges required to microkernel and the Xen cannot change the memory perform its mission. hypervisor for separation.16,17 management and is blocked A microkernel is like an OS from using privileged instruc- kernel but is devoid of the usual panoply of tions, such as controlling interrupt handling device drivers, network stacks, and file or processor faults. system drivers. It specializes in strongly ISOSCELES stipulates that only the separating memory and processor time into separation layer operates in the most partitions, along with minimal support for privileged mode. All other software compo- interpartition communication (IPC). The nents run in an unprivileged mode and rely seL4 microkernel is distinguished by a on the separation layer to delegate to them formal proof of its correctness on certain only those special privileges needed for their processor architectures.18 Hypervisors offer missions. Although something like this the additional ability to provide each parti- “kernel-versus-userland” distinction exists in tion an emulated environment, giving the general purpose OSs, systems based on component the illusion that it is running on microkernels extend this demotion to its own hardware. This permits partitions to components such as device drivers, file run general-purpose OSs (e.g., Linux, system drivers, and protocol stacks. These Berkeley Software Distribution, Microsoft often are relatively complex components, Windows). So-called type 1 hypervisors (e.g., and their failure should not jeopardize other Xen) support both low-level separation and parts of the overall system. For example, the the emulation capabilities. compromise of a network device driver in a monolithic kernel would threaten every Minimize Privilege aspect of the system’s operation. (The driver Going hand-in-hand with separation, the would run with full privileges in the kernel principle of minimal privilege, or least memory space.) In a microkernel, the device

360 Biomedical Instrumentation & Technology September/October 2018 FEATURE © Copyright AAMI 2018. Single user license only. Copying, networking, and distribution prohibited. driver is just another user-level component, with each other. Modern medical devices also and compromise can only affect other must communicate with external systems. components that interact with it. This interaction carries inherent risk, as one party must trust another to provide valid and Minimize Complexity timely messages. The safety and liveness of The trustworthiness of software tends to critical processes depends on this trust being correlate inversely with complexity; the more warranted. complex code is, the less trustworthy it is. The reference architecture addresses this Many factors influence this, including the challenge in several ways. First, it restricts number of attack surfaces, the difficulty of internal communications by policy to correctly developing code, and the effort predefined channels. A defective or compro- required to verify and validate desired mised component is incapable of opening properties. The reference architecture new channels to other components. These encourages the construction of components channels are specified and checked in the that are as small and simple as possible. design’s AADL model, then translated to the Many application components can perform enforcement mechanism in the microkernel their primary functions without all of the or hypervisor. accouterments of typical applications run- External communications are further ning in a general purpose OS. regulated by an integral Example “extra baggage” The reference architecture firewall that is independent of includes general purpose C allows designers to the internal communicating and other standard software place less trustworthy component. A default-deny libraries, arbitrary software components in a firewall policy ensures that the languages, and full access to subordinate role to more device only communicates with all system calls. The Linux 2.6 trustworthy peers. a “whitelist” of approved kernel has 338 system calls.19 parties. COTS OSs often ship The Standard C library adds with open policies to ease 500 kB (and often much more) binary code.20 integration and development overheads; The presence of this code in the address these policies must be explicitly tightened space of a component provides an attack down. surface that facilitates remote code execution The framework encourages communica- using techniques such as return-oriented tions patterns that limit the degree of trust programming.21 required. In many cases, the relationships In contrast, ISOSCELES specifies a between components can be ordered by an minimal component runtime environment evaluation of their relative trustworthiness or that includes only essential IPC primitives. by their relative criticality. For example, some Components can leverage the architecture’s components are measurably more complex service layer using IPC and avoid including than others or contain third-party code service layer code in their own memory libraries that cannot be fully evaluated. partitions. Using this methodology, compo- Components that communicate directly with nents (in the microkernel realization of external networks might be less trustworthy, ISOSCELES) can be shrunk from hundreds as they could be compromised by external to tens of kilobytes and only expose a very attack. Safety-related components should be narrow IPC attack surface. The Architecture treated as more critical than other operational Analysis and Design Language (AADL) components and should not directly depend modeling tools (see leverage models of on those external-facing components.15 The correctness section) help specify and reference architecture allows designers to construct appropriately minimal designs. place less trustworthy components in a subordinate role to more trustworthy peers. Manage Trust Relations Designs may select communications Even in the simplest systems, a device’s mechanisms on a per-channel basis to software components must communicate provide the desired separation, including:

www.aami.org/bit 361 FEATURE © Copyright AAMI 2018. Single user license only. Copying, networking, and distribution prohibited.

• Synchronous interpartition remote and the configuration of the device and procedure calls. its components. • Message queues. • The authentication and authorization • Shared memory marked as read-only to services control the access that users are some components. granted to particular device capabilities. • Asynchronous event signaling. Users may be granted roles (e.g., clini- cian, administrator) that, through policy, Leverage Common Services dictate the operations they may perform The service layer shown in Figure 1 con- on the device. tains several components that support • A user interface service provides display common medical device needs. A medical and human input devices to allow device device based on the reference architecture users to monitor and control it. would include implementations of these • The key service controls cryptographic services; device designers simply select operations, as discussed below. those required, avoiding the need to fully develop them in house. Leverage Cryptography for • A logging service can direct device log Confidentiality and Integrity Software engineering messages to a local persistent store and/ Medical devices require information protec- has made progress in or over the network to a remote logging tion to preserve confidentiality and integrity. machine-generated host. This is vital to several tasks for the medical proofs from formal • The time service provides current device: models of systems. synchronized time, as well as precision • Verifying software before executing it When the system delayed signals, to trigger timed actions. • Transferring sensitive data to or from implementation is • A storage service manages the persis- persistent storage generated from these tent storage devices on the device, which • Validating software updates for the models, in part or in may include multiple encrypted logical device whole, developers partitions. • Communicating with external services can reap additional • The cyber-physical abstraction layer (e.g., pharmacies) advantages of reduced component isolates the medical applica- • Communicating with peer devices in an engineering effort and a tions from directly interfacing with ICE correct design. medical sensors and actuators. This Cryptography plays a central role in adds flexibility by allowing devices to satisfying these requirements. Any compo- map logical sensor or actuator roles to nent in a design may include a cryptographic different physical devices, thereby library to satisfy its information protection reducing the verification and validation needs. This includes standard services (e.g., impacts if a sensor is updated. It also storage, update, ICE) and device-specific can support networked devices. For components (e.g., a drug library component example, a future infusion pump might that must synchronize with an external leverage a blood oxygen sensor that pharmacy). The reference architecture calls resides on a different medical device in for a cryptographic key management service the same ICE.22 (CKMS), which provides a mechanism to • A firewall service controls access to data control cryptographic materials such as keys, networks connected to the device. shared secrets, and certificates. This enforces Internal components that need network a policy that controls which operations may access do so through this internal be performed by which components on any firewall, which exposes a single external given cryptographic datum. The CKMS may address. in turn use a hardware security module, if • An update service manages secure available (e.g. a trusted platform module). device firmware upgrades in the field, Such devices are designed to be tamper which may occur over the network or resistant, and keys stored within them are through an attached storage device. protected even from attackers who attempt • Device control and device configuration to disassemble a device. services support the operating life cycle

362 Biomedical Instrumentation & Technology September/October 2018 FEATURE © Copyright AAMI 2018. Single user license only. Copying, networking, and distribution prohibited.

Leverage Models of Correctness Conclusion While system testing is the common The ISOSCELES medical device reference approach for verifying the correctness of an architecture currently is under development. implementation, analytical proofs of correct- When completed, it will provide FDA-re- ness are even more valuable. Software quired documentation and an exemplar engineering has made progress in device to demonstrate the use of the architec- machine-generated proofs from formal ture. One prototype is based on the Xen models of systems. When the system hypervisor16 and an embedded computer implementation is generated from these using an ARM CPU (dual-core Cortex-A7) models, in part or in whole, developers can reap additional advantages of reduced engineering effort and a correct design. The ISOSCELES reference architecture is represented in an AADL model.23,24 This language has well-defined semantics for hardware and software components of a system, with attributes that support reasoning about system-level characteristics of a design. Figure 2 shows example communications flows for part of a design. These flow annotations specify the allowed information flows between particular components operating in different partitions (e.g. from Figure 2. AADL models of interaction between components in an ISOSCELES (Intrinsically Secure, diagnostics to interoperability services), then Open, and Safe Cyber-Physically Enabled, Life-Critical Essential Services) design. Abbreviation out to network services (e.g., electronic used: OEM, original equipment manufacturer. health records). Flows can be checked at design time against a security policy that might forbid information flowing directly from unregulated components, such as the interoperability services flowing to the class III therapy services. In addition, tools convert these modeled flows into the runtime communications tables, which provides the tie between the model that was analyzed and the actual implementation. The models also support temporal analysis. The model may specify tolerances on latencies between events or the amount of jitter tolerated in a cyclic schedule. Tools can examine models and evaluate whether the process schedule will meet all of the requirements, accounting for factors such as interpartition transition overhead. The tools also can generate the runtime schedules. AADL also supports modeling features that permit reasoning about faults and fault management strategies.25 Formalizing the assumptions about failure modes and effects Figure 3. A laboratory prototype of a patient-controlled analgesia (PCA) pump demonstrating analysis means that the design’s hazards can ISOSCELES (Intrinsically Secure, Open, and Safe Cyber-Physically Enabled, Life-Critical Essential be assessed against fault behavior and Services) partitioning on the seL4 separation kernel. The ISOSCELES-based PCA pump software runs on the small Intel Atom embedded development board. That board is connected to modified propagation in both qualitative and quantita- surplus pump hardware to demonstrate functionality. Abbreviation used: HDMI, high-definition tive ways. multimedia interface.

system-on-chip (SoC). It uses a commodity preted as necessarily representing the official compute module and interfaces with the policies or endorsements, either expressed or electromechanical components of a decom- implied, of the DHS, U.S. government, missioned patient-controlled analgesia pump. Government of Israel, or National Cyber Another prototype runs on ARM, Intel Atom, Bureau in the Government of Israel. and AMD G-series–based SoCs running the Several artifacts are being developed in seL4 or NOVA microkernels with Genode collaboration with Food and Drug Administra- application framework (Figure 3).26 tion (FDA) engineers through the National ISOSCELES requirements, designs, example Science Foundation/FDA Scholar-in-Residence implementations, and associated development program, which is providing partial support for tools will be made available as open source this work. products. Medical device firms may use these artifacts as a starting point to develop their References proprietary designs. Original equipment 1. ASTM F2761. Medical Devices and Medical Sys- manufacturers (OEMs) will need to select a tems—Essential Safety Requirements for Equipment Original equipment hardware and separation approach based on Comprising the Patient-Centric Integrated Clinical manufacturers can use their unique product needs. Next, if not already Environment (ICE)—Part 1: General Requirements the ISOSCELES platform available, OEMs should port the chosen and Conceptual Model. West Conshohocken, PA: services on top of separation approach to their hardware and ASTM International; 2009. that separation layer particular interface devices. They can use the and refine features, ISOSCELES platform services on top of that 2. IEEE 11073-10207-2017. IEEE Health informatics— such as the specific separation layer and refine features, such as Point-of-care medical device communication Part key management the specific key management and authentica- 10207: Domain Information and Service Model for and authentication tion approaches, to satisfy their market needs. Service-Oriented Point-of-Care Medical Device Com- approaches, to satisfy ISOSCELES was developed specifically to munication. Piscataway, NJ: IEEE; 2017. their market needs. target ICEs with interoperable devices, with strong user and machine-to-machine 3. Almohri H, Cheng L, Yao D, Alemzadeh H. authentication and configurable networking On Threat Modeling and Mitigation of Medical and key management. The current form of Cyber-Physical Systems. In: The Second IEEE/ ISOSCELES targets external devices with ACM International Conference on Connected Health: modern processors, communications, and Applications, Systems and Engineering Technologies power supplies and is not intended for (CHASE), July 17–19, 2017, Philadelphia. 114–9. extremely power-constrained environments. Although the architecture principles remain 4. Martins G, Bhatia S, Koutsoukos X, et al. Towards important in those environments, the power a Systematic Threat Modeling Approach for constraints driven by the ever-shrinking Cyber-physical Systems. In: 2015 Resilience Week device volumes will challenge current (RWS). Piscataway, NJ: IEEE; 2015:1–6. technologies due to increased memory and computation requirements. 5. National Institute of Standards and Technology. Managing Information Security Risk: Organization, Acknowledgments Mission, and Information System View. Special This material is based on research sponsored Publication 800-39. Gaithersburg, MD: National by the Department of Homeland Security Institute of Standards and Technology; 2011. (DHS) Science and Technology Directorate; Homeland Security Advanced Research 6. National Institute of Standards and Technology. Projects Agency (HSARPA), Cyber Security Guide for Conducting Risk Assessments. Special Division (DHS S&T/HSARPA/CDS) Broad Publication 800-30. Gaithersburg, MD: National Area Announcement number HSHQDC-14- Institute of Standards and Technology; 2012. R-B0005; the Government of Israel; and the National Cyber Bureau in the Government of 7. AAMI TIR57. Principles for medical device infor- Israel via contract number D16PC00057. The mation security—risk management. Arlington, views and conclusions contained herein are VA: Association for the Advancement of Medical those of the authors and should not be inter- Instrumentation; 2016.

8. UL 2900. Software Cybersecurity for Network-Con- 18. Heiser G, Elphinstone K. L4 Microkernels: The nectable Products. Washington, DC: UL; 2017. Lessons from 20 Years of Research and Deploy- ment. ACM Transactions on Computer Systems. 9. TrapX. Attackers Target Blood Gas Analyzers. 2016:34(1):1–30. Available at: https://trapx.com/wp-content/up- loads/2017/08/Case_Study_TrapX_Healthcare_ 19. Linux Syscall Reference. Available at: http:// MEDJACK_BGA.pdf. Accessed Aug. 2, 2018. syscalls.kernelgrok.com. Accessed Aug. 2, 2018.

10. Lozhkin S. Hospitals are under attack in 2016. 20. Eta Labs. Comparison of C/POSIX standard Available at: https://securelist.com/blog/re- library implementations for Linux. Available at: search/74249/hospitals-are-under-attack-in-2016. www.etalabs.net/compare_libcs.html. Accessed Accessed Aug. 2, 2018. Aug. 2, 2018.

11. Brook C. Sergey Lozhkin on How He Hacked 21. Shacham H. The geometry of innocent flesh on His Hospital. Available at: https://threatpost. the bone: return-into-libc without function calls com/sergey-lozhkin-on-how-he-hacked-his-hospi- (on the x86). In: Proceedings of the 14th ACM Con- tal/116314. Accessed Aug. 2, 2018. ference on Computer and Communications Security, Oct. 29 through Nov. 2, 2007, Alexandria, VA. New 12. Fox-Brewster T. Medical Devices Hit by Ran- York: ACM; 2007:552–61. somware for the First Time in US Hospitals. Available at: www.forbes.com/sites/thomasbrew- 22. King A, Arney D, Lee I, et al. Prototyping closed ster/2017/05/17/wannacry-ransomware-hit-re- loop physiologic control with the medical device al-medical-devices. Accessed Aug. 2, 2018. coordination framework. In: Proceedings of the 32nd International Conference on Software Engineering, 13. Department of Homeland Security. Advisory (ICS- May 1–8, 2010, Cape Town, South Africa. New MA-17-250-02A): Smiths Medical Medfusion 4000 York: ACM; 2010:1–11. Wireless Syringe Infusion Pump Vulnerabilities (Update A). Available at: https://ics-cert.us-cert. 23. AS5506. Architecture Analysis & Design Language gov/advisories/ICSMA-17-250-02A. Accessed Aug. (AADL). Warrendale, PA: SAE International; 2004. 17, 2018. 24. Software Engineering Institute. Open Source 14. Department of Homeland Security. Advisory AADL Tool Environment (OSATE). Available at: (ICSMA-18-037-02): GE Medical Devices Vulnera- http://osate.org. Accessed Aug. 17, 2018. bility. https://ics-cert.us-cert.gov/advisories/ICS- MA-18-037-02. Accessed Aug. 17, 2018. 25. SAE International. SAE Architecture Analysis and Design Language (AADL) Annex Volume 1: Annex 15. Larson BR, Jones P, Zhang Y, Hatcliff J. Principles A: ARINC653 Annex, Annex C: Code Generation and Benefits of Explicitly Designed Medical Device Annex, Annex E: Error Model Annex AS5506/1A. Safety Architecture. Biomed Instrum Technol. Available at: www.sae.org/standards/content/ 2017;51(5):380–9. as5506/1a. Accessed Aug 17, 2018.

16. Xen Project. Homepage. Available at: www.xenpro- 26. Genode Labs. Documentation of the Genode OS ject.org. Accessed Aug. 2, 2018. Framework. Available at: https://genode.org/documentation/index. Accessed Aug. 2, 2018. 17. Carpenter T, Hatcliff J, Vasserman EY. A reference separation architecture for mixed-criticality medical and iot devices. In: Proceedings of the 1st ACM Workshop on the Internet of Safe Things, Delft, Netherlands, Nov. 6–8, 2017. New York: ACM; 2017.