Technical Report on the Brittle Kernel Release 2016-05

Cliff L. Bifﬂe

June 09, 2016

Contents

1 Introduction 1 1.1 What Brittle Is...... 1 1.2 What Brittle Isn’t...... 1

2 Kernel Architecture 3 2.1 Basic Concepts...... 3 2.2 Architecture Rules...... 4 2.2.1 What’s In a Kernel?...... 5 In Scope...... 5 Out Of Scope...... 5 2.3 Kernel, System, Application...... 5 2.4 A Tour of the Kernel Objects...... 6 2.4.1 Contexts...... 6 2.4.2 Memory...... 7 2.4.3 Gates...... 7 2.4.4 Interrupts...... 8 2.4.5 The Object Table...... 8 2.5 About Keys...... 9 2.5.1 Key Semantics...... 9 2.5.2 Parts of a Key...... 9 2.5.3 Brands...... 9 2.5.4 Generations and Revocation...... 10 2.6 Syscalls...... 10 2.6.1 Syscall Descriptor Convention...... 10 2.6.2 Copy Key...... 11 2.6.3 IPC...... 11 Message Descriptors...... 12 Key Maps...... 12 The Send Phase...... 12 The Receive Phase...... 13 2.7 Boot Process and Initial Environment...... 13

3 Kernel Object Reference 15 3.1 Context...... 15 3.1.1 Branding...... 15 Reply Keys...... 15 Service Keys...... 16 3.1.2 Invalidation...... 16

i 3.1.3 Methods...... 16 Read Register (1)...... 16 Write Register (2)...... 17 Read Key Register (3)...... 17 Write Key Register (4)...... 18 Read MPU Region Register (5)...... 18 Write MPU Region Register (6)...... 19 Make Runnable (7)...... 19 Get Priority (8)...... 20 Set Priority (9)...... 20 Read (Low/High) Registers (10/11)...... 20 Write (Low/High) Registers (12/13)...... 21 3.2 Gate...... 21 3.2.1 Branding...... 21 Service Keys...... 21 Transparent Keys...... 22 3.2.2 Methods...... 22 Make Client Key (1)...... 22 3.3 Interrupt...... 22 3.3.1 Branding...... 22 3.3.2 Invalidation...... 23 3.3.3 Methods...... 23 Set Target (1)...... 23 Enable (2)...... 23 3.4 Memory...... 23 3.4.1 Mappable Memory...... 24 3.4.2 Hierarchy...... 24 3.4.3 The Device Attribute...... 24 3.4.4 Branding...... 24 3.4.5 Invalidation...... 24 3.4.6 Methods...... 25 Inspect (1)...... 25 Change (2)...... 25 Split (3)...... 26 Become (4)...... 27 Peek (5)...... 28 Poke (6)...... 28 Make Child (7)...... 29 3.5 Null...... 29 3.5.1 Message Elision Rule...... 30 3.5.2 Branding...... 30 3.5.3 Methods...... 30 3.6 Object Table...... 30 3.6.1 Branding...... 30 3.6.2 Methods...... 30 Mint Key (1)...... 30 Read Key (2)...... 31 Get Kind (3)...... 31 Invalidate (4)...... 32 3.7 Slot...... 32 3.7.1 Branding...... 32 3.7.2 Methods...... 33

4 Case Study: The Serial Demo 35 ii 4.1 Tasks...... 35 4.2 Driver Operation...... 35 4.3 Startup...... 36

5 Case Study: FreeRTOS 37 5.1 Introduction...... 37 5.2 About FreeRTOS/Brittle...... 37 5.2.1 What It Is Not...... 38 5.3 Structure of the Port...... 38 5.3.1 The Code (High Level)...... 38 5.3.2 The Approach...... 39 Contexts Model Execution Priority Levels...... 39 Messages Model Supervisor Calls...... 39 Context Switches Multiplex the Task Context...... 39 The Message Dispatch Loop Multiplexes the Interrupt Context...... 40 Application Code Runs In Both Contexts...... 40 5.4 Discussion...... 41 5.4.1 Things Shown...... 41 5.4.2 Problems Encountered...... 41

iii iv CHAPTER 1

Introduction

Warning: Brittle is immature. It has been used in some demos, but not for anything “real.” Parts of the design are still in ﬂux. I’ll try to call attention to the particularly unstable parts in this document using boxes like this one.

1.1 What Brittle Is

Brittle is a microkernel intended for high-reliability embedded applications on ARMv7-M architecture processors, such as the Cortex-M4. These processors don’t have a conventional Memory Management Unit, which limits their ability to run traditional operating systems with (paged) memory protection. Instead, Brittle is designed to use the ARMv7-M Memory Protection Unit to provide isolation. Brittle is a third-generation microkernel. Its design is heavily inspired by EROS/KeyKOS, MINIX 3, and the L4 family, particularly seL4. Like other third-generation microkernels (broadly speaking), Brittle... • Focuses on minimality and security, • Expresses all authority through explicit capabilities, • Moves other mechanisms with security implications outside the kernel, • Blurs the line between a traditional microkernel and a hypervisor, and • Targets a very small kernel codebase (in Brittle’s case, less than 2500 sloccount lines of code). Unlike its peers, Brittle explicitly targets systems with between 16 and 200 kiB of RAM.

1.2 What Brittle Isn’t

Brittle is not a complete operating system; it is only a kernel. By analogy: putting the Linux kernel on a computer doesn’t allow one to run Firefox. Linux is only a kernel; there’s a whole lot of operating system that must be added to make a complete system. This is even more true of Brittle than of Linux, because Brittle’s design is so minimal. Like other third-generation microkernels, Brittle doesn’t even include hardware drivers in the kernel. You can write an application around the kernel directly, but the intent is that a system layer wraps the kernel and provides common reusable services that the kernel does not, such as mutexes and timers, or even a full POSIX model. Applications would then be written to target this system layer.

1 Technical Report on the Brittle Kernel, Release 2016-05

The system layer also insulates applications from the details of the hardware, because the Brittle kernel itself is explicitly not portable. Its design is ARMv7-M-speciﬁc, and exposes ARMv7-M-speciﬁc APIs and abstractions. Applications that wish to run on other types of processors should be written to a portable abstraction layer, provided outside the kernel by the system.

Note: I don’t consider Brittle’s non-portability to be a problem, because the entire kernel sources (at <2500 lines) are signiﬁcantly smaller than the architecture-speciﬁc support code for a single CPU in most kernels.

2 Chapter 1. Introduction CHAPTER 2

Kernel Architecture

2.1 Basic Concepts

This section introduces some ideas that will help you follow the next few sections, which elaborate the same ideas in more detail. At a high level, Brittle’s architecture is... 1. Object-oriented. Every resource and service that the system can see in the kernel is represented as an object bundling together state and operations on that state. 2. Capability-oriented. The only way for a program to refer to objects in the kernel — and thus to inspect them or activate their operations — is by explicit use of a capability, or key. A key is a combination of an object reference and a set of rights, operations on the object that a program holding the key can perform. Keys are held by the kernel on behalf of programs and cannot be directly inspected or manufactured. 3. Messaging-oriented. The kernel provides programs with a single efﬁcient message-transfer operation called IPC 1 , which is used both to operate on kernel objects, and to communicate between application tasks. Messages are small, ﬁxed-size structures that can carry both data and keys. These concepts are closely intertwined. Programs cannot inspect objects directly; objects live inside the kernel, and programs are isolated from the kernel using the ARMv7-M MPU. If a program wants to read information from the kernel, or trigger an operation in the kernel, there is only one way to do it: by sending a message to a key that refers to the desired object, and receiving another message in response. Because keys are held by the kernel in protected memory, programs cannot, in general, escalate their authority except in carefully designed ways. A program can only gain new authority (over kernel facilities or resources) by receiving a key from another program that already had that authority. Resources are managed the same way. The kernel does not contain an allocator for memory 2 or other resources. If a program wants the kernel to perform an operation that requires resources, it must already hold the resources, and must donate them to the kernel in exchange for a key to the desired new object. Resources are named by keys and donated in messages. For example, to create a new kernel object that requires 512 bytes of RAM for book-keeping, a program would send a message including a key to a 512-byte Memory object. Certain events can cause an object to be invalidated and its keys revoked. This causes all keys to that object to simultaneously and atomically become null keys (keys to an object called Null), which reject all IPC operations with an error code. For example, when a program donates resources (i.e. an object) to the kernel, the kernel invalidates the donated object, which ensures that no program continues to hold keys to it. Thus control of the object is reliably taken away from the program.

1 This operation is called “IPC” because the term is widely understood and roughly accurate, but Brittle’s IPC is used for more than “inter-process communication.” 2 Okay, technically, the kernel does contain a simple bump-pointer allocator used during boot. It is not, however, used after that, nor is it available for use by programs.

3 Technical Report on the Brittle Kernel, Release 2016-05

2.2 Architecture Rules

The kernel is designed and implemented with the following rules in mind. Rule 1 (keys). All kernel authority, services, and resources flow from explicit capabilities, or “keys,” which are trans- ferrable, revocable, and unforgeable. There is no ambient authority and no other in-kernel access control mechanism. The things a program can do to affect the system and other programs can be determined completely by inspecting the capabilities it holds. Merely holding a capability is not enough to perform any operation; the capability must be explicitly named when exercising the operation. This helps to avoid confused deputy scenarios when a key is delivered to a third-party unexpectedly. This has a subtle but important implication: all application code must run in the processor’s unprivileged mode, or it could subvert the capability system. Rule 2 (minimality). The privileged codebase should be minimal. Kernel code runs in the processor’s “privileged” mode, which makes it harder to isolate and reason about. Thus, a bug in the kernel can have serious, subtle, and wide-ranging effects. It is necessarily the most trusted of an embedded application’s trusted codebases. In Brittle, we interpret minimality two different (but related) ways. First, the set of abstractions and operations provided by the kernel should be as small as possible (analogous to Liedtke’s Minimality Principle from L4). Second, the actual line count of the privileged codebase should be as low as possible. This shall not be interpreted as permission to obfuscate the code; clarity is more important than brevity for brevity’s sake. However, the careful application of abstractions and refactoring can make code the right size for what it’s describing. Rule 3 (no implicit allocation). All resource allocation is explicit and under system (not kernel) control. The kernel does not, for example, allocate book-keeping structures from a kernel heap as a side effect of operations. The memory layout of kernel objects, and their allocation/deallocation, is managed entirely by unprivileged code. Rule 4 (predictable timing). All kernel operations shall have predictable execution times. More specifically, kernel operations are always constant-time with respect to any parameters that can vary at runtime. The execution time cannot vary with e.g. the number of tasks waiting on a particular event. (Some operations are linear in configuration parameters, such as the number of task priorities.) This enables Brittle to support hard real time systems and applications. Rule 5 (proxying). All kernel operations can be transparently proxied. Primitives provided to the system and application should be designed with proxying in mind. Applications must not be able to tell whether a capability they hold allows direct communication with the kernel, or whether it’s being intercepted and emulated by the system. This is critical to allow systems to implement their own security and access control policies, hardware abstraction layers, etc. It’s also important for both virtualization and test. This rule has important and far-reaching implications for the design of the messaging mechanism. Rule 6 (enough rope). The kernel shall protect itself from the system, not protect the system from the system. Brittle uses the hardware’s isolation features to keep stray pointer writes (for example) in the system from breaking any kernel invariants. This is important. But Brittle makes no attempt to keep the system from doing stupid things. If the system would like to revoke all its own authority, or give a buggy driver ultimate power over all applications... be our guest. Put another way: we assume you know what you’re doing.

4 Chapter 2. Kernel Architecture Technical Report on the Brittle Kernel, Release 2016-05

Rules 1, 2, and 3 are common in third-generation microkernels such as seL4. The rest are unusual, and have caused Brittle to look pretty different from its peers.

2.2.1 What’s In a Kernel?

Our interpretation of the rules above have led to the following general scope of the kernel.

In Scope

Isolation support. Describing parts of the physical address space and modeling application access rights. (Primarily through Memory objects.) Multiprogramming (thread) support. Sharing the single physical CPU between multiple (potentially isolated) programs. (Primarily through Context objects.) Interprocess communication support. Once we have isolated programs, we need a controlled way for them to communicate with one another when necessary. (Through the IPC mechanism and Gate objects.) Unprivileged driver support. The ARMv7-M architecture normally reserves some features important for writing drivers, particularly interrupt handling, for use by privileged code. The kernel provides mechanisms to expose these features to unprivileged system code. (Primarily Interrupt objects.) Kernel resource management. Facilities for unprivileged code to reason about and control kernel resource allocation and use. (In particular, the The Object Table.)

Out Of Scope

Two services often found in kernels — even microkernels — are missing in Brittle. Drivers. Applications can implement drivers in unprivileged code; the Brittle kernel doesn’t typically have any SoC depenencies. Time. The kernel has no notion of time, timeouts, or time-slices. Applications can implement this as needed. 1

2.3 Kernel, System, Application

Viewed simply, any piece of code is either part of the kernel, or not. This distinction is often cast (thanks to Unix) as that of kernel vs. userland. In a Unix-style monolithic kernel like Linux, system calls are provided by the kernel, and are fundamentally different from calls implemented by libraries or inter-process communication. The situation is different with a microkernel like Brittle. The kernel is designed to be extended by programs running in “userland,” and the equivalent of “system calls” — messages to objects — behaves the same whether it acts on kernel-implemented objects or components of an application. This means the object model presented by the kernel is extensible: programs can deﬁne new object types, proxy existing ones to apply access control policies, or even emulate a newer or older kernel API version, and other programs can’t tell the difference. This gives Brittle an excuse for its rather aggressive minimality, and insistence on implementing mechanism over policy. By not including (for example) a notion of IPC timeouts in the kernel, Brittle is not saying your application shouldn’t use IPC timeouts! Rather, the kernel doesn’t need IPC timeouts to function, itself, and moreover isn’t sure how exactly you’d like timeouts to behave. Rather than providing a particular scheme that you’d have to live with or work around, Brittle leaves the decision up to you. In general, Brittle does this with everything it thinks it can get away with; for more details, see the section What’s In a Kernel?.

1 The kernel’s ignorance of time is very unusual among microkernels. From what I can tell, only MINIX 3 shares this property.

2.3. Kernel, System, Application 5 Technical Report on the Brittle Kernel, Release 2016-05

Does this mean that every application should implement its own SysTick driver, time-slicing, and the like? Certainly not. We expect this sort of thing will get wrapped up into a reusable layer between the kernel and applications. We refer to this layer as the System. It is unprivileged but trusted code. The System is unprivileged because, like all programs on the Brittle kernel, it runs in the processor’s unprivileged mode. This means there are things it cannot do, mostly related to subverting the kernel’s invariants. At the same time, the System is trusted because it has tremendous authority over the functioning of the application. Indeed, when Brittle boots the ﬁrst program (assumed to be part of the System), it hands that program a key that grants explicit authority to subvert all compartmentalization and isolation outside the kernel! This authority is scary but vital; the System may have a legitimate need to peer into isolated programs’ memory, for example to implement Unix-like read and write operations. A trusted System will use this power for good, in support of the application’s mission, and not (for example) redistribute it to potentially malicious programs downloaded from the network. (See the section on the boot process to learn more about the roots of authority.)

Note: The distinction between kernel and System will be familiar to adherents of L4, but may seem more foreign to people familiar with the KeyKOS/EROS family of systems. (They have a similar distinction but it’s less clear in the documentation.) Among ﬁrst-generation microkernels, the role of the System is most similar to the idea of “kernel personalities” from Mach.

2.4 A Tour of the Kernel Objects

This section gives a brief introduction to each type of object implemented in the current version of the kernel, and some idea of how they relate and interact. Descriptions of objects and their operations here are informal but essentially correct. See the Kernel Object Reference for pedantic details.

2.4.1 Contexts

The ﬁrst object a program encounters is a Context, because Contexts are used to represent programs themselves. The boot process sets up one Context at startup, to run the application’s entry point. More speciﬁcally, a Context is a virtual representation of the processor’s unprivileged (Thread) execution mode. It provides storage for all the unprivileged processor state, so that it can be “backed up” when stopping a program to process interrupts or implement multitasking. The program whose state is stored in a Context is said to be “running in” or “inhabiting” the Context. Contexts extend the hardware processor model in two ways. First, they add two sets of virtual registers, both of which are used to hold keys. • The MPU Region Registers virtualize the hardware’s MPU registers, so that each Context effectively gets its own copy of the MPU. These registers either hold keys to Memory, to allow a program running in the Context to access memory or peripherals, or Null to leave a region unused. • The Key Registers hold keys of any type for use with the IPC operation (below). Second, Context extends the processor model with two virtual instructions, or syscalls: • Copy Key copies a key from one Key Register to another — similar to how the mov instruction operates on normal registers.

6 Chapter 2. Kernel Architecture Technical Report on the Brittle Kernel, Release 2016-05

• IPC transmits a message from the program to some other object, designated by a key in one of the Key Registers. It’s important to recognize that a program’s authority is entirely embodied in the contents of its Context’s virtual registers (MPU Region and Key). • A program cannot operate on, refer to, or even demonstrate the existence of, any object except those whose keys are held in the Key Registers of its Context. • A program cannot read, write, or execute a single byte of memory except those permitted by the keys loaded in its Context’s MPU Region Registers. Because of this, a Context isn’t much use without something to load into its MPU Region Registers... which brings us to our next object.

Note: For more information, see the Context entry in the Kernel Object Reference.

2.4.2 Memory

Memory objects represent regions of the physical address space. These may be RAM, ROM, peripherals, or even unmapped space that would fault if accessed. There is an initial set of Memory objects created at boot; from there, new objects can only be derived from existing ones, not created whole-cloth. Keys to Memory contain data describing what sorts of accesses can be performed using that key. It’s thus possible to create both read-write and read-only keys to the same Memory — perhaps keeping the read-write key for yourself and handing the read-only keys out to clients. Memory objects form a hierarchy. Programs can create child Memory that has access to a subset of an existing Memory object (its parent). This provides an easy way to isolate a program or grant a server access to part of the client’s address space. Programs can also divide Memory objects, destructively, into pieces. This is how programs pay the kernel for any new objects they wish to create — for example, additional Contexts to implement multitasking. Programs whittle Memory down to the size required for the desired object, and then send the Memory a “become” message that donates the memory and transforms it into a different kernel object. Programs can read and write the insides of Memory objects freely (if the key allows) by sending “peek” and “poke” messages, or by loading the key into a Context’s MPU region registers for direct access using load and store instructions. Either way, once Memory is donated to the kernel and becomes a different type of object, access is atomically revoked to protect kernel state.

Note: For more information, see the Memory entry in the Kernel Object Reference.

2.4.3 Gates

Gates serve as IPC rendezvous points for programs running in Contexts. Clients access Gates using specially-marked transparent keys; messages sent to transparent keys ﬂow right through the Gate to the program waiting on the other side. If no program is waiting to receive the message, the program sending the message can optionally block. This puts the program’s Context into a sleeping state until some other program is ready to receive the message. Alternatively, the sender can opt not to block, and the message is discarded. This style of messaging is called synchronous rendezvous, and means that Gates themselves don’t need to provide any storage for messages: messages are directly conveyed through Gates from sender to recipient.

2.4. A Tour of the Kernel Objects 7 Technical Report on the Brittle Kernel, Release 2016-05

Note: For more information, see the Gate entry in the Kernel Object Reference.

2.4.4 Interrupts

The zoo of objects described thus far is enough to implement multi-process programs with memory isolation, using polling to detect hardware events. But polling can be expensive; it’s sometimes better to put a program to sleep waiting for a hardware event, using an interrupt. Brittle virtualizes hardware interrupts in an object called (predictably) Interrupt. More importantly, Brittle converts hardware interrupts into messages. Each Interrupt object is associated with a single hardware interrupt request line. When the interrupt occurs, the Interrupt object sends a message to a Gate, containing information about which interrupt ﬁred. A program can receive this message, react to it, and decide when (and if) to re-enable the Interrupt for another round.

Note: Brittle’s Interrupt object models both NVIC-routed external interrupts, and the SysTick Timer exception. It cannot be used to intercept architectural faults or exceptions such as Hard Fault.

Interrupts can be configured to send a message to any Gate, or reconfigured on the fly, by passing a key via the “set target” operation.

Note: For more information, see the Interrupt entry in the Kernel Object Reference.

2.4.5 The Object Table

The Object Table is a singleton which provides programs with a facility for enumerating all kernel objects.

Note: The Object Table is a particularly unusual aspect of Brittle’s design that is likely to surprise readers familiar with other third-generation microkernels. If you’re familiar with (say) seL4 and are skimming the docs, now would be a good time to stop skimming.

The Object Table presents itself as a ﬁxed-size table (size chosen at build time) consisting of slots. Each slot is either empty or refers to a kernel object of the types listed above. Programs can hold keys to empty slots (represented as Slot objects). They form a second currency, alongside Memory objects: a key to an empty slot represents the right to increase the number of living objects, and is required to split a Memory object in half. Programs can also hold keys to the Object Table itself. With a key to the Object Table, a program can make its own rules: • It can “mint” a key to any kernel object out of thin air. • It can “inspect” the contents of a key to determine whether the key refers to a native kernel object, or a program through a Gate. • It can “invalidate” an object, causing all existing keys to be immediately and atomically revoked. These powers are intended for programs that implement the system layer atop the kernel. The assumption is that such programs will hold Object Table keys closely, and not hand them out to less trusted programs. However, this is not enforced, because it doesn’t need to be — holding an Object Table key still doesn’t let you violate any of Brittle’s invariants. So have fun and remember Rule 6.

8 Chapter 2. Kernel Architecture Technical Report on the Brittle Kernel, Release 2016-05

Note that the Object Table itself is an object, and is visible inside itself at slot #1.

Note: For more information, see the Object Table entry in the Kernel Object Reference.

2.5 About Keys

2.5.1 Key Semantics

To programs, keys appear to be opaque values that can be freely copied but not directly 1 created, inspected, or destroyed. Programs always refer to keys indirectly, by giving the index of a Context key register containing the desired keys. Speciﬁcally, without holding additional authority, programs can only perform three operations on a key: 1. Copy the key into a different key register (overwriting the destination). 2. Send a message to the object designated by the key. 3. Receive a message from the object designated by the key. The send and receive operations are collectively referred to as invoking a key, and can be combined into a “call.” For more details on these operations, see Syscalls. By design, programs cannot do any of the following without holding additional authority: • Determine the kind of kernel object designated by a key (e.g. Context vs. Gate). • Determine if two keys designate the same kernel object. • Determine if two keys give the same rights on their designated objects.

2.5.2 Parts of a Key

Conceptually, a key combines: 1. A reference to a kernel object, and 2. A set of operations on that object that the key enables. In implementation, a key is a kernel data structure built from three parts, which will be explained in more detail shortly: 1.A brand, which distinguishes kinds of keys to a particular object from one another. 2.A generation, which distinguishes keys to successive objects, created in the same location, from one another. 3. A pointer to a kernel object.

2.5.3 Brands

Every key bears a brand, a 64 bit value chosen when the key was created (minted). The brand is indelible: it cannot be altered except by minting a new key. It is also protected: the kernel keeps track of it on behalf of programs, and programs cannot generally discover the brand of keys they hold. When a key is copied (for example, by attaching it to a message) the copies bear the same brand as the original.

1 System components can use keys to the Object Table to create and inspect keys indirectly.

2.5. About Keys 9 Technical Report on the Brittle Kernel, Release 2016-05

Brands allow two keys to the same object to behave differently. The meaning of a brand is entirely up to the object designated by the key. Kernel objects specify the meaning of brands so that the System can mint appropriate keys, but transparent objects (particularly Gates) pass the brand up to programs for interpretation. It may encode access permissions, unique client identiﬁers, etc.

2.5.4 Generations and Revocation

Each key contains a generation and pointer. These are used together to provide efﬁcient revocation while avoiding dangling pointer bugs. Whenever a program invokes a key, the kernel compares the key’s generation to a generation ﬁeld stored inside the object. If the two match, the invocation proceeds normally. If the two differ, however, it means the object has been invalidated since the invoked key was minted. The kernel rewrites the offending key to Null. Because the generation is checked before any invocation, the key gets nulled before any program can observe a dangling pointer. (Copying a key around doesn’t indirect the contained pointer, and thus can’t observe a dangling one.) This guarantees that invalidation appears atomic, by lazily nulling any remaining keys to an invalidated object as they are touched by programs. Note that the generation is independent of the brand. This means that all keys to an object, regardless of brand, are revoked simultaneously during invalidation. Invalidation is available to programs through the Invalidate (4) method. Some objects perform additional work during invalidation, e.g. to remove themselves from kernel internal queues; this is described on each object’s page in the Kernel Object Reference.

Warning: The generation ﬁeld is currently 32 bits wide, which is narrow enough that it may roll over during a program’s execution. This is not theoretical: once object destruction is exposed, a naughty program could repeatedly create and destroy an object, forcing rollover in hours. There is currently no way to detect or correct this. I’m evaluating two options: requiring System intervention at rollover, or expanding generations to 64 bits.

2.6 Syscalls

Brittle implements a small set of syscalls, which act as virtual instructions that extend the processor. Syscalls exist for two reasons: 1. To let programs manipulate the Key Registers that Brittle adds to the processor model. 2. To let programs communicate with one another. Syscalls are accessed using the ARMv7-M SVC instruction.

Note: Brittle currently hooks every SVC, which means it can’t virtualize other operating systems’ syscalls. We are likely to ﬁx this by adding a “foreign” bit to the Context.

2.6.1 Syscall Descriptor Convention

Every Brittle syscall requires a descriptor loaded in processor register r4. The top four bits of the descriptor are a sysnum, or syscall number, which determines the operation to perform.

10 Chapter 2. Kernel Architecture Technical Report on the Brittle Kernel, Release 2016-05

Sysnum Operation 0 IPC 1 Copy Key The remaining 28 bits of the descriptor are interpreted differently by each syscall.

2.6.2 Copy Key

Reads a key from one of the current Context’s Key Registers, and writes a duplicate of it into another. All processor registers are left unchanged.

Table 2.1: Descriptor Bit Fields Hi Lo Name On Entry On Return 31 28 Sysnum 1 (Copy Key) (preserved) 27 24 Source Index of Key Register containing key to copy. (preserved) 23 20 Target Index of Key Register to receive copy. (preserved) 19 0 Reserved Should be zero. (preserved)

2.6.3 IPC

Brittle is built around a synchronous rendezvous messaging model. This means that messages are sent from one object to another directly, without being buffered in the kernel. Programs interact with objects using the IPC system call, which deﬁnes an operation consisting of two optional phases: • The send phase transmits a message from the sender’s Context to an object designated by a key held in a key register. • The receive phase receives a message from a remote object, again designated by a key held in a key register.

Note: Microkernel fans will recognize this as being very similar to Liedtke’s SendAndWait mechanism from L4.

The two phases can be combined into three types of IPC operations, plus a fourth special variety: 1. Send. A message is sent; once delivery of the message is accepted, the sender resumes without waiting for a response. This operation can optionally be marked non-blocking; a non-blocking send does not wait for delivery to be accepted if the recipient is not ready. 2. Receive. The program retrieves a message through a key (which should be a Gate key). If a message is already available, this operation completes immediately. If no message is available, the program blocks. (Receive operations are always potentially blocking.) 3. Send-then-receive. This performs a send, atomically followed by a receive, potentially on two unrelated keys. It is typically used by server programs to reply to one request and accept the next. 4. Call. A specialized form of send-then-receive, a call IPC fabricates a reply key for the program’s Context and sends it with the message. The object receiving the message can use the reply key to issue a single reply back to the sender.

Note: Technically there is a ﬁfth variety: if neither phase is requested, the syscall simply returns to the caller. This is not very interesting.

Each phase of an IPC transfers a single message. A message consists of

2.6. Syscalls 11 Technical Report on the Brittle Kernel, Release 2016-05

• The syscall descriptor. • Five words of data. • Four keys. When a program receives a message, one more datum is included: the brand of the key used to send the message through a Gate.

Message Descriptors

The ﬁrst word in a message is called the descriptor, and controls the IPC operation. Its ﬁelds are as follows.

Table 2.2: Descriptor Bit Fields Hi Lo Name On Entry On Return 31 28 Sysnum 0 selects IPC operation – 27 24 Source Key register index of source key for receive phase. – 23 20 Target Key register index of target key for send phase. – 19 19 Block If 1, caller is willing to block in ﬁrst phase. – 18 18 Receive If 1, enables receive phase. – 17 17 Send If 1, enables send phase. – 16 16 Error Signals error to receiver. Error in operation. 15 0 Selector Varies Varies

Key Maps

When keys are sent from or received into key registers, the registers are chosen according to an additional syscall parameter, the key map. A key map packs several four-bit key register indices into a single word. For the purposes of this section, and for the deﬁnition of kernel methods presented in the Kernel Object Reference, the registers named by the four positions in the keymap will be referred to as k0 through k3.

Table 2.3: Key Map Bit Fields Hi Lo Name 15 12 k3 11 8 k2 7 4 k1 3 0 k0

Note: The top 16 bits of the key map are currently unused, to allow for future expansion. These bits should be zero.

The same register index may appear multiple times in a key map. For sent keys, this causes the same key to be sent in multiple positions. For received keys, this causes multiple keys to be delivered to the same register, and it is not deﬁned which comes last. A Context’s key register 0 permanently contains a key to Null. This means register index 0 can be used in a key map in any “don’t-care” positions without accidentally transmitting or receiving authority.

The Send Phase

A sent message contains

12 Chapter 2. Kernel Architecture Technical Report on the Brittle Kernel, Release 2016-05

• The descriptor in r4. • Five data words taken from registers r5 through r9. • A key map in r10. • Four keys taken from the k0 - k3 registers named by the key map. If the IPC operation is a call, the ﬁrst key transmitted (k0) is not chosen by the key map, but is rather a freshly-minted Reply Key. A blocking send phase indicates success by continuing to the next phase (if any). A non-blocking send phase cannot indicate success or failure.

The Receive Phase

The receive phase uses the same descriptor in r4 as the send phase, but ignores the contents of r5 through r10. IPC involving a receive phase takes an additional key map in r11. After receipt of a message, the program gets: • A sanitized version of the descriptor in r4. • Five data words in registers r5 through r9. • The brand of the Gate key used to send the message, in r10/r11. • Four keys, delivered into the k0 - k3 registers named by the key map. The received descriptor is sanitized: the key index fields are zeroed, so that the recipient doesn’t learn anything about how the sender organizes their keys. The key delivered into the first position chosen by the key map (k0) is conventionally a reply key, whether it’s a real-live Reply Key to a Context, or something else (such as a Gate key for a testing framework). Servers that expect call-style IPCs agree to send a response back on the reply key. The received Gate key brand (in r10/r11) can be used to distinguish callers from one another, encode application- defined permissions, etc. If a program tries to receive using a key that doesn’t permit it (including keys to objects that are not Gates) it will instead receive an exception message.

2.7 Boot Process and Initial Environment

When the processor starts up, it executes the kernel’s reset handler. This routine sets up kernel state and then begins booting the application. The kernel is intended to be compiled separately from the application, and the same kernel binary can be used for several different applications. This means the kernel binary needs to be informed, somehow of the application’s initial requirements and entry point. This information is conveyed through a block of ROM called AppInfo. The kernel is linked to expect the AppInfo block at a particular address — typically at the end of the kernel’s ROM segment, but this is configurable at build time. The AppInfo block is fully specified by the common/app_info.h file in the kernel sources, but at a high level, it contains the following information: • A magic number used to detect kernel-application version mismatches, or empty ROM where the application should be. • The application’s initial program counter and stack pointer. • The number of extra object table slots the application expects to need.

2.7. Boot Process and Initial Environment 13 Technical Report on the Brittle Kernel, Release 2016-05

• The number of external interrupts the application plans to service. • The number of Memory objects needed to describe the application’s use of address space, with their locations and sizes. • An additional section of memory, donated to the kernel to set up the application. The kernel processes AppInfo and sets up 1. The Object Table. 2. A table mapping hardware interrupt requests to Interrupt objects. 3. An initial Context. The ﬁrst three slots in the Object Table are always occupied by three well-known objects, created at this time: Index Object 0 The Null object. 1 The Object Table. 2 The initial Context. Starting at index 3 are the Memory objects requested in the AppInfo block. All remaining slots in the Object Table are initialized to contain Slot object placeholders. The initial Context is conﬁgured to begin executing code at the application’s initial program counter. Its key registers are initially null, save for k1, which is seeded by the kernel with a key to the Object Table. This gives the program a mechanism for generating any other authority it requires.

14 Chapter 2. Kernel Architecture CHAPTER 3

Kernel Object Reference

3.1 Context

A Context models the processor’s unprivileged state with Brittle-specific extensions: • The sixteen general-purpose registers plus the program status word, for storing the program’s execution state. • A separate copy of the BASEPRI register, so that separate Contexts can mask and unmask interrupts indepen- dently. • Sixteen Key Registers for storing keys. • A configurable number 1 of MPU Region Registers for defining the program’s address space. One Context is created by the kernel during boot. Programs can create additional Contexts using the Become (4) method on Memory.

3.1.1 Branding

Brands separate Context keys into two classes, reply keys and service keys.

Reply Keys

A reply key grants the authority to deliver a single reply message to the program inhabiting the Context. Reply keys have the top bit of their brand set; the other 63 bits encode a call count. The Context internally tracks the call count, and only honors reply keys with the right count, to ensure that reply keys cannot be reused. (The call count is incremented at each successful reply, and whenever Make Runnable (7) is used to interrupt a Context awaiting reply.) Reply keys are transparent — they implement no native methods, and instead just forward any received messages through to the program inhabiting the Context. The only way to obtain a reply key 2 is by invoking a key with a call-style invocation. The kernel will internally fabricate a reply key and send it with the message. When the send phase completes, the Context atomically switches to expecting a reply through a reply key.

1 The number of MPU region registers can be conﬁgured at build time. The current default is six. The number of MPU region registers must be no larger than the actual number of MPU regions implemented by the hardware. 2 Except by guessing, of course. With an Object Table key a program can fabricate fake reply keys. If it can guess the call count, it can fake a reply. This is one reason why call counts are so large (63 bits). Currently they’re zeroed at Context creation, but we may randomize them in the future to make guessing really hard.

15 Technical Report on the Brittle Kernel, Release 2016-05

Service Keys

A service key grants the authority to perform maintenance on the Context, e.g. inspecting and altering register values. Service keys have the top bit of their brand clear, and implement the methods described below.

Warning: Currently, any Context service key can be used to call any method listed below. This is temporary. The service key brand will be further deﬁned to allow weakened service keys.

3.1.2 Invalidation

At invalidation of a Context, the kernel does the following additional work: • Removes it from any wait queues. • Resets its state to stopped. • Advances the expected brand of its kor-context-reply-key, invalidating them as well. • If the invalidated Context is current (i.e. a program is invalidating its own Context), causes a context switch to choose a new current Context.

3.1.3 Methods

Read Register (1)

Reads a register, by index, from this Context. Only the registers speciﬁed as “callee-save” by the ARM Procedure Call Standard, plus BASEPRI and the stack pointer, are stored in the Context; others are stored on the stack. Indices for use with this method are as follows: Index Register 0 r4 1 r5 2 r6 3 r7 4 r8 5 r9 6 r10 7 r11 8 BASEPRI 9 r13 (stack pointer)

Warning: Because this method reads a single register per IPC, it can become a bottleneck for programs (like debuggers) that need to read the entire register set. For better performance in such programs, see Read (Low/High) Registers (10/11).

Call

• d0: register index.

16 Chapter 3. Kernel Object Reference Technical Report on the Brittle Kernel, Release 2016-05

• d0: register contents.

Exceptions

• k.bad_argument if the register index is invalid.

Write Register (2)

Writes a register, by index, in this context. Only the registers speciﬁed as “callee-save” by the ARM Procedure Call Standard, plus BASEPRI and the stack pointer, are stored in the context; others are stored on the stack. Indices for use with this method are identical to those used with Read Register (1).

Warning: Because this method writes a single register per IPC, it can become a bottleneck for programs (like debuggers) that need to write the entire register set. For better performance in such programs, see Write (Low/High) Registers (12/13).

Call

• d0: register index • d1: value

Empty.

Exceptions

• k.bad_argument if the register index is invalid.

Read Key Register (3)

Reads a key from one of this Context’s key registers, by index. Note that there are currently 16 key registers. Key register 0 always reads as null.

Call

• d0: key index.

No data. • k1: key from Context.

3.1. Context 17 Technical Report on the Brittle Kernel, Release 2016-05

Exceptions

• k.bad_argument if the key index is not valid.

Write Key Register (4)

Writes a key into one of this Context’s key registers, by index. Note that there are currently 16 key registers. Because key register 0 is permanently null, index 0 is treated as invalid for this operation and will return an exception.

Call

• d0: key index • k1: key

Empty.

Exceptions

• k.bad_argument if the key index is not valid (including 0).

Read MPU Region Register (5)

Reads out the contents of one of this Context’s MPU region registers. The number of MPU region registers per Context is conﬁgurable at build time 1.

Call

• d0: region index

No data. • k1: region key

Exceptions

• k.bad_argument if the region index is not valid for this Context.

18 Chapter 3. Kernel Object Reference Technical Report on the Brittle Kernel, Release 2016-05

Write MPU Region Register (6)

Alters one of this Context’s MPU region registers. The number of MPU region registers per Context is conﬁgurable at build time 1. The change takes effect when this Context next becomes current, unless it is already current (i.e. it is modifying itself), in which case it takes effect immediately, before the reply is sent. This implies that a Context using this operation to remove its own access to stack will never receive the reply, taking a fault instead. Real Memory keys can be loaded directly into the region registers. Any other type of key will be treated as a null key and confer no authority.

Note: This is probably going to change; bogus keys should be rejected.

Call

• d0: region index • k1: region key

Empty.

Exceptions

• k.bad_argument if the region register index is not valid for this Context.

Make Runnable (7)

Switches this Context into “runnable” state. The practical effect of this depends on this Context’s current state: • If blocked waiting to send, without receive phase, the send is cancelled and the Context receives no notiﬁcation. • If blocked in receive, without send phase, the Context immediately receives a k.would_block exception. • If blocked in send-receive or call, the send phase is skipped and the Context immediately receives a k.would_block exception. In the case of a call, any issued reply keys become invalid. • If stopped, the Context is simply resumed. • If already runnable, nothing happens.

Note: Careful reading of this list above will show that a Context trying to make itself runnable will always succeed but receive an exception anyway.

Call

Empty.

3.1. Context 19 Technical Report on the Brittle Kernel, Release 2016-05

Empty.

Get Priority (8)

Gets the current priority of this Context.

Call

Empty.

• d0: priority

Warning: This API may change; priorities may need to be capabilities.

Set Priority (9)

Alters the current priority of this Context. If this Context is runnable, this might trigger a Context switch.

Call

• d0: priority

Empty.

Warning: This API may change; priorities may need to be capabilities.

Read (Low/High) Registers (10/11)

Each of these two methods reads a block of five kernel-maintained registers from this Context. There are ten total such registers, so five are the “low” registers, and five are the “high”. They are ordered in the same way as for Read Register (1). This operation is intended to make “swapping” — multiplexing multiple logical tasks across a single Context — faster, and is also useful in debuggers.

Call

Empty.

20 Chapter 3. Kernel Object Reference Technical Report on the Brittle Kernel, Release 2016-05

dn Low High d0 r4 r9 d1 r5 r10 d2 r6 r11 d3 r7 r12 d4 r8 r13

Write (Low/High) Registers (12/13)

Each of these two methods writes a block of five kernel-maintained registers into this Context. There are ten total such registers, so five are the “low” registers, and five are the “high”. They are ordered in the same way as for Read Register (1). This operation is intended to make “swapping” — multiplexing multiple logical tasks across a single Context — faster, and is also useful in debuggers.

Call

dn Low High d0 r4 r9 d1 r5 r10 d2 r6 r11 d3 r7 r12 d4 r8 r13

Empty.

3.2 Gate

Gates provide a rendezvous point for Contexts (and optionally Interrupts) using IPC.

3.2.1 Branding

Gate key brands separate Gate keys into two groups: service keys and transparent keys (or client keys). The two groups are distinguished by the MSB of the brand (set for transparent, clear for service). Only service keys have access to the methods described below. Any invocations on a transparent key ﬂow right through the Gate to any Context receiving on the far side (using a service key).

Service Keys

Service keys have the top bit clear. Programs holding a service key can invoke the methods described below. Remaining bits in the service key brand are reserved and should be zero. That is, in the version described by this manual, the only valid service key brand is Brand(1) << 63.

3.2. Gate 21 Technical Report on the Brittle Kernel, Release 2016-05

Programs can receive from a Gate service key, which will rendezvous with another program sending through a transparent key, blocking if necessary. Because the sender is always using a transparent key, any message received in this way will have the top bit of the sender brand set.

Transparent Keys

Transparent keys have the top bit set. The brand is passed, otherwise uninterpreted, to a program using a receive IPC on a service key.

3.2.2 Methods

Make Client Key (1)

Derives a transparent (client) key for this gate with the given brand. The brand must be a valid brand for a transparent key — that is, it must have its MSB set. This operation is intended for servers that want to distribute identiﬁable keys to different clients.

Call

• d0: brand low 32 bits. • d1: brand high 32 bits.

• k1: derived key.

Exceptions

• k.bad_argument if the given brand does not have its MSB set.

3.3 Interrupt

An “interrupt” object represents a hardware interrupt, manages its enabled status, and generates messages when the interrupt occurs. The messages go to a “target” key, loaded by the Set Target (1) method. The target may be Null The interrupt is initially disabled, so no messages will be sent until the object receives an Enable (2) message. (No messages will be sent anywhere useful until it also receives a Set Target message.) The interrupt is disabled when the message is generated, and remains disabled until the object receives another Enable message. The Enable message has the option of clearing any potentially queued interrupts that arrived while the interrupt was disabled, or leaving them enabled so they will be processed immediately. Whether the driver wants to clear pending interrupts will depend on the peripheral being serviced.

3.3.1 Branding

Interrupt key brands should be zero.

22 Chapter 3. Kernel Object Reference Technical Report on the Brittle Kernel, Release 2016-05

3.3.2 Invalidation

On invalidation of an Interrupt object, the kernel also disconnects it from the IRQ forwarding apparatus.

Warning: This is... probably not right.

3.3.3 Methods

Set Target (1)

Loads a new target key for this interrupt. When the interrupt ﬁres it will be converted into a message to the target key. Any existing target key will be discarded. The target key can be null, to disable delivery.

Call

No data. • k1: target

Empty.

Enable (2)

Enables the interrupt, acknowledging any outstanding interrupt.

Call

• d0: clear pending (1) or leave as-is (0).

Empty.

3.4 Memory

A Memory object represents a section of physical address space. It may describe RAM, ROM, peripherals, or even unmapped space that will fault if accessed. Keys to Memory are used whenever a program needs to reference some address space to the kernel, including describing the accessible address space for a Context. Memory objects are also the primary currency used to pay for the creation of kernel objects (the other being slots).

3.4. Memory 23 Technical Report on the Brittle Kernel, Release 2016-05

3.4.1 Mappable Memory

If a Memory object describes a naturally-aligned power-of-two-sized section of address space, it is mappable. This means it meets the restrictions of the ARMv7-M MPU and can be loaded into a Context’s MPU Region Registers for direct access by programs.

3.4.2 Hierarchy

Memory objects can be assembled to form a hierarchy by creating new child Memory objects derived from parent Memory. The children have access to a (non-strict) subset of the parent’s address space, and can be separately revoked. A root Memory is one without a parent. A set of roots are created during the boot process; they can be split into more manageable pieces as needed, but they remain roots. No two root objects overlap, so a root and its children have ultimate authority over a section of address space. A leaf Memory is one without any children. On startup, all the roots are also leaves; when a new child is created, it is a leaf; and so on. Certain operations that destroy or change the identity of Memory can only be applied to leaf Memory, to avoid dangling children. A root which is also a leaf has total authority over a section of address space: no parents nor children make claims to it. Such total Memory objects are the only kind that can be donated to the kernel using Become (4).

3.4.3 The Device Attribute

Because the ARMv7M architecture memory-maps all peripherals, the same Memory objects can be used to describe both normal RAM and device I/O space. This means it’s possible to confuse the two. In general, the kernel has no opinion on this; see Rule 6 (enough rope). However, if applications could trick the kernel into accepting donations of device registers in place of RAM, it could break the kernel’s invariants. To prevent this, applications can mark some of the root Memory in their AppInfo struct as device memory. This attribute is sticky: it persists across splits, and applies to any children. The kernel will not accept donations of Memory so marked.

Warning: There is currently no way for the application to detect device memory given a key. This is an oversight.

3.4.4 Branding

Mappable memory keys use the brand to control application access to the corresponding section of address space, as well as to determine the memory ordering and cache behavior of accesses through a particular key. To reduce possible impedance mismatches with the hardware, we use the hardware’s own encoding: the brand is exactly the value that would be loaded into the MPU’s Region Attribute and Size Register, but right-shifted 8 bits to lop off some irrelevant fields. Memory objects will refuse to create keys if the brand specifies an undefined encoding for the AccessPermission field. If a Memory object is not mappable, its brand bits are currently undefined and should be zero.

3.4.5 Invalidation

At invalidation of a Memory object, the kernel updates the cached contents of the MPU registers. If the invalidated Memory was accessible to programs (through use of an MPU Region Register) it atomically becomes inaccessible.

24 Chapter 3. Kernel Object Reference Technical Report on the Brittle Kernel, Release 2016-05

This is strictly only necessary if the invalidated object was one of the loaded regions, but it’s currently cheaper to update the registers than to detect this case.

3.4.6 Methods

Inspect (1)

Retrieves information about a Memory object and the key used to access it.

Call

Empty.

• d0: base address. • d1: Region Attribute and Size Register (RASR) equivalent contents, or zero if the region is not mappable. • d2: size in bytes. • d3: attributes (bit 0 = device, bit 1 = mappable).

Change (2)

Derives a new key to the same Memory object, with the ARMv7-M Region Attribute and Size Register (RASR) portion of the brand replaced. This operation is designed to derive weaker keys from stronger ones — for example, given a read-write key, derive a read-only one. As such, the new RASR value must imply equal or less access than the original, or it will be refused. This includes the Subregion Disable bits, which can only be set — not cleared — by this operation.

Note: It is possible, if slightly weird, to use this operation to create a key with all Subregion Disable bits set. Such a key confers no access.

No such validation is applied to the other attribute fields, such as cacheability and memory ordering. These fields can be changed arbitrarily. All bits in the RASR value that are reserved in the ARMv7-M spec should be zero. The SIZE and ENABLE fields in the RASR have no effect on the brand and should be zero.

Call

• d0: new RASR value

No data. • k1: new key with requested RASR brand

3.4. Memory 25 Technical Report on the Brittle Kernel, Release 2016-05

Exceptions

• k.bad_argument if the RASR value would increase access, or if it attempts to set Subregion Disable bits in a Memory object too small to support them (less than 256 bytes in size). • k.bad_operation if applied to a non-mappable Memory object.

Split (3)

Breaks a Memory object into two pieces, called bottom and top, divided at an arbitrary point within this object. The bottom half starts at the same base address as the original object, and has size equal to the split position; the top half starts just after the bottom half, and occupies the rest of the space taken by the original object. The device attribute is preserved. Each of the two pieces will be individually checked to see if it is mappable, and marked accordingly. This operation produces one net new object. To justify this use of resources, callers are required to donate a Slot key. The Slot is consumed and all keys revoked. This operation destroys this object, revoking all keys. Keys to the new Memory objects representing the top and bottom halves are sent in the reply. The returned keys have the same brand as the key used to split.

Note: Splitting is impossible in the following circumstances: 1. When the brand of the key used to split has any subregion disable bits set. 2. When this Memory object has any children.

Call

• d0: split point, as a byte offset from the start of the object. • k1: slot key being donated

No data. • k1: bottom part • k2: top part

Exceptions

• k.bad_argument if the split point is not within the object. • k.bad_operation if the region cannot be split for the reasons listed above. • k.bad_kind if the donated key is not a slot key.

26 Chapter 3. Kernel Object Reference Technical Report on the Brittle Kernel, Release 2016-05

Become (4)

Uses the address space described by this Memory object to create a new kernel object of a specified type. This object must be large enough to contain the new object (see below), and may be larger. Sizes are defined in terms of the configuration-time constant P, the number of priority levels. This object must not be device memory. The kernel only accepts donations of normal RAM. If the operation is successful, this object is destroyed, revoking all keys. The reply message contains the only extant key to the new object, with a default brand.

Note: Currently, “default brand” means zero. This will be revised. It would be desirable to allow the caller to specify the brand, but currently we can’t validate the brand until after we destroy the Memory, which would make for a bad user experience.

Note: Access to this operation will be eventually controlled by the brand.

The type codes for each type of object, the required donation size, and the role of the message ﬁelds/keys are given in the table below. Object Type Type Code Size Data Parameter 2 Key Parameter 1 Context 0 448 — Key to unbound Reply Gate Gate 1 16P — — Interrupt 2 48 Vector number (-1 for SysTick) —

Note: It is not possible to turn a Memory object into another type of kernel object if any of the following conditions apply: 1. If the key used has any subregion disable bits set. 2. If this Memory object is too small for the target object type. 3. If this Memory object is marked as device memory. 4. If this Memory object has any children (it is not a leaf). 5. If this Memory object has a parent (it is not a root).

Call

• d0: type code from table above • d1: type-speciﬁc argument

No data. • k1: new key

3.4. Memory 27 Technical Report on the Brittle Kernel, Release 2016-05

Exceptions

• k.bad_argument if the object type code is unrecognized. • k.bad_operation if this object is not suitable for use with Become, for any of the reasons listed above.

Peek (5)

Reads a word of data from the address space corresponding to this Memory object. For the purposes of this operation, the address space is represented as an array of words, with the ﬁrst word at offset zero. This allows a Memory object to be used without knowing its physical address, and without having to load it into a Context’s MPU Region Register. The key used must confer read access.

Warning: Currently, the operation is performed without regard for the ordering and cache behaviors speciﬁed by the key. This is not deliberate.

Call

• d0: offset

• d0: word of data

Exceptions

• k.bad_argument if the offset is out of range. • k.bad_operation if the key used does not confer read access.

Poke (6)

Writes a word of data into the address space corresponding to this Memory object. For the purposes of this operation, the address space is represented as an array of words, with the ﬁrst word at offset zero. This allows a Memory object to be used without knowing its physical address, and without having to load it into a Context’s MPU Region Register. The key used must confer write access.

Warning: Currently, the operation is performed without regard for the ordering and cache behaviors speciﬁed by the key. This is not deliberate.

Call

• d0: offset • d1: word of data

28 Chapter 3. Kernel Object Reference Technical Report on the Brittle Kernel, Release 2016-05

No data.

Exceptions

• k.bad_argument if the offset is out of range. • k.bad_operation if the key used does not confer write access.

Make Child (7)

Makes a new child Memory object, with this object as its parent. The child can describe any subset of this object’s address space. It will inherit any device attribute, will be checked for mappability, and the initial child key inherits the access permissions from the key used to invoke this method. As this creates a net new object, a slot key donation is required.

Call

• d0: base address of child. • d1: size of child, in bytes. • k1: slot key to donate.

No data. • k1: key to child object.

Exceptions

• k.bad_operation if the key used to invoke this object has subregion disable bits set. • k.bad_argument if the given base/size is outside the parent’s address space. • k.bad_kind if the alleged slot key is not, in fact, a slot key.

3.5 Null

The Null object is the target of revoked keys. It refuses all IPC with exceptions. There is one Null object, created by the kernel during boot. It cannot be duplicated or destroyed. This is the main difference between the Null object and Slot.

3.5. Null 29 Technical Report on the Brittle Kernel, Release 2016-05

3.5.1 Message Elision Rule

Messages to Null inside the kernel may be elided. Because a message to Null has no observable effect, message elision is not directly observable except in terms of timing. This does not apply to message sends initiated by programs inhabiting Contexts, only sends originating inside the kernel that do not carry reply keys, speciﬁcally: • Messages generated by Interrupt objects. • The reply transmitted by any kernel object in response to a method invocaiton. In either of these cases: • If a kernel object is preparing to send a message to key K, and • It discovers that K refers to the Null Object, • It may discard the message without sending.

3.5.2 Branding

Null key brands are ignored.

3.5.3 Methods

None.

3.6 Object Table

The Object Table tracks all kernel objects. It is the source of authority: it can mint any sort of key to any object, determine if keys point to the same object, and extract brands.

Warning: Applications should carefully consider their use of Object Table keys! They are powerful and should be closely held. We expect the system to proxy, extend, and limit the Object Table API if any of it is exposed to low assurance application components. See Rule 6 (enough rope).

3.6.1 Branding

The brands of Object Table keys should be zero.

3.6.2 Methods

Mint Key (1)

Mints a key to any object, speciﬁed by table index, with an arbitrary (but valid) brand. The resulting key will be valid for the object’s current generation, but revoked (like any other key) at the next invalidation of the object.

Note: Objects may impose validity rules on their brands. These rules are given in the relevant sections of the Kernel Object Reference. The Object Table will not mint a key with an invalid brand.

30 Chapter 3. Kernel Object Reference Technical Report on the Brittle Kernel, Release 2016-05

Call

• d0: object index in table • d1: brand low bits • d2: brand high bits

No data. • k1: newly minted key

Exceptions

• k.index_out_of_range if the index is not within the object table. • k.bad_brand if the object vetoes the suggested brand.

Read Key (2)

Inspects a key and reads out its table index and brand. This is essentially the reverse of Mint Key, and is the only way to tell whether two keys refer to the same object, and the only general way to read out a key’s brand.

Call

No data. • k1: key

• d0: table index • d1: brand low bits • d2: brand high bits

Get Kind (3)

Reports the true kind of the object inhabiting a particular object table slot. Kinds are reported using values from the table below. Code Kind 0 Null 1 Object Table 2 Slot 3 Memory 4 Context 5 Gate 6 Interrupt

3.6. Object Table 31 Technical Report on the Brittle Kernel, Release 2016-05

Call

• d0: object index in table

• d0: kind code

Exceptions

• k.index_out_of_range if the index is not within the object table.

Invalidate (4)

Advances an object’s generation number, causing all outstanding keys to become invalid. After invalidation, you can use Mint Key (1) to produce a new, valid key.

Warning: Generation numbers are currently relatively small (32 bits), so it is possible to use this operation enough that the Generation rolls over. This could cause previously invalidated keys to become valid again, unless some System component has scavenged such keys.

Call

• d0: object index in table • d1: rollover permitted ﬂag (0: exception on rollover; 1: allow)

Empty.

Exceptions

• k.index_out_of_range if the index is not within the object table. • k.causality if rollover would occur but has not been permitted.

3.7 Slot

Slot objects are placeholders that occupy unused slots in the Object Table. They generally behave like the Null object, but can be donated to a Memory object’s Split (3) method to create new objects, whereas the Null object cannot be destroyed.

3.7.1 Branding

Slot key brands are ignored.

32 Chapter 3. Kernel Object Reference Technical Report on the Brittle Kernel, Release 2016-05

3.7.2 Methods

None.

3.7. Slot 33 Technical Report on the Brittle Kernel, Release 2016-05

34 Chapter 3. Kernel Object Reference CHAPTER 4

Case Study: The Serial Demo

The serial demo is a simple demo application for the STM32F407 microcontroller, distributed with the Brittle sources in the demo/ascii directory. At boot, it repeatedly transmits the printable subset of the ASCII character set using a UART, at 115200 bps on pin PA2. It demonstrates the following: • How to write applications that directly target Brittle, without a system layer like FreeRTOS. • How to structure a multi-process application with isolation of authority (though not memory isolation at the moment). • How to write an interrupt-driven hardware peripheral driver using Brittle’s Interrupt object.

Note: Since there isn’t a convenient library for applications targeting Brittle yet, this demo has to do a lot by hand. This will change as parts get factored out into a library.

4.1 Tasks

At a high level, the demo consists of three tasks. Each task gets its own Context, and thus its own virtual copy of the processor and Key Registers to hold authority. The tasks are: Driver Manages the transmit side of the UART, responding to Client requests and interrupts. Holds keys that allow it to receive requests and service the Interrupt object. Client Invokes the UART driver over IPC to send an inﬁnite stream of ASCII characters. Holds a single key that lets it submit UART transmit requests to the driver. Idle Puts the processor into low power mode when neither of the other tasks has any work to do — i.e. when waiting for a transmit-complete interrupt. Holds no keys, so it cannot interfere with the other tasks, except possibly by scribbling over their memory, since the tasks aren’t currently isolated in memory.

4.2 Driver Operation

The UART driver is implemented in demo/drv/uart/driver.cc, and provides a client library in demo/drv/uart/client.cc.

35 Technical Report on the Brittle Kernel, Release 2016-05

On task startup, the driver initializes the UART, along with the shared resources (clocks and reset lines) required to make the UART work. It then enables the interrupt and enters a service loop. The service loop repeatedly does the following: • Receives any waiting transmit requests. • Loads data into the UART transmit register. • Blocks waiting for a message from the interrupt. • Repeats. The driver services two Gates: one to receive requests from the Client task, and one to receive interrupt messages. At any given time, the driver is receiving from one of the two Gates. This provides ﬂow control: while the UART is busy, Client requests will block, since the driver is only listening for interrupt messages. When the UART is idle, the driver only listens for client messages. This lets the IPC mechanism serve as a sort of natural buffer. Clients attempting to transmit will block if the UART is busy, and their requests will be processed as it becomes available (in priority order).

4.3 Startup

The demo is started directly by the kernel, meaning it wakes up in the boot environment. This means it initially holds an Object Table key granting tremendous authority. The ﬁrst goal is to shed that authority — but only after using it to set up the rest of the demo. The demo startup routine (in demo/ascii/main.cc) contains a basic memory allocator, which is initially handed all of the system’s RAM by the boot process. It subdivides RAM into small blocks, and then donates them to the kernel to create new objects, using Memory‘s Become (4) method. The created objects include: • The Interrupt object used to service interrupts from USART2. • The Gate used to transfer interrupt messages to the driver task, and the Gate used by the client to communicate with the driver. • The Contexts representing the client and idle tasks. After generating and wiring these objects, the startup routine then mints a Memory key giving access to USART2’s registers, and loads it into its own MPU region registers. It then calls the driver’s entry point, effectively becoming the driver task.

36 Chapter 4. Case Study: The Serial Demo CHAPTER 5

Case Study: FreeRTOS

Warning: This section is slightly out of date.

5.1 Introduction

FreeRTOS is a popular open source embedded operating system that targets a similar class of processors as the Brittle kernel. Brittle and FreeRTOS have different perspectives on several important points. For example, the current Brittle kernel does not support destruction of kernel objects at runtime, and will likely never support in-kernel priority inheritance 2. Thus, one might reasonably question whether Brittle could support the same sort of dynamic applications as FreeR- TOS. But in fact Brittle was designed for this sort of thing. To show how, we have ported FreeRTOS V9.0.0 1 to run on Brittle as a System layer, along with some simple demo applications. We’ll refer to the combined system as FreeRTOS/Brittle, by analogy to GNU/Linux.

5.2 About FreeRTOS/Brittle

FreeRTOS/Brittle is not a FreeRTOS API emulation layer or simulator. It is the actual FreeRTOS code, derived from the ARM_CM3 port, including the scheduler. This means that all FreeRTOS features are supported, and behave identically to the native ARM Cortex-M3 FreeRTOS port, with a couple of notable exceptions that will be highlighted in this chapter. To be clear, the FreeRTOS System layer implements: • Allocation and deletion of OS objects, such as tasks and queues, on the ﬂy out of a System-layer heap. • Mutexes with priority inheritance. • A notion of time, including operation timeouts and time-slicing with preemption. This serves as an existence proof that Brittle’s abstractions are sufﬁcient to implement these features outside the kernel. By running on top of the Brittle kernel, FreeRTOS gains several features that are missing from the ARM_CM3 port 3:

2 Brittle is unlikely to natively support priority inheritance because, so far, I have not been able to ﬁgure out how to implement priority inheritance using constant-time algorithms. Brittle is, however, likely to provide primitives to make it easier to implement priority inheritance in the system. 1 Actually v9.0.0-rc2. v9.0.0 was running a bit late. 3 There is an ARM_CM3_MPU port that provides a form of memory isolation, but because FreeRTOS lacks a messaging primitive for efﬁciently calling between isolated tasks, I’ve found that isolation terribly awkward to use.

37 Technical Report on the Brittle Kernel, Release 2016-05

• A memory-protected environment where access to peripherals, etc. must be whitelisted through explicit capabilities, to avoid bugs that access unexpected areas of the address space. • The ability to run entirely in unprivileged code, which means that memory protection is always enforced — even in interrupt handlers. • The ability to run a hybrid system combining FreeRTOS drivers and native Brittle drivers, which beneﬁt from further isolation and robustness guarantees. Applications can gradually migrate drivers into isolated environ- ments to incrementally improve robustness.

5.2.1 What It Is Not

FreeRTOS/Brittle may not be suitable for all FreeRTOS applications. • The port uses more resources: about 16kiB of additional ROM, and a few kiB of additional RAM, compared to the native port. • Performance is somewhat reduced: interrupt response latency is up by about a factor of 10, in the current port, which also affects the speed of preemptive context switches. • The port is missing some features: custom handling of ARMv7-M faults is not currently piped through into FreeRTOS.

5.3 Structure of the Port

5.3.1 The Code (High Level)

FreeRTOS is separated into a portable core and a collection of system-specific port layers that adapt the core to particular architectures. This work is modeled after the Cortex-M3 port, which in the FreeRTOS sources lives at the following path: FreeRTOS/Source/portable/GCC/ARM_CM3 FreeRTOS/Brittle, in turn, lives here: FreeRTOS/Source/portable/GCC/ARM_CM3_brittle Like the ARM_CM3 port, FreeRTOS/Brittle consists of only two source files in this directory, and no changes to FreeRTOS’s portable core. The ARM_CM3 port uses privileged code, interrupts, and long stretches of assembly to provide processor-specific hooks for the portable core. It’s 434 lines of code 4 . FreeRTOS/Brittle does the same thing, but uses unprivileged code, kernel objects, and messages. It compares favorably to the native port at 537 lines of code. Interestingly, it is implemented almost entirely in straight C, with only two explicit assembly language instructions.

Note: When comparing port sizes, note that FreeRTOS/Brittle does not support tickless idle, which would otherwise add a hundred or more lines of support code. This support is not absent because of limitations in Brittle, but because the tickless idle support proved fragile, littered with assumptions, and hard to port.

4 As with every other “lines of code” measurement in this report, this ﬁgure was generated using David A. Wheeler’s SLOCCount tool.

38 Chapter 5. Case Study: FreeRTOS Technical Report on the Brittle Kernel, Release 2016-05

5.3.2 The Approach

Note: There are many possible ways of mapping FreeRTOS concepts onto Brittle concepts. This is the particular one we used.

The current implementation of Brittle doesn’t allow kernel objects to be destroyed once they have been created. FreeRTOS, on the other hand, can create and destroy objects (like tasks) freely. Instead of trying to map FreeRTOS objects to Brittle objects, then, we opted to treat the Brittle kernel like a hypervisor, and use its abstractions to deﬁne a machine model for FreeRTOS to target.

Contexts Model Execution Priority Levels

FreeRTOS/Brittle deﬁnes, at minimum, two Contexts: • The Task Context models the processor’s Thread execution mode and is used to run FreeRTOS task code. • The Interrupt Context(s) model the processor’s Handler execution mode, with one Context per priority level used. These are used to run interrupt handlers for hardware interrupts (including the SysTick Timer), and also to implement some virtual interrupts described below.

Note: The current port uses a single Interrupt Context for simplicity, which is equivalent to setting all interrupts to the same priority. We’ll refer to the singular “Interrupt Context” for the rest of this chapter.

Messages Model Supervisor Calls

The Task and Interrupt Contexts share access to a Gate, called the System Gate. Both Contexts are given branded keys to the System Gate during initialization; the Task Context is given the right to send to it, and the Interrupt Context to receive from it. The FreeRTOS port layer sends Brittle IPC messages through the System Gate to perform two operations — the speciﬁc operations in the ARM_CM3 port that require privileged CPU instructions: • Requesting a context switch (from the implementation of vTaskYield, originally implemented by pending a PendSV exception). • Enabling/disabling interrupts (also used by the FreeRTOS critical section code, originally implemented by altering BASEPRI). The Interrupt Context holds Service Keys to both itself and the Task Context. It uses the authority granted by those keys to perform these “privileged” operations: it can rewrite its own BASEPRI directly, and it can rewrite the Task Context’s BASEPRI while the Task Context is blocked in IPC.

Context Switches Multiplex the Task Context

Because the single Task Context is shared by all FreeRTOS task code, the FreeRTOS scheduler treats it just like a processor’s register set: when it’s time to preform a (FreeRTOS) context switch, the registers from the Context are saved into the (FreeRTOS) task control block, and the registers from another task control block are loaded in to replace them. The Interrupt Context performs this “task swap” using the Service Key it holds to the Task Context.

5.3. Structure of the Port 39 Technical Report on the Brittle Kernel, Release 2016-05

The Message Dispatch Loop Multiplexes the Interrupt Context

FreeRTOS/Brittle configures the Interrupt Context to run an infinite loop receiving and handling IPC messages. These messages come from two sources: 1. Code running on the Task Context, requesting the “system call” services described above. 2. Hardware interrupts, modeled with kernel Interrupt objects, including the SysTick Timer. The dispatch loop looks at two parts of the recieved message: • The brand of the key used to send the message, which distinguishes messages from Interrupt objects from “syscall” messages from the Task Context. • The message selector field, which distinguishes operations. In response, it performs System operations, including calling application interrupt handler routines.

Application Code Runs In Both Contexts

FreeRTOS allows application code to run in interrupt context — both to implement new drivers, and to add functional- ity to FreeRTOS’s own interrupt service routines, such as the one for the SysTick Timer. This means application code is running in the processor’s privileged mode, where certain isolation properties (particularly memory protection) are broken. FreeRTOS/Brittle does something similar but safer: when needed, System code running in the Interrupt Context will call out to application routines, either to dispatch hardware interrupts, or to implement FreeRTOS interrupt hooks. The most visible piece of application code that runs in the Interrupt Context is not an interrupt handler, but the traditional entry point, main. The main routine in a FreeRTOS application is responsible for creating any initial FreeRTOS objects that are required, and then starting the scheduler. The AppInfo block in the FreeRTOS/Brittle image tells the kernel to start the initial program in the Interrupt Context, not the Task Context, running main. The program’s main eventually calls vTaskStartScheduler in FreeRTOS, which kicks off the Interrupt Context message dispatch loop and yields to the Task Context. This has one interesting implication for the port. FreeRTOS does not expect its setup code to be called “from an interrupt.” In fact, many of the routines applications need to use during setup are technically unsafe for use in interrupts, such as xTaskCreate to create a task. The most visible reason why: they use the non-ISR version of critical sections under the hood. But FreeRTOS/Brittle runs main from the equivalent of an interrupt, so we have to lift this restriction. The port does so by unifying the ISR and non-ISR implementation of interrupt control, used by critical sections: • The Interrupt Context holds its own Context Key in a particular key register, and can directly adjust its BASEPRI value. Thus it acts somewhat privileged. • The Task Context holds a specially branded key to the System Gate in the same key register. The Interrupt Context’s message dispatch loop recognizes this key’s brand, and emulates the Context protocol for adjusting the BASEPRI register only. Thus, FreeRTOS/Brittle code running on either context can adjust BASEPRI by sending the same bytes to the same key register, but we didn’t have to give the Task Context its own key (with all the authority that would convey).

40 Chapter 5. Case Study: FreeRTOS Technical Report on the Brittle Kernel, Release 2016-05

5.4 Discussion

5.4.1 Things Shown

You can build dynamic applications on a static kernel. The port allocates a ﬁxed set of kernel objects before starting main, and these objects live for the duration of the application — even when the application dynamically creates and destroys tasks, mutexes, and the like. Brittle is usable from C. Brittle is implemented in C++11, but was designed to be usable from legacy languages such as C and assembler. As FreeRTOS/Brittle is entirely C (and C90 at that), this seems to have worked. Expensive but important algorithms like priority inheritance can be implemented outside the kernel. There are no constant-time algorithms for general priority inheritance, of the sort required by FreeRTOS, and so priority inheritance cannot be implemented inside the Brittle kernel. FreeRTOS/Brittle suggests that this might be okay.

5.4.2 Problems Encountered

Poor context-switch latency. The context switch latency between FreeRTOS tasks is 10-50x slower in FreeR- TOS/Brittle than in a native FreeRTOS port. The culprit: the relatively high cost of Brittle IPC in the current implementation. The IPC path has not been optimized and can take 1000 cycles. Because the FreeRTOS scheduler is implemented outside the kernel and interacts with kernel objects via IPC, every context switch generates around 20 IPCs. We’ve been able to reduce this by adding bulk operations to the Context protocol, such as Read (Low/High) Registers (10/11). Once the design has stabilized, a fast-path implementation for inter-Context IPC should be able to reduce the IPC cost by a factor of 10, as shown by the similar fast-path implementations in EROS, L4, and MINIX 3.

5.4. Discussion 41