OS Perspective on Android

David E. Culler CS162 – Operating Systems and Systems Programming http://cs162.eecs.berkeley.edu/ Lecture 27 December 5, 2019

cs162 Recall: What is an ?

• Special layer of that provides application software access to hardware resources – Convenient abstraction of complex hardware devices – Protected access to Shared resources – Security and Authentication – Communication amongst logical entities

appln appln appln OS

Hardware

12/5/19 UCB CS162 Fa19 L27 2 In OS news this term …

12/5/19 UCB CS162 Fa19 L27 3 State of things

• Pixel 4 shipped in Oct 2019 with kernel 4.14, an LTS release from November 2017 and will be stuck there – “Thanks to kernel updates never happening, this means every new release of Android usually has to support the last three years of LTS (Long Term Support) kernel releases (the minimum for is 4.9, a 2016 release). – it is still supporting kernel 3.18, which is five years old now. • 5.4 mainline 11/25/2019

12/5/19 UCB CS162 Fa19 L27 4 But, what do you think ?

• What direction will the future of Linux take? • We have seen so many of the enhancements driven by the needs of the cloud – Proportional share scheduling – Cgroups / Containers – KVM – paravirtualization for virtual machines – Concurrency support – Journaling • Versus single-user small form factor android experience • Will software engineering aspects of THE most common platform dictate?

12/5/19 UCB CS162 Fa19 L27 5 The System “stack” you now know

Application runtime

User libc subsystems ctrl syscall handlers scheduler mem mgmt syscall tbl intrpt handlers

Kernel file sys intr tbl networking Page Tables user / grp Drivers

regs

Processor Memory Devices Hardware

• /dev

12/5/19 UCB CS162 Fa19 L27 6 What’s in Android ?

• Tiers of the Android system architecture • Linux kernel is the part at the bottom – How is Linux used? – How is it extended for this setting? – What matters in the OS? • What OS functionality is outside the kernel? • How do the demands on it differ?

https://source.android.com/devices/architecture

12/5/19 UCB CS162 Fa19 L27 7 A somewhat different picture

• Kernel provides process &mem mgmt., file system, & Drivers – Does not run Linux binaries – No libc – Syscalls ? • Single “user” => UID per App – In security sandbox + VM • Different user navigation experience – Threaded thru Apps • “Binder” IPC is the key

12/5/19 UCB CS162 Fa19 L27 8 THE User (you) vs UID

• Underlying Linux is multi-user • Each Android App is assigned a distinct Linux UID – Permissions per App – By default, only the App can access its files (User permissions) • Each App runs as its own Linux process* – Android starts the process whenever any of the App’s components need to be executed – Not just main() or when you click on its icon, but … • Each process has its own VM – Like JVM, not “OS virtual machine”. Bytecode interpreter. – VM => Android Run Time (ART)

* Can arrange for multiple Apps to share UID, but they also run as one process, share the same VM. Must be signed with same certificate

12/5/19 UCB CS162 Fa19 L27 9 The Android Mantra

• “The mobile-app experience differs from its desktop counterpart in that a user's interaction with the app doesn't always begin in the same place. Instead, the user journey often begins non-deterministically.” • The user experience threads through various apps, each potentially in direct interaction along the way – taking over the entire small screen • Apps build on the functionality in others

• A more cooperative ecosystem … hmm

12/5/19 UCB CS162 Fa19 L27 10 Components of Android App

User Interactions

Activities APP Services

Broadcast Receivers Providers Content System • Not just exec main() and syscall … • Lots of entry points (call backs) • Process started “whenever needed” 12/5/19 UCB CS162 Fa19 L27 11 App Components / Entry Points

• Activities – Entry point for interacting with the User. – Represents a single screen with a user interface • Services – Run in the background. No user interface • Content Providers – Entry point for publishing named data items, identified by URI scheme – Manages shared App data that can be stored in file system, SQLite, on the Web, or on any accessible persistent storage • Broadcast Receivers – Entry point for system events. Outside regular App flow

12/5/19 UCB CS162 Fa19 L27 12 “The Operating System ?”

• The Syscalls Recall: A Kind of Narrow Waist

• The processor/mem Word Processing Compilers Web Browsers resources delivered to Email Web Servers processes and threads via Databases Application / Service Portable OS OS scheduling User Interface System • The hardware resources Portable OS Kernel supporting high level Software Platform support, Device Drivers Hardware x86 PowerPC ARM abstractions via drivers, PCI file systems, etc. Ethernet (10/100/1000) 802.11 a/b/g/n SCSI IDE Graphics • The system libraries (libc) 9/12/19 cs162 fa19 L5 23 • The system utility apps – bash, ps, top, … • The convention for utilizing file structure – /etc, /proc, /bin, … 12/5/19 UCB CS162 Fa19 L27 13 Extending the OS w/o new syscalls ?

Application runtime open /dev/subsys read, write, ioctl on fd User libc subsystems process ctrl syscall handlers scheduler mem mgmt syscall tbl intrpt handlers

Kernel file sys intr tbl networking Page Tables user / grp Drivers

regs

Processor Memory Devices Hardware • Drivers - add modules to kernel that communicate with user process thru ops

12/5/19 UCB CS162 Fa19 L27 14 Closed Hardware

• Primarily ARM, ARM64 • x86 & x86_64 – but only the microprocessor, not the x86 peripherals that intrinsic to PCs, Laptops, and Servers

• Audio, Camera, Graphics, WiFi, GPS, Flash, Power management

• No paging, no HDDs, no external busses, …

12/5/19 UCB CS162 Fa19 L27 15 The Linux Kernel – Major Changes

• Binder – Android Specific IPC / RPC mechanism – THE key to the entire application framework • ashmem - Anonymous (or Android) – driver-based augment (replace shmat) for small memory & extensive process sharing • PMEM – process memory allocator – Large physically contiguous memory blocks mapped into • logger – system logging facility • wakelocks – User-controlled means of controlling whether system enters low-power states and what wakes it up • Out-of-memory handling – Kill processes under low memory conditions • Paranoid networking – restrict I/O to certain proc’s • Alarm timers, Timed gpio from user space • yaffs2 flash file system, … https://elinux.org/Android_Kernel_Features Android Systems Programming 12/5/19 UCB CS162 Fa19 L27 16 ashmem

(“/dev/ashmem”, O_RDWR) Application open ioctl on fd to set name and size runtime mmap on fd to establish region User ??? subsystems process ctrl syscall handlers scheduler mem mgmt syscall tbl intrpt handlers

Kernel file sys intr tbl networking Page Tables user / grp Drivers

regs

Processor Memory Devices Hardware • Named regions known independent of /exec • Reference counted, clean up, low mem reclaim

12/5/19 UCB CS162 Fa19 L27 17 – the glibc replacement

• Design philosophy – KEEP IT REALLY SIMPLE – Lightweight wrapper around kernel facilities • BSD libc + Linux-specific bits (w/o GPL) – Threads, process, signals, and bit more • pthread bundled in, not separate – Based on futexes – very short code paths in common case – Only implements simple, most useful parts » No process-shared mutex or CV, read/write locks,, pthread_cancel, … » Limited -local storage – pthread_mutex_t and pthread_cond_t are only 4 bytes • No multicore support • No sysV IPC, semaphores, shared memory segments, message queues, …

12/5/19 UCB CS162 Fa19 L27 18 Aside: Linux FUTEX

• Kernel does not provide lock/unlock syscall • Locks are implemented as efficient (eg. test-and- test-and-set) spin-locks at user level using atomic hardware primitives – Common case: grab an uncontended lock – make it fast – Restructure app if it is not common (e.g., ticket locks) • But don’t spin long holding the CPU when other thread is holding the lock – Syscall traps to the kernel until a certain condition becomes true, i.e. a change occurs that might free the lock • FUTEX is not exposed in libc, it is used within pthread implementation • Three states for lock implementation – Free, busy-no-waiters, busy-waiters (why ?)

12/5/19 UCB CS162 Fa19 L27 19 FUTEX cont

#include #include

int futex(int *uaddr, int futex_op, int val, const struct timespec *timeout );

• uaddr points to a 32-bit value in user space – placed in region of shared memory, w/I process or created using mmap(2) or shmat(2) across processes • futex_op – FUTEX_WAIT – if val == *uadder sleep till FUTEX_WAIT » Atomic check that condition still holds – FUTEX_WAKE – wake up at most val waiting threads – FUTEX_FD, FUTEX_WAKE_OP, FUTEX_CMP_REQUEUE • timeout – ptr to a timespec structure that specifies a timeout for the op

12/5/19 UCB CS162 Fa19 L27 20 Adjusting the OS/User contract

Application runtime

User libc Lock Object subsystems process ctrl syscall handlers scheduler mem mgmt syscall tbl intrpt handlers

Kernel file sys intr tbl networking Page Tables user / grp Drivers

regs

Processor Memory Devices Hardware • User level deals with locks & synch • OS mantains thread waiters list & schedules

12/5/19 UCB CS162 Fa19 L27 21 Android-Specific Features in its Linux

• System Properties: Key/Value store shared by all processes – Like windows “registry” – Alternate to /proc, /etc . . . • User/Group management – No /etc/passwd or /etc/groups – groups used for process permissions – Each App gets own UID & GID (getgrouplist rtns 1 item) – Hardcoded subsystem names (e.g., ”radio”) & GIDS • No /etc/services – C library hardcodes constant list of services • DNS resolver – List of servers from system properties, per-process DNBS server list, increased security • pthread real-time support

12/5/19 UCB CS162 Fa19 L27 22 Administrivia

• Survey Incentive: 70+% participation => 5+% of final direct from practice test • Review session here, Thurs 2-3:30 regular time • HWs deferred – so can focus on Project 3

12/5/19 UCB CS162 Fa19 L27 23 Understanding the Application Framework

• Components – Activities – Services – Content Providers – Broadcast Receivers • All connected via IPC • A unique aspect of the Android system design is that any app can start another app’s component.

12/5/19 UCB CS162 Fa19 L27 24 Activities

• facilitate key interactions between system & app • Keeping track of what the user currently cares about to ensure that the system keeps running the process that is hosting the activity – what is on screen • Knowing that previously used processes contain things the user may return to (stopped activities), and thus more highly prioritize keeping those processes around. • Helping the app handle having its process killed so the user can return to activities with their previous state restored. • Providing a way for apps to implement user flows between each other, and for the system to coordinate these flowa. – The most classic example here being share.

12/5/19 UCB CS162 Fa19 L27 25 Ensemble of cooperating Activities

• Although the activities work together to form a cohesive user experience each one is independent of the others. • a different app can start any one of these activities if the app allows it. – Get location – Send email – Acquire microphone stream – …

12/5/19 UCB CS162 Fa19 L27 26 Services

• component that runs in the background to perform long-running operations or to perform work for remote processes. • A service does not provide a user interface. • Started services tell the system to keep them running until their work is completed. • Bound services run because some other app (or the system) has said that it wants to make use of the service. – This is basically the service providing an API to another process.

12/5/19 UCB CS162 Fa19 L27 27 Content Provider

• Manages a shared set of app data stored: – in the file system, – in a SQLite database, – on the web, or on any other persistent storage location that your app can access. • Through the content provider, other apps can query or modify the data if the content provider allows it. • To the system, a content provider is an entry point into an app for publishing named data items, identified by a URI scheme. • App can decide how it wants to map the data it contains to a URI namespace – hands out those URIs to other entities which can in turn use them to access the data.

12/5/19 UCB CS162 Fa19 L27 28 Aside: URI

• A Uniform Resource Identifier (URI) is a string of characters that unambiguously identifies a particular resource. Follows predefined syntax. • A Uniform Resource Locator (URL) is a URI that specifies the means of acting upon or obtaining the representation of a resource – http://example.org/wiki/Main_Page => resource (wiki/Main_Page) is obtainable from domain example.org via http protocol • Android: URIs can persist after their owning apps have exited. – The system only needs to make sure that an owning app is still running when it has to retrieve the app's data from the corresponding URI.

12/5/19 UCB CS162 Fa19 L27 29 Broadcast Receiver

• enables the system to deliver events to the app outside of a regular user flow, allowing the app to respond to system-wide broadcast announcements. – Boot completed, power connected / disconnected, battery low / okay, …

12/5/19 UCB CS162 Fa19 L27 30 Illustration

12/5/19 UCB CS162 Fa19 L27 31 When does a component run?

• Whenever it needs to… • “When the system starts a component, it starts the process for that app if it's not already running and instantiates the classes needed for the component.” – The jvm assumption is pretty deep – But c++ is supported too (what about C? Go?) • Each App is in different process, under different User, with user-only file permissions • How does an App activate a component in another App? • Deliver a message to the system that conveys intent to start the particular component – The system intervenes in all App-to-App interactions – IPC !!! 12/5/19 UCB CS162 Fa19 L27 32 Intents – an Active Message

• A message that activates a component – Activities & Services: it describes the action to perform – Broadcast Receivers: simply an announcement • How is the recipient of the intent identified? – What ”binds” the two participants? – Explicit Intent specifies the component name – Implicit Intent species the action, can be invoked on any App that can perform that action

• Content Providers are not activated by intents – They are a more passive entity – Activated by a request from a “Content Resolver” » Pulled, not pushed

12/5/19 UCB CS162 Fa19 L27 33 Recall: RPC, Serialization

Inter-Process Communication (IPC) • Mechanism to create a communication channel RPC Concept between distinct processes bundle • User to User, Kernel to User, Same machine or different ones, Different programming languages, … args Client (caller) • Serialization format understand by both call send Client • Can have authentication and authorization mechanism r = f(v1, v2); associated with it Stub return receive • Failure in one process isolated from the other unbundle • But may have to take exceptional measures ret vals • Many uses and interaction patterns • Logging process, Window management, … • Moving some system functions out of kernel to user space bundle 9/26/19 cs162 Fa19 L9 24 Server (callee) ret vals return send RPC Binding Server res_t f(a1, a2) Stub • call receive How does client know which machine to send unbundle RPC? • Binding: the process of converting a user-visible args 9/26/19 cs162 Fa19 L9 34 name into a network endpoint • Static: fixed at compile time • Dynamic: performed at runtime • Dynamic Binding • Most RPC systems use dynamic binding via name • Here IPC within one system, service • Why dynamic binding? all proc serve one user • Access control: check who is permitted to access service • Fail-over: If server fails, use a different one • Still IDL allows • Registry at server binds to RPC server stubs 9/26/19 cs162 Fa19 L9 40 interpretation 12/5/19 UCB CS162 Fa19 L27 34 Binder

• AIDL (Android Interface Definition Language) defines interface with method of remote services – Parser generate Java class (Client Proxy and Service Stub) • Java API wrapper – Access to Binder

12/5/19 UCB CS162 Fa19 L27 35 Intent

• Name of Component to start [optional] • Action – View, Pick, Send • Data – URI that references the data to be acted on and/or the MIME type of that data • Category – string containing additional information about the kind of component that should handle the intent. • Extra • Flags

12/5/19 UCB CS162 Fa19 L27 36 In a picture

• Point-to-point or publish/subscribe among components in multiple Proc / App • User task naturally crosses activities – Example: mp3 player UI [Activity] distinct from mp3 player backend [service], could be in separate processes

12/5/19 UCB CS162 Fa19 L27 37 A different system/proc relationship

• Traditional OS is passive – It runs processes when they are exec’d by a running process » init, shell, … – Just one entry point (main), can register handlers – Daemons (continuous running background processes) are controlled by scripts in /etc/rc.conf, /etc/rc.d … – System keeps thing running and reacts to process actions – To interact with a process, it must be running • Android takes a more proactive role – All installed Apps are runnable – App is started when another App needs it – Many entry points for different puposes – App functions as client and server

12/5/19 UCB CS162 Fa19 L27 38 Binder IPC

open “/dev/binder” ioctl on fd write_read, Threads & context User

subsystems process ctrl syscall handlers scheduler Binder mem mgmt syscall tbl intrpt handlers

Kernel file sys intr tbl networking Page Tables user / grp Drivers

regs

Processor Memory Devices Hardware

• GNU/Linux: pipes, shared memory, message queues, sockets. • Android: Binder – derived from Palm OS

12/5/19 UCB CS162 Fa19 L27 39 Binder Protocol

• Write and Read

12/5/19 UCB CS162 Fa19 L27 40 BINDER IPC => RPC

• How is the sender-recipient binding established? – What needs to be running? • Marshaling / Unmarshaling the arguments? – Same machine, so same data representation – Different processes, different address spaces • How are the threads scheduled ? – Concurrent threads or co-routines ? • What about the return result – AND completion?

• Binder originated with BeOS, acquired by Palm, became OpenBinder, then adopted by Android

12/5/19 UCB CS162 Fa19 L27 41 Binder Thread Scheduling

Binder recipient has bounded thread pool to service incoming intents Binder

• The initiator thread effectively transfers to the recipient activity • Max of 15 binder threads per process • Max 1 MB of transaction buffer

12/5/19 UCB CS162 Fa19 L27 42 Manifest file – components and connections • Names the package • Describes each of the App components – Activities, Services, Broadcast Receivers, Providers – Basic properties, e.g., class name – Declare capabilities, e.g, device configs it can handle – Intent Filters • Permissions it needs (system or other Apps) • Permissions others Apps must have to access it • Hardware and software requirements

12/5/19 UCB CS162 Fa19 L27 43 Example

12/5/19 UCB CS162 Fa19 L27 44 Intent

“An intent is a message defined by an Intent object that describes an action to perform, including – the data to be acted upon, – the category of component that should perform the action, and – other instructions.”

• In binder terminology, when one process sends data to another process it is called a transaction – Carries transaction data

12/5/19 UCB CS162 Fa19 L27 45 Intent Filters

• When an app issues an intent to the system, the system locates an app component that can handle the intent based on intent filter declarations in each app's manifest file. • The system launches an instance of the matching component and passes the Intent object to that component. • If more than one app can handle the intent, then the user can select which app to use.

• The pub/sub aspect – the system will find a recipient that can receive the [intent] message and perform the action – Human does the resolution

12/5/19 UCB CS162 Fa19 L27 46 Novel variant of serialization . . .

• Intent message is passed through BINDER to recipient process • IDL defines each of its data types • Binder takes actions while mapping objects • File Descriptor: – Binder: DUP the FD into the recipient process so it too can access the file • Reference to an object (in source address space) – Create an anonymous shared memory segment – Provide a remote reference handle to the object – Reference count them and inform owner when reclaimable – Preserve the object if source exits

12/5/19 UCB CS162 Fa19 L27 47 Intent RPC

• The Intent is a one-way message to take an action – Might pop up a screen allow user to do something and be done • If it needs to communicate a result or convey that it is done, it does so with a callback intent – startActivityForResult to initiate request – onActivityResult() Callback

12/5/19 UCB CS162 Fa19 L27 48 Binding - Services

• Service_manager (handle 0) runs first – BINDER_SET_CONTEXT_MGR ioctl to let Binder know • Provides registry service to other processes • Other processes interact with service_manager

12/5/19 UCB CS162 Fa19 L27 49 Invoking a service

12/5/19 UCB CS162 Fa19 L27 50 Permission

• A central design point of the Android security architecture is that no app, by default, has permission to perform any operations that would adversely impact other apps, the operating system, or the user. • app must publicize the permissions it requires by including tags in the app manifest. • Access to some hardware features (such as Bluetooth or the camera) require an app permission.

12/5/19 UCB CS162 Fa19 L27 51 Content Providers

• ContentResolvers communicate synchronously with ContentProviders via fixed CRUD API • A content provider may want to protect itself with read and write permissions, while its direct clients also need to hand specific URIs to other apps for them to operate on. • when starting an activity or returning a result to an activity, the caller can grant the receiving activity permission access the specific data URI in the intent, regardless of whether it has any permission to access data in the content provider corresponding to the intent. – Allows user action to drives ad-hoc granting of fine-grained permission. Reducing permissions needed by Aps

12/5/19 UCB CS162 Fa19 L27 52 Protection Levels

• Normal permissions – where your app needs to access data or resources outside the app's sandbox, but where there's very little risk to the user's privacy or the operation of other apps. • Signature Permissions – Granted to an app at install time, but only when the app that attempts to use a permission is signed by the same certificate as the app that defines the permission. • Dangerous permissions – cover areas where the app wants data or resources that involve the user's private information, or could potentially affect the user's stored data or the operation of other apps.

12/5/19 UCB CS162 Fa19 L27 53 Activity Lifecycle

• Goes through states • Callbacks to handle transitions between states

12/5/19 UCB CS162 Fa19 L27 54 File System – really?

Recall: From Storage to File Systems Recall: Components of a File System

I/O API and Variable-Size Buffer Memory Address syscalls File path

Logical Index, File System Block Typically 4 KB Directory File Index Structure Structure One Block = multiple sectors File number Ex: 512 sector, 4K block “inumber” Flash Trans. Layer Hardware Data blocks Devices Sector(s)Sector(s) Sector(s) Phys. Block Phys Index., 4KB Physical Index, 512B or 4KB Erasure Page “inode” … HDD SSD Each component can be on disk or (temporarily) in memory. Must be brought into memory to be

10/31/19 CS162 ©UCB Fa19 Lec 18.2 accessed. Ex: Creating a file (as transaction) 11/5/19 CS162 ©UCB Fa19 Lec 19.2

• Find free data block(s)

• Find free inode entry

• Find dirent insertion point Free space ------map

• [log] Write map (used) … Data blocks • [log] Write inode entry to point to Inode table block(s) • [log] Write dirent to point to inode Directory entries tail head

done pending start commit Log: in non-volatile storage (Flash or on Disk) 11/5/19 CS162 ©UCB Fa19 Lec 19.41 12/5/19 UCB CS162 Fa19 L27 55 YAFFS – Yet Another Flash File System

• Designed specifically for NAND flash in 2002 • uses journaling, error correction, and verification techniques tuned to the way NAND behaves and typically fails • Log structured as blocks of chunks – Data Chunks and Object Headers (directory, hardlink, soft link, …) – Efficient garbage collection of chunkless blocks • RAM structures cache and find things quickly in log – For each file object, YAFFS holds tree of tnodes to find data chunks in file – Object has parent link and name – Directory structure – doubly linked sibling list – Object number has table – find object from obj_iod – Cache

12/5/19 UCB CS162 Fa19 L27 56 YAFFS - tnode

12/5/19 UCB CS162 Fa19 L27 57 Android power mgmt - WakeLock

• Linux has two power management methods – APM ( Advanced Power Management) – ACPI ( Advanced Configuration and Power Interface) – Neither are sufficient for smartphones • Android tries to put the system to sleep as soon as possible • Apps request the Power Manager to stay awake

Wake Lock CPU Screen Keyboard PARTIAL_WAKE_LOCK ON OFF OFF

SCREEN_DIM_WAKE_LOCK ON DIM OFF

SCREEN_BRIGHT_WAKE_LOCK ON BRIGHT OFF

FULL_WAKE_LOCK ON BRIGHT BRIGHT

12/5/19 UCB CS162 Fa19 L27 58 That’s a cs162 tour of what’s in it

12/5/19 UCB CS162 Fa19 L27 59 My take … • Linux kernel provides hardened basics – process, threads, scheduling, memory mgt, address spaces, device drivers, … – Recent advances bring little additional benefit to this platform – Mainlining likely to result in factoring common core • Android extensions provide fundamentally greater isolation AND integration – The cooperative ensemble of system & apps, defined in total by a manifest may infuse to laptop, PC, & cloud – Container (esp. with VM too) is step, but more potential • with shared objects is powerful – cf, Go Channels • Deep power proportionality is critical • All file systems will be SSD soon, so … • Much more room to utilize connectedness

12/5/19 UCB CS162 Fa19 L27 60 So What do you think?

12/5/19 UCB CS162 Fa19 L27 61 Post-cs162 perspective

Course Structure: Spiral

intro

12/4/19 UCB CS162 Fa19 L1 29

12/5/19 UCB CS162 Fa19 L27 62