Security Event Auditing of Infotainment and Communications Control: Implementation Approaches

K. Dorozhkina

Security event auditing of infotainment and communications control: implementation approaches

Karina Dorozhkina

Kaspersky, 39A/2 Leningradskoe Shosse, 125212, Moscow, Russian Federation

Abstract. Today’s cars are deservedly called smart because they are no longer merely vehicles but complex devices communicating with each other and the outside world. With integrated infotainment (head unit ECU or HU) and communication (telematics ECU or TCU) control domains, they offer numerous ways to make the driver's life more comfortable. HUs and TCUs are powered by Linux or Android OS, similar to your mobile phone or tablet, and their hardware performance is more on a par with PCs than embedded systems. But as well as greater convenience, these enhanced vehicle capabilities bring greater responsibilities as the attack surface grows. For instance, it has been shown by several research groups that unauthorised remote access can result in CAN bus injection. Countering complex cyberattack scenarios requires security event monitoring inside the vehicle’s ECUs and the ability to deliver this information to vehicle operators (e.g., OEM or fleet owner) for incident discovery and response. Continuous asset monitoring in Security Operations Centers became the standard in enterprise-grade security with Endpoint Detection & Response solu- tions (EDR). Upcoming automotive cybersecurity regulations (UNECE WP.29/GRVA and ISO/SAE 21434) also cover this process. This paper gives a detailed description of security events from HU and TCU critical for malicious activity analysis as well as key mechanisms for their extraction. The target plat- forms are automotive embedded Linux distributions such as Automotive Grade Linux (AGL) and GENIVI. Mature systems such as udev and Linux Auditing will be reviewed along with the relatively new extended Berkeley Packet Filter (eBPF).

Keywords: Head unit, Telematics, Linux, Audit, eBPF

1 Introduction

Complex and comprehensive vehicle attack scenarios include payload delivery to cloud and internally connected ECUs (like head units and telematic control units) through MiTM attacks or external interface exploitation, long-term persistence of malicious code in the compromised ECUs and its lateral movement to other domains. The target of such attacks are unlikely to be individuals, but rather enterprises with business processes tightly connected with vehicle fleets. For example, in 2018 and

16 Security event auditing of infotainment and communications control

2019 successful cyberattacks were carried out against large car-sharing fleets in the US and Australia. Infotainment control and telematics provide the driver with a lot of useful functionality. It’s hard to imagine driving now without navigation maps or a USB port for a device with your favourite music. But, more importantly, it’s not just entertainment that can be provided by these ECUs – safety functions such as crash reporting and emergency warning by eCall or over-the-air software update (OTA) are key features of modern vehicles. All this functionality leads to a growing number of external interfaces in today’s smart cars (Fig. 1).

Fig. 1. External interfaces of a smart car

This transformation from a simple vehicle to a sophisticated connected device has a downside – a growing number of potential vulnerabilities. With the ability to con- nect to the outside world, HUs and TCUs can be used as an entry point to car inter- nals. These domains obviously need to be isolated from other domain networks. But it

17 K. Dorozhkina

turns out that in the real world the situation isn’t so clear cut and often it’s possible to gain access to the powertrain or body control via CAN bus, or hacking the TCU or HU via external interfaces. The main aim here is to make infotainment and communication control more secure and more resistant to potential attacks. Leaving prevention functionality for a future talk, this paper focuses on the important function of secure auditing. This function makes it possible for incident specialists to analyse malicious activity and report to a driver or OEM. Of course, this is not the only mechanism that should be used. Strong authentication and authorisation, strict policies and critical updates to remove old vulnerabilities should all be applied. But secure auditing is also an important part of car protection, giving an understanding of what went wrong and how. And it’s not just for cases of remote hacking – secure auditing can assist insurance companies with forensic analysis in cases of physical theft [1]. Numerous studies [1, 2] as well as upcoming automotive cybersecurity regulations (UNECE WP.29/GRVA and ISO/SAE 21434) stipulate that security logging is an obligatory part of securing a modern car. But there is no definition of which events are important and should be logged. The aim of this paper is to address the theme of security events in detail. Before we go into an in-depth explanation of security events and extraction methods, let’s define the scope of potential infotainment and communication control operating systems (OS). Security event groups are independent from the OS, but the mechanisms for collecting events may differ. It’s impossible to describe all cases, so this paper will concentrate mainly on two popular automotive Linux distributives, Automotive Grade Linux (AGL) and GENIVI, which are both based on the yocto framework. Renesas R-Car H3 board, using ARM architecture, was used as a refer- ence hardware platform.

2 Security event groups

2.1 Defining the scope OSs such as GENIVI, AGL or QNX are highly complex software with a non-stop event flow. It’s impossible to collect and report all system activity due to disk space limitations or network bandwidth if we want to send these logs to the cloud. Moreo- ver, we need to filter out events to simplify future analysis and collect only those events that relate to possible incidents. Redundant event collecting can also lead to performance degradation. For example, we aren’t interested in open(), read() and write() syscalls for all filesystem nodes, but only for those that have restricted access, such as configuration files. Defining the full scope of security events requires a threat model covering all possible attack surfaces [1]. But for simplicity and as a case study for this paper, let’s review an existing Tesla attack (Fig. 2) [3]. The entry point was via Wi-Fi spoofing and the subsequent exploiting of a vulner- ability in a WebKit browser. In this context, the collecting of network events seems reasonable. Another step in this attack was patching setresuid() syscall to get the root privilege. This in turn leads to the idea of collecting specific syscalls information

18 Security event auditing of infotainment and communications control

4 related to privilege escalation and processes creation. Further manipulations led to access being gained to the gateway through its reprogramming. This indicates that the filesystem is heavily used in attempts by the intruder to achieve persistence. From this case we see three types of security events: network events, syscall events related to privilege escalation, and process creation and filesystem events.

Fig. 2. TESLA attack scheme.

As was mentioned above, it’s not only remote attacks that are possible but also physical attacks involving theft or cases of car sharing. TCU and HU external interface activity such as USB activity must be recorded and it adds one more event group for connected device information. The following sections will describe which mechanisms can be used for extracting these events.

2.2 Enabling auditing systems in yocto distributions As our target systems are AGL and GENIVI distributions, both based on the yocto framework, a custom setup for all systems is necessary that will be described below. Almost all systems for security auditing are disabled in yocto images built from scratch. So, it’s necessary to define a custom layer with a kernel and image recipes for adding extra kernel config options and deploying the required packages [11]. Below are some examples.

Additional kernel config example for Linux Auditing system:

CONFIG_AUDIT=y CONFIG_AUDITSYSCALL=y CONFIG_AUDIT_WATCH=y CONFIG_AUDIT_TREE=y CONFIG_INTEGRITY_AUDIT=y

Additional custom yocto layer packages:

IMAGE_INSTALL_append += " audit" IMAGE_INSTALL_append += " audit auditd "

19 K. Dorozhkina

5 6

IMAGE_INSTALL_append += " udev-rules " IP address assigning to USB to Ethernet adapter: IMAGE_INSTALL_append += " iproute2 “ network_broadcast_address=169.254.255.255 network_interface_name=eth0 network_local_address=169.254.245.57 network_address_family=AF_INET network_address_action=added network_interface_address=169.254.245.57 network_address_mask_length=16 network_address_scope=global

Such network events can serve as the start point in incident investigation as they relate to external interface activity. After a malicious connection is established an attacker may try to invoke syscalls like setresuid() in the Tesla example or launch binaries. It is then interesting to analyse the syscalls sequence and process tree creation.

2.4 Syscall events

The Linux Auditing system was first introduced in kernel version 2.6 and is available in all the latest distributions, including GENIVI and AGL. The core functionality is Fig. 3. Security event groups and collecting methods the logging of kernel activity for future incident investigation. As is shown in this paper, using only this mechanism is not enough because it doesn’t cover the full net- 2.3 Network events work or device activity (Fig. 3), but can give a lot of information about syscalls used, including process creation events. Based on these events we can track the whole pro- Linux distributions such as Genivi and AGL use the Netlink service for communica- cess tree with arguments and analyse executed sequences. For example, we can detect tion between the kernel and user space. This allows communication with the rtnetlink spoofed firmware updates and its initiator or compare the current process flow with a system that provides read/write access to routing tables. Rtnetlink messages are based normal flow and detect malicious or unusual executed binaries. on netlink messages that can be received via netlink socket creation with the In order to track syscall activity, we need to add a rule for the syscall in question to NETLINK_ROUTE protocol. Such messages contain information about the addition the auditing system: and removal of interfaces, address assigning, route changes or wireless connection. As was shown previously, the entry point for the Tesla attack was Wi-Fi spoofing. In -a exit,always -F arch=b64 -S execve this context rtnetlink can be very useful to track ethernet interface activity. There are -a exit,always -F arch=b32 -S execve several examples of network events that can be captured (some fields are adapted to human readable format). We will then receive an event whenever execve is used: Connection to Wi-Fi AP: audit_message=audit(1591177619.614:338225): argc=2 network_access_point_bssid=c8:ff:28:7e:fb:a7 a0=“/usr/bin/python" a1="/var/health/health.py" au- network_interface_action=added dit_message_type=EXECVE network_interface_state=UP(NOT RUNNING) network_interface_name=wlan0 audit_message=audit(1591177579.537:338192): argc=6 network_access_point_ssid=WorkshopAP a0="/sbin/agetty" a1="-8" a2="-L" a3="ttyS1" a4="115200" a5="xterm" audit_message_type=EXECVE

20 Security event auditing of infotainment and communications control

IP address assigning to USB to Ethernet adapter: network_broadcast_address=169.254.255.255 network_interface_name=eth0 network_local_address=169.254.245.57 network_address_family=AF_INET network_address_action=added network_interface_address=169.254.245.57 network_address_mask_length=16 network_address_scope=global

2.4 Syscall events The Linux Auditing system was first introduced in kernel version 2.6 and is available in all the latest distributions, including GENIVI and AGL. The core functionality is the logging of kernel activity for future incident investigation. As is shown in this paper, using only this mechanism is not enough because it doesn’t cover the full network or device activity (Fig. 3), but can give a lot of information about syscalls used, including process creation events. Based on these events we can track the whole process tree with arguments and analyse executed sequences. For example, we can detect spoofed firmware updates and its initiator or compare the current process flow with a normal flow and detect malicious or unusual executed binaries. In order to track syscall activity, we need to add a rule for the syscall in question to the auditing system:

-a exit,always -F arch=b64 -S execve -a exit,always -F arch=b32 -S execve

We will then receive an event whenever execve is used: audit_message=audit(1591177619.614:338225): argc=2 a0=“/usr/bin/python" a1="/var/health/health.py" au- dit_message_type=EXECVE audit_message=audit(1591177579.537:338192): argc=6 a0="/sbin/agetty" a1="-8" a2="-L" a3="ttyS1" a4="115200" a5="xterm" audit_message_type=EXECVE

21 K. Dorozhkina

There are other mechanisms for logging process creation such as the Connector kernel module available via the Netlink system. But it lacks information about process arguments. The Linux Auditing System can also be fine-tuned via different filters, for example, by process pid and its logging surface can be extended with filesystem events.

2.5 Filesystem events The Linux Auditing system can not only track syscall activity but also filesystem access. It’s possible to track the type of access, for example, read or write, who got access and on which node. Rule example:

auditctl -w /etc/pam.d/test_file -p rwxa

Event example for file creation:

audit_message=audit(1591597008.259:20456): item=1 name="/etc/pam.d/test_file" inode=69210 dev=b3:02 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 audit_message_type=PATH “

It should already be clear that instead of logging everything in the system, the audit tells us only what we ask for. There is no difficulty in implementing the auditing logic itself, the main question is what to log. We need to analyse ourselves the possible attack vectors, potential files with sensitive information, syscalls granted root privi- leges and, based on all this information, define the threat model and audit rules. Ex- amples of sensitive files and directories may be /etc subdirectories related to authentication, for example, /etc/pam.d, and /etc/pam.conf, or directories containing ssl certificates such as /usr/share/ca-certificates or /etc/ssl/certs.

2.6 Device events All previous event group analysis was based on possible remote access. But there is also the possibility of physical attack based on exploitation of other HU or TCU external interfaces such as USB for example. An attacker may use a USB to Ethernet adapter or USB flash with malicious software update. The Linux udev system makes it possible to manage different devices, including notifications about device addition, removal or changes. Udev will report all device information, including vendor and product ID, by which it’s possible to detect device type. With this information we can detect when a device not intended for in-car use was attached. Or the information can provide a start point for further investigation after a USB flash was inserted. Below is an example of such a udev event, making it

22 Security event auditing of infotainment and communications control

8 possible to define by vendor and product ID that a D-Link Ethernet adapter was attached: host_hwd_device_authorized=no host_hwd_device_property_SUBSYSTEM=usb host_hwd_device_property_PRODUCT=2001/4a00/100 host_hwd_device_vendor_id=0x2001 host_hwd_device_property_ID_VENDOR_FROM_DATABASE=D-Link Corp. host_hwd_device_property_DEVNAME=/dev/bus/usb/001/008 host_hwd_device_property_DEVTYPE=usb_device

3 Common event provider eBPF

It was shown in previous sections that Linux distributions, including AGL and GENIVI, expose different mechanisms for collecting security events. Only the most informative and commonly used are listed above. To get the full picture, it’s necessary to use them in a bundle, implementing ways for them to interact. This also raises the question of a single universal mechanism for collecting such events. Extended Berkeley Packet Filter (eBPF) was only introduced quite recently. By kernel version 4.9 it had developed from its classic Berkeley Packet Filter origin into a complex tracing, profiling and instrumental system. It’s not clear if this was intended or not, but it’s now possible to cover all the functionality described above with eBPF (Fig. 4).

Fig. 4. Security event sources and dynamic tracing with eBPF.

23 K. Dorozhkina

eBPF allows us to write programs that can be attached to the kernel routine via kprobe. Below are bcc and bpftrace frameworks implemented for writing eBPF programs and facilitating the first steps [6]. These frameworks give quite a lot of examples of tracing process and network activity, but lack examples of how a new USB device can be traced by setting kprobe on Linux USB Core API. However, an example is shown below, using the bcc framework. The Python environment required by eBPF programs results in C-code anyway, which is then compiled into eBPF byte code and executed on a kernel virtual machine:

from bcc import BPF bpf_text = """ #include #include

int kprobe__usb_get_dev(struct pt_regs* ctx, struct usb_device* dev) { //... return 0; }; int kretprobe__usb_get_dev(struct pt_regs* ctx) { //... return 0; } """ b = BPF(text=bpf_text) while 1: (task, pid, cpu, flags, ts, msg) = b.trace_fields() print(msg)

The usb_get_dev() function from USB Core API is called to increment the refer- ence count of the USB device structure, in other words when a USB device is attaching. Probing this function with eBPF allows us to get the usb_device structure at the moment of the function call and extract all the necessary information, including vendor ID and product ID. It’s also possible to instrument a function return point via kretprobe.

4 eBPF compared to other auditing systems

As eBPF is a brand-new feature and there are a lot of blank areas, it’s often said that it’s hard to write an eBPF program. It iso if you are a system administrator, which is why they should probably set up the Linux Auditing system instead. But if you are

24 Security event auditing of infotainment and communications control

10 writing kernel modules, the eBPF system is quite intuitive and its flexibility and cov- erage is impressive. One advantage of eBPF is that it can be extended to provide prevention functionality, dropping unwanted packets or blocking malicious USB devices. It improves on the Linux Auditing system which can only log. In terms of prevention functionality, using eBPF programs is also safer than writing kernel modules with the same kprobe because the kernel runs a verifier on each eBPF program. In contrast, a buggy kernel module can crash the whole system. Using kprobe gives us almost unlimited possibilities to install a probe anywhere in the kernel. eBPF limitations are the same as kprobe limitations. Attaching kprobe will return -EINVAL if you attempt to install a probe in the code that implements kprobes (mostly kernel/kprobes.c and ``arch/*/kernel/kprobes.c``, but also functions such as do_page_fault and notifier_call_chain). But such limitations are not critical for our task of auditing security events in automotive systems. Also, when comparing eBPF with the Linux Auditing system, the latter’s reports include lengthy logs related to just one event. Just for comparison, one execve() call will produce the following sequence with the final EOE message type:

audit_message=audit(1591177619.646:338231): arch=c000003e syscall=59 success=yes exit=0 a0=557d6e961c70 a1=557d6e961f50 a2=557d6e95ff20 a3=557d6e957010 items=2 ppid=9806 pid=9808 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="touch" exe="/bin/touch.coreutils" subj=System key=(null) audit_syscall=execve au- dit_message_type=SYSCALL audit_message=audit(1591177619.646:338231): cwd="/" au- dit_message_type=CWD audit_message=audit(1591177619.646:338231): item=0 name="/bin/touch" inode=88 dev=03:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=_ nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 audit_message_type=PATH audit_message=audit(1591177619.646:338231): argc=2 a0="touch" a1="/tmp/healthy" audit_message_type=EXECVE audit_message=audit(1591177619.646:338231): item=1 name="/lib/ld-linux-x86-64.so.2" inode=306 dev=03:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=_ nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 au- dit_message_type=PATH audit_message=audit(1591177619.646:338231): procti- tle=7368002D6300746F756368202F746D702F6865616C746879 au- dit_message_type=1327

25 K. Dorozhkina

audit_message=audit(1591177619.646:338231): audit_message_type=EOE

It’s obvious that Linux Auditing output should be processed to reduce event traffic size. In eBPF, by contrast, we can log what we want. So, eBPF is preferable if you need a single mechanism for secure auditing and also want to implement protection from malicious events in future. But as eBPF is a relatively new feature, for backward compatibility and support of older kernel versions it’s necessary to use mechanisms such as udev, rtnetlink and audit, described in the previous sections.

5 Conclusion

As modern HUs and TCUs are relatively powerful hardware with operating systems such as Linux, Android or QNX, it’s possible to track all malicious activity and log it for future incident analysis using different system instruments and tracing features out of the box. This paper highlights several event domains that may be of interest for analysing and reconstructing attacks as well as a comparison of several Linux mechanisms for extracting events. Of course, just switching on all the available auditing systems is not enough. Deep analysis of threats needs to be carried out and a threat model description created with all the attack vectors. After that it’s possible to configure the security auditing system. This approach can be applied to other OSs, such as the popular real-time QNX. Despite being POSIX compliant it has other mechanisms for event extraction without audit, udev and eBPF, but the secure auditing approach and event group is the same and can be discussed in future talks. Due to continuous system development and updates, it’s necessary to use as up-to- date software as possible both to avoid old vulnerabilities and to make use of the latest features such as eBPF, extending opportunities for any kernel routine tracking.

References

1. The European Union Agency for Cybersecurity (ENISA): Cyber Security and Resilience of smart cars. Good practices and recommendations (2016). 2. ACEA Homepage, https://www.acea.be/publications/article/acea-principles-of- automobile-cybersecurity, last accessed 2020/06/05. 3. Sen Nie, Ling Liu, Yuefeng Du: Free-fall: Hacking Tesla from wireless to CAN bus. BlackHat USA 2017. 4. J. Corbet: Extending extended BPF. Linux Weekly News, 2014. 5. B. Gregg. Linux Extended BPF (eBPF) Tracing Tools. http://www.brendangregg.com/ebpf.html, 2016. 6. IO Visor Project. BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more. https://github.com/iovisor/bcc, 2015.

26 Security event auditing of infotainment and communications control

7. S. McCanne and V. Jacobson. The BSD Packet Filter: A New Architecture for User-level Packet Capture. In USENIX winter, volume 93, 1993. 8. Dave (Jing) Tian, Grant Hernandez, Joseph I. Choi, Vanessa Frost, Peter C. Johnson, Kev- in R. B. Butler: LBM: A Security Framework for Peripherals within the Linux Kernel. IEEE Symposium on Security and Privacy, 2019. 9. Linux kernel documentation Homepage, https://www.kernel.org/doc/, last accessed 2020/06/05. 10. Michael Kerrisk Linux man Homepage, https://man7.org, last accessed 2020/06/05. 11. Yocto Project Development manual Homepage, https://www.yoctoproject.org/docs/1.8.2/dev-manual/dev-manual.html, last accessed 2020/06/05. 12. GENIVI Resource Documents Homepage, https://www.genivi.org/resource-documents, last accessed 2020/06/05.