Cisco IOS XR memory forensics analysis Solal Jacob Agence Nationale de la Sécurité des Systèmes d’Information 2019 Plan I I- IOS XR internals & forensics analysis I II - Attack simulation I III - Detection ANSSI Cisco IOS XR memory forensics analysis 2/36 I- IOS XR internals & forensics analysis I We would like to be able to analyse a router to know if it was compromised I For that we want to develop memory forensics tools to detect advanced attack I We choose IOS XR because it’s an exotic system used on core routers ANSSI Cisco IOS XR memory forensics analysis 3/36 IOS XR I Used in Cisco routers (1200, ASR9000, ...) I 32 bits version only I Based on QNX 6.4 ANSSI Cisco IOS XR memory forensics analysis 4/36 QNX I Microkernel released in 1982, now part of Blackberry I Used in embedded system : Routers, Infotainment, Telematics (Westing House, AECL, Air traffic Control, General Electric) I Source was released then closed again ANSSI Cisco IOS XR memory forensics analysis 5/36 QNX architecture I Fault tolerent I Reduced kernel attack surface I Conform to posix standard I Customizable by OEM ANSSI Cisco IOS XR memory forensics analysis 6/36 QNX Security & Forensics I Some CVEs I No hardening before 6.6 I Troopers 2016, QNX : ”99 Problems but a Microkernel ain’t one !” (Vuln in message passing & IPC) I Recon 2018, ”Dissecting QNX” (Mitigation & PRNG) I No forensics papers or presentations ANSSI Cisco IOS XR memory forensics analysis 7/36 QNX startup I The IPL, Inital Program Loader, initialize the hardware, configure the memory controller, load the system image in RAM and jump to it I The startup code make further hardware initilization, launch the microkernel procnto in virtual mode, put all configs info into the system page I procnto run the boot script and launch other processes (path manager, network stack, ...) ANSSI Cisco IOS XR memory forensics analysis 8/36 QNX Firmware I IFS : Image file system, read-only (procnto, bootscript, drivers, ...) I EFS : Embedded file system, read-write (program, data, utilities, ...) I Combined image that can be flashed directly on NAND Figure – Combined image I Blackberry provide tools to create and read those images ANSSI Cisco IOS XR memory forensics analysis 9/36 Communication between processes I IPC : Use a message passing system I Messages are synchronous and directed towards channels and connections rather than threads I A thread create a channel to receive messages I An other thread can make a connection by ”attaching” to that channel, then send messages Figure – Combined image ANSSI Cisco IOS XR memory forensics analysis 10/36 The message passing system I Channels and connections have each an assigned file descriptor I When a thread create a channel, it can register a path I Processes can open these path via the path manager, that return the file descriptor needed to communicate I Messages are passed by being copied from the address space of one thread to the address space of an other thread I There is very few syscall under QNX 6.4 (~100) I The libc hide the message passing system like Linux libc hide syscall ANSSI Cisco IOS XR memory forensics analysis 11/36 Linux libc syscall wrapper I fd = open(”file”) ; fd = syscall(open_syscall_number, "file"); I write(fd, ”abcd”, 4) ; ret = syscall(write_syscall_number, fd, "abcd", 4); I close(fd) ; ret = close(close_syscall_number, fd); ANSSI Cisco IOS XR memory forensics analysis 12/36 QNX libc message wrapper I fd = open(”myfile”) ; fd = ConnectAttach(PATHMGR_COID, "myfile", 1, 0, 1); sent_msg.type = IO_CONNECT sent_msg.data = "myfile" sent_msg.path_len = strlen("myfile"); MsgSend(fd, sent_msg, sent_msg_size, reply_msg, reply_msg.size); ConnectDetach(fd); We connect to the service and we ask for a fd for this path (reply->pid is the pid of the process that handles the hard disks) fd = ConnectAttach(reply->nd, reply->pid, reply->chid, 0, 0); MsgSend(fd, sent_msg, sent_msg_size, reply_msg, reply_msg_size) ANSSI Cisco IOS XR memory forensics analysis 13/36 QNX libc message wrapper I write(fd, ”abcd”, 4) ; sent_msg_buffer.type = IO_WRITE sent_msg_buffer.nbytes = 4 sent_msg.buffer.data = "abcd" MsgSend(fd, sent_msg_buffer, sent_msg_buffer_size, ret_msg_buffer, sizeof( ret_msg_buffer)); I close(fd) ; sent_msg.type = IO_CLOSE sent_msg.size = sizeof(sent_msg); ret = MsgSend(fd, sent_msg_buffer, sizeof(sent_msg), 0, 0); ConnectDetach(fd); ANSSI Cisco IOS XR memory forensics analysis 14/36 Memory acquisition I Request memory mapping via the memory manager services I Interfaced via mmap_device_memory I All physical memory is directly addressable I No kernel drivers needed ANSSI Cisco IOS XR memory forensics analysis 15/36 Memory acquisition tool I Transfer the memory content via a network socket (to a listening netcat) on QNX I Cisco add it’s own services and network stack I They use a modified version of GCC to generate specific executables I A second process manager services is used to launch those executables (They have a JID instead of PID) I Cisco modifed top and others commands to list only application with a JID I It’s difficult to generate binary that link to the libsocket and the Cisco network stack I To create a socket it’s possible to use the message system directly ANSSI Cisco IOS XR memory forensics analysis 16/36 Socket without libsocket sprintf(buff, "/dev/socket/2/1/0/1610612736/%d", getpid()); int sock = open(buff, 2, 0); struct sockaddr_in client; ... int ret; my_io_devctl_t msg; iov_t siov[2]; iov_t riov[2]; msg.i.type = _IO_DEVCTL; msg.i.combine_len = sizeof(msg.i); /*msg.i.dcmd = 0x80100603; under QNX 6.4.0 */ msg.i.dcmd = 0x80048004; msg.i.nbytes = sizeof(client); msg.i.zero = 0; (siov)->iov_base = (void*)&msg.i; (siov)->iov_len = sizeof(msg.i); (siov + 1)->iov_base = (void*)client; (siov + 1)->iov_len = sizeof(client); (riov)->iov_base = (void*)&msg.o; //(siov)->iov_base; (riov)->iov_len = sizeof(msg.o); ret = MsgSendv(sockfd, siov, 2, riov, 1); ANSSI Cisco IOS XR memory forensics analysis 17/36 Executing the acquisition tool on IOS XR I The memory acquistion tool can be transfered to IOS XR via ssh I It can’t be run directly because Cisco removed the chmod tool I To made the file executable, there is a trick : copy a binary with executable right in the user writable directory and then copy a binary over it ANSSI Cisco IOS XR memory forensics analysis 18/36 Analysis of the memory dump I The pidin tool, that list lot of system informations, was studied I It reveal the use of syspage_entry structure, that point to lot of other interstings structures I To read differents structures from the dump we need to know their physical adresses I The virtual address of the syspage_entry struct can be listed via pidin I The syspage_entry structure is in the adress space of procnto I procnto cr3 value is needed to convert syspage_entry virtual address to physical one ANSSI Cisco IOS XR memory forensics analysis 19/36 procnto address space I To find procnto cr3 value, qemu is used to list tlb and find ”constant” value I It use identity mapping to translate virtual addresses to physical one I The cr3 value is constant across boot, the value can be found in the IPL (that launch procnto) I procnto virtual adress can be convert to physical one thanks to this value I syspage_entry and lot of user structures can be read I Processes, memory map, channels, file descriptor, ... can then be listed ANSSI Cisco IOS XR memory forensics analysis 20/36 Connection and channel graph ./qnxmemdump (1966121) I QNX processes use IPC, known sbin/devc-pty (139283) sbin/io-pkt-v4-hc (110609) as channel, to communicate with other process proc/boot/devc-con-hid (4103) I All the structures containing proc/boot/io-hid (4102) informations about connections and channels are readable from proc/boot/io-usb (4101) the memory dumps I A graph could be created to proc/boot/procnto-smp-instr (1) visualize all the connections I The graph can be used, for proc/boot/devb-eide (8200) example, to know if a process use the network stack (as in proc/boot/slogger (4100) proc/boot/pci-bios (4099) microkernel drivers are processes) ANSSI Cisco IOS XR memory forensics analysis 21/36 Processes address space I Each processes as it’s own address space I To extract each processes and their memory allocation, for further analysis, the physical address of their differents segments in memory are needed I For that we need their cr3 value I They are found by following structures linked to syspage_entry I Once we have cr3 we use it to read the PTE and other structures in order to do virtual to physical translation (PAE is used un IOS XR) I We can then access all the address space of a processes (the segment of the executables mapped in memory and the different allocation made by the processes) ANSSI Cisco IOS XR memory forensics analysis 22/36 Reconstructing the binaries I Only the data and text segment address is listed in procnto structures I QNX and IOS XR binary structure differs (but QNX binaries are also found in IOS XR) I Thery’re both ELF ANSSI Cisco IOS XR memory forensics analysis 23/36 QNX binaries I QNX binaries are dynamic and have differents kind of segments loaded I We can’t know the address of the dynamic segments I In memory text segment is a direct mapping of the offset zero of the binary I So, it’s easy to read the ELF header I The header can be used to rebuild a partial binary containing only the text and data segments ANSSI Cisco IOS XR memory forensics analysis 24/36 IOS XR binaries I The text segment is always located at 0x1000 in the binaries (it start with a NIAM header)