CEC 450 Real-Time Systems Lecture 10 – Device Interface Drivers and MMIO October 27, 2020 Sam Siewert MMIO Interfacing to Off-Chip Devices Sam Siewert 2 Camera Demo - Linux Devices Computer Vision Application Demo Camera Applications – OpenCV API – User Space API (Examples) On Linux Laptop – V4L2 API – Kernel, User Space Interface (Example) Hit “q” in imshow window – UVC – Driver, Kernel Space Module (driver page) To properly shutdown – USB – Driver, Kernel Space Module Reset USB bus and device sudo cat /proc [PIDs, meminfo, If you happen to exit badly – e.g. cpuinfo,interrupts, slabinfo, devices, bus/pci, With Ctrl-C bus/input, …] [sigint could catch and handle] Use syslog instead of printf sudo lshw, lsusb, lspci, lsscsi, lsmod, lscpu, • Buffers for output in slack slabtop [more] time • Avoids immediate dmesg, top, iostat from sysstat [more]; blocking I/O that can lsof, netstat, strace, vmstat [more] delay a service • Similar features Trace, Profile, Debug Tools (coming up …) – • Add custom time-stamp – KernelShark (sudo apt-get install Linux kernel modules ), kernelshark • Must use “printk” – wireshark (sudo apt-get install wireshark), – ftrace, sysprof, systemtap, etc. • Kernel Modules, Drivers Sam Siewert 3 Embedded I/O (HW View) Analog I/O – DAC analog output: servos, motors, heaters, ... – ADC analog input: photodiodes, thermistors, ... Digital I/O – Direct TTL I/O or GPIO – Digital Serial (I2C, SPI, ... - Chip-to-Chip) – Digital Interface to UART/Line Driver, Serializer- deserializer (Serdes), or Physical I/F Digital I/O to Line Driver Interface (e.g. RS232) 10/100 MII, 1G RGMII, 10G XGMII interfaces to Serdes to Phy Sam Siewert 4 CPU Core I/O (HW View) Word – Register Control/Config, Status, Data – Typical of Low-Rate I/O Interfaces (RS232) Block – FIFOs (2-32K), Dual-Port RAM and DMA – Typical of High-Rate I/O Interfaces System Memory Map (32 or 64 bit space) – Physical Memory (SDRAM, DDR, SRAM) – BootROM (Typically Flash) – Configuration and Control Registers Sam Siewert 5 Example 4Gb System Memory Map 0xFFFF_FFFF 1 Mbyte Boot ROM device Boot ROM (Flash) (reset vector address @ high address) 0xFFF0_0000 0xFFEF_FFFF 4015 Mbytes unused 0x0500_0000 0x04FF_FFFF 16 Mbytes Memory Mapped IO MMIO (PCI BARs for Device 0x0400_0000 Function Registers) 0x03FF_FFFF 32 Mbytes unused (space left for memory upgrades) 0x0200_0000 0x01FF_FFFF Main Working Memory for OS/Apps Working Memory (e.g. 32 Mbytes SRAM, SDRAM, DDR) 0x0000_0000 Sam Siewert 6 OS and App Use of 32 Mb Memory 0x01FF_FFFF App Code Heap Loadable App modules (.text, .data, .bss, .rom) ISR Stack ISR_STACK_SIZE WDB Pool WDB_POOL_SIZE _end+1 _end Loadable VxWorks image System Code (.text, .data, .bss, .rom) 0x0010_8000 System Stack 32K for kernel – grows down 0x0010_0000 0x0009_FFFF Bootrom Image 608 Kbyte Boot_rom image 0x0000_8000 Bootrom Stack 12K stack for boot – grows down 0x0000_5000 256 Bytes bootDev, unitNum, Boot Parameters 0x0000_1200 procNum, flags 0x0000_07FF Interrupt Vector Table 2 Kbytes for IRQ 0-15 Handlers 0x0000_0000 (128 Bytes, 32 Dwords of Code) Sam Siewert 7 Platform/RTOS I/O (FW View) Driver Interfaces Used By Applications – Top-Half: Application Interface – Bottom-Half: FW Interface to HW Application Interface Examples – Word-at-a-Time I/O Drivers Serial byte streams Device Configuration, Control, and Status – Block I/O Drivers Disk, Flash, High-Rate Network (Typically DMA) – Stack I/O Drivers Filesystem, TFFS/DOC, TCP/IP Sam Siewert 8 Platform/RTOS I/O (FW View) Device Interface Examples – Word-at-a-Time RS-232/422 UART, GPIO, RELAY, ADC – Block I/O Common Flash Interface, Ethernet Link Layer – Stack I/O Last Stack Layer Interface to HW/SW Layer – HW/SW Abstraction Exported To Higher Layers – Only This Layer Must Change When HW is Changed – E.g. Flash Filesystem MTD in TFFS Sam Siewert 9 Driver Design Application(s) open/close, read/write, Application Interface EAGAIN, Block, creat, ioctl Data, Status If Output If Input – Application Policy Ring-Buffer Full then Ring-Buffer Empty then {SemTake {SemTake – Blocking/Non-Blocking or EAGAIN} or EAGAIN} else else {Process and Return} {Processand Return} – Multi-thread access – Abstraction Output Input SemGive Device Interface Ring-Buffer Ring-Buffer – SW/HW Interface – Immediate Buffering ISR – Interrupt Service Routine Hardware Device HAL Interface Sam Siewert 10 Cached Memory and DMA Cache Coherency – Making sure that cached data and memory are in sync – Can become out of sync due to DMAs and Multi-Processor Caches – Push Caches Allow for DMA into and out of Cache Directly – Cache Snooping by HW may Obviate Need for Invalidate Drivers Must Ensure Cache Coherency – Invalidate Memory Locations on DMA Read Completion cacheInvalidate() – Flush Cache Prior to DMA Write Initiation cacheFlush() IO Data Cache Line Alignment – Ensure that IO Data is Aligned on Cache Line Boundaries – Other Data That Shares Cache Line with IO Data Could Otherwise Be Errantly Invalidated Sam Siewert 11 Disadvantages of Abstracted Driver Generally the Software Engineering Value Outweighs any Inefficiency PDL – Peripheral Driver Library, Focus on Device Interfaces [Devices Interfaces and HAL] Full Driver Abstraction Adds Overhead and Complexity – Function calls and table look-up – More Code than Single Layer API Adds Buffering and Copy Features – Can Use Zero-Copy Buffering (Pointers) to Minimize Impact – Zero-Copy Buffers can be Complex Sam Siewert 12 Advantages of Abstracted Driver Portability – If SW/HW Interface changes, change BH – If Application Interface changes, change TH Testability (Test HW Interface and User Space Application Interface Separately) Single Point of Access and Maintenance Enforce Multi-Thread Usage Policies Separate Buffering and ISR from Usage Common Application Entry Points Scheduled I/O (Most Work in Task Context rather than ISR Context) Sam Siewert 13 Driver Writer Resources Linux Device Drivers – Kernel Modules (Loaded After Boot) – Built-in Drivers (Loaded with Kernel Image) – Linux 2.6.x Device Drivers, by Rubini and Corbet [Web Version] – Jerry Cooperstein’s Linux Device Drivers [current] - http://www.coopj.com/, http://www.coopj.com/LDD/ – Linux Foundation [most current] VxWorks RTOS Device Interfaces – All applications and drivers are kernel modules (E.g. Btvid analog NTSC Video Decoder) – Driver entry point registration table – Section 3.9 of VxWorks Programmer’s Guide, pages 166-174 – VxWorks Programmer’s Guide, Appendix D for x86 Architecture – pages 431-478 Sam Siewert 14 Zephry for IoT Linux Foundation Real-Time IoT Nano / Micro-kernel Sam Siewert 15 What is IoT? Internet of Things (Intel paper on Zephyr / Curie- Quark) – Similar to older idea of “Sensor Networks”, but high bandwidth Internet sensors and actuators – Nodes MUST have at least local/wide area wireless interface BLE - Bluetooth Low Energy - mobile phone or tablet 802.11 - Wireless Ethernet 4G LTE or 5G Cellular Security is a major challenge Opportunity is smart devices and “digital twins” 4G LTE, 5G – E.g. Pilot walk-around with tablet for aircraft health and status – Telemetry to manufacturer for predictive maintenance – Simple configuration, provisioning, upgrades Zephyr board support (ARM NXP, Intel, …, TI TIVA?) Xbee, Zigbee, etc. BLE Sam Siewert 16 More IoT Study Digital Circuits - CEC220, CEC222 Embedded Systems - Micro class (CEC320) and lab (CEC322) Undergraduate research on IoT with ICARUS group, Dr. “A”, Dr. Zapata Rivera CEC495 - Topics class on IoT Graduate programs – CU ESE program – Coursera Online Classes - https://www.colorado.edu/ecee/msee – MIT Sloan IoT business program – Stanford IoT Certificate program – UC Irvine Embedded Systems - https://www.coursera.org/specializations/iot – U. of Washington - https://www.pce.uw.edu/certificates/embedded-and-real- time-systems-programming Sam Siewert 17 Required Software Basic Nano / Micro Kernel Network stack software – FreeRTOS – BLE network stack – Zephyr – 802.11 network stack – RTEMS – Cellular modem App interfaces – AOI, iOS L7 SCP Service SCP Client L6 Semi-structured file Semi-structured file L5 Sockets API Sockets API L4 TCP TCP L3 IP IP Router/Switch IP L2 IEEE 802.3z - GigE 802.11abgn - Wireless L1 Cat-6 UTP Cat-6 UTP Internet Sam Siewert 18.