Achieve Fastest System Startup Sequences. How to tune an Embedded System. Embedded Systems Design Conference ARM vs. x86 July 3, 2014 Kei Thomsen MicroSys Electronics GmbH Agenda • Target: reduce startup time of embedded systems. • Meaning of Fast-Boot. • Comparison of Operating Systems, function, benefit and startup times. • Embedded Systems, boot medium and behavior. • Identifying time consuming artifacts. • Summary. ESDC, July 3, 2014 2 © MicroSys Electronics GmbH Meaning of Fast-Boot? • Sense of time depends on the Application • Is a Windows Start below 60 seconds fast? • Do I want to wait 2 seconds for my camera to start? • An endoscope must be available after a power fault within 0.8 seconds, including graphic on 2 screens: Not possible? It is possible! • Fast-Boot = System start time from reset to running application in shortest time. ESDC, July 3, 2014 3 © MicroSys Electronics GmbH Comparison of Operating Systems, function, benefit and startup times. Standard Operating System Real Time Operating System (RTOS) Linux, Windows (CE) OS-9, QNX, VxWorks … • „All Inclusive Plus“. • „Bed & Breakfast“. • extensive and complex. • simple and fast. • Many functions already included • Additional functions are added and build in. and configured. • Simple to set up, as everything • Simple to create and extend the is included. functionality volume. • Many components available, • Only functions, which are really were it is difficult to understand used / needed. No „nice to have what it is for, or needed by other features“. components. • Long startup times • Very fast startup times ESDC, July 3, 2014 4 © MicroSys Electronics GmbH Function und benefit in Linux (Embedded) • The Kernel-Start of a Linux System initializes the build-in Kernel driver. Which are they and which are not used or maybe used later? • System services are started. Which one in which order? And what are they for? Which one can I remove? • Hugh functionality. But which functions and services are really used for the correct function of the final system? • Are late optimizations and reductions of the base system still fulfilling the warrant system features? ESDC, July 3, 2014 5 © MicroSys Electronics GmbH Suggestions for Linux optimizations • Best way to start with is a minimal system and add / activate the needed function on demand. Reduction of the overall system size and number of services / processes. • Every Linux System is different and is based on a ready to use Linux-Distribution (LTIB, Debian, ELDK …). • For a start time optimization a consolidated Linux knowledge for the used distribution is needed. • First to understand the concept of the system start function: SysV-Init, BSD-Init, busybox, launchd, runit, systemd, upstart or others? Less System Services, less CPU time needed at startup. • Try different mass storage medias with different file system types. Faster loading of OS, Services and Applications. ESDC, July 3, 2014 6 © MicroSys Electronics GmbH Function and benefit in Real Time Operating Systems (RTOS) • RTOSe based on different architecture • Microware OS-9 • WindRiver VxWorks • QNX • Kernel is build of single components, which are exactly adapted and selected for the system. • MMU, Cache, Interrupt & Exception Handling • Driver • Middleware • Includes only the required functionality. No unused functions wasting space and time. ESDC, July 3, 2014 7 © MicroSys Electronics GmbH Function and benefit in Real Time Operating Systems (RTOS) • Small, clear, fast • Function of the components simple to detect and identifiable by name • Kernel, PCF, ePicIrq, SPPro1000e • One single Developer can oversee and understand the complete functional volume and its components • System understanding without magic • Additional functions and features are added and configured. • Clearly arranged, reproducible, simple to validate ESDC, July 3, 2014 8 © MicroSys Electronics GmbH Comparison of boot medium SD/µSD CF-Card NOR Flash NAND Flash Connection SD Controller IDE Controller CPU Bus NAND Controller Bus width (Bit) 1 or 4 8 8 / 16 / 32 / 64 8 Flash Type MLC (SLC very rare) MLC (SLC very rare) NOR MLC and SLC Speed (MB/sec.) >20 5 – 8 (>20 DMA) 10 - 80 5 - 8 Size (typical) 1 - 32 GB 32 MB - 32 GB 2 - 64 MB ¼ – 4 GB Price/MB Extrem low cost Very low cost Expensive Low cost OS Startup (Sec.) 1,5 2,4 0,3 2,4 Pro Simple to replace Simple to replace Extrem fast, direct Direct soldered chips execution with high capacity Contra Most time only Most time only Less capacity Defects by writing, consumer quality (MLC) consumer quality (MLC) difficult to replace Driver effort Medium Medium to high Only for writing High Recommendation Periodic replacement Periodic replacement No file system Main use as “read only” device Frustration factor Medium Medium Low High (Values of an OS-9 System with graphical application) MLC = Multi Level Cell NAND Flash ESDC, July 3, 2014 SLC = Single Level Cell NAND Flash 9 © MicroSys Electronics GmbH Embedded System boot time optimization • Compressed or uncompressed OS-Image • Is the load time longer than the decompression time? • Small OS-Image • Faster load of the OS • Driver in the OS might be faster than U-Boot driver due to IRQ and DMA • Late loading of components therefore faster • Large OS-Image • „As we are already loading, load everything in one chunk “ • No loading of components needed • As the boot mediums, boards, operating systems and components are too different, there is no generic statement possible and the optimization has to be determined individual. ESDC, July 3, 2014 10 © MicroSys Electronics GmbH Embedded System boot time optimization • Example system: • FreeScale i.MX53 1GHz • Linux (ELDK 5.4 from Denx) • Application QT Graphic • Regular Start (before): 24 seconds • Uncompressed OS Image: less 3 seconds • SD instead of NAND (OS Image and file system): less 4 seconds • Now 17 Seconds ESDC, July 3, 2014 11 © MicroSys Electronics GmbH Embedded System time consuming actions at startup • Boot medium • Ethernet – Link Auto Negotiation • 1 - 3 seconds for link detection • 5+ seconds timeout (if no cable attached) • If the kernel is waiting for completion, these are 1 - 5+ seconds, doing nothing, just waiting. • USB - Bus Explore • Each Hub 2-3 seconds to detect and start • Each Device ~1 second to detect and start • Better: start Ethernet & USB after the scheduling is available in the kernel so that the detection can run as background task during the remaining system startup. • In our example: another 6 seconds saved. ESDC, July 3, 2014 12 © MicroSys Electronics GmbH Identify time consuming artifacts • Almost every Embedded Hardware has a serial connection for the system start (U-Boot, Kernel messages, Shell console). • Most Terminal Programs like TeraTerm can do data logging with time stamps. • Hint: Use the real serial (COM1) of the PC if possible. The USB-Serial-Adapter are using a large buffer, so that multiple lines are transferred at once and getting the same time stamp. • Recommended baud rate >= 115200 ESDC, July 3, 2014 13 © MicroSys Electronics GmbH Time stamp analysis [Wed Sep 05 13:53:06.961 2012] NAND read: device 0 offset 0x1fb00000, size 0x300000 [Wed Sep 05 13:53:07.471 2012] 3145728 bytes read: OK Δt 0.5s [Wed Sep 05 13:53:07.491 2012] NAND read: device 0 offset 0x200000, size 0x400000 [Wed Sep 05 13:53:08.171 2012] 4194304 bytes read: OK Δt 0.7s [Wed Sep 05 13:53:08.211 2012] ## Booting kernel from Legacy Image at 01000000 ... [Wed Sep 05 13:53:08.211 2012] Image Name: Linux-3.3.1 [Wed Sep 05 13:53:08.211 2012] Created: 2012-06-06 6:21:20 UTC [Wed Sep 05 13:53:08.211 2012] Image Type: PowerPC Linux Kernel Image (gzip compressed) [Wed Sep 05 13:53:08.211 2012] Data Size: 3486786 Bytes = 3.3 MiB [Wed Sep 05 13:53:08.751 2012] ## Flattened Device Tree blob at 00c00000 [Wed Sep 05 13:53:08.751 2012] Booting using the fdt blob at 0xc00000 [Wed Sep 05 13:53:08.751 2012] Uncompressing Kernel Image ... OK Δt 3.1s [Wed Sep 05 13:53:11.811 2012] Loading Device Tree to 00ffa000, end 00ffffff ... OK … [Wed Sep 05 13:53:15.201 2012] ADDRCONF(NETDEV_UP): eth0: link is not ready [Wed Sep 05 13:53:16.221 2012] PHY: mdio@ffe24000:01 - Link is Up - 1000/Half Δt 1.0s [Wed Sep 05 13:53:16.221 2012] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready ESDC, July 3, 2014 14 © MicroSys Electronics GmbH Summary • Not quick nor simple to tune a system afterwards. • Many hardware understanding and operating system knowledge needed. • Appropriate Operating System selection: • Not only looking for fancy features • Commercial Real Time Operating Systems are not more expensive than a “free” Linux system. • Add components and function on demand, instead of trying to remove then afterwards. • Selection of the boot medium. • Logging with time stamp to analyze the system startup. ESDC, July 3, 2014 15 © MicroSys Electronics GmbH Question? More information in the exhibition area at MicroSys Kei Thomsen OS-9 Development, embedded System Design & Engineering Email: [email protected] Internet: www.microsys.de MicroSys Electronics GmbH Mühlweg 1 D-82054 Sauerlach ESDC, July 3, 2014 16 © MicroSys Electronics GmbH .