DOCSLIB.ORG

Tuning Guide

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Tuning Guide Red Hat Enterprise Linux for Real Time 7 Tuning Guide Advanced tuning procedures to optimize latency in RHEL for Real Time Last Updated: 2020-10-01 Red Hat Enterprise Linux for Real Time 7 Tuning Guide Advanced tuning procedures to optimize latency in RHEL for Real Time Jaroslav Klech Red Hat Customer Content Services [email protected] Sujata Kurup Red Hat Customer Content Services [email protected] Marie Doleželová Red Hat Customer Content Services Jana Heves Red Hat Customer Content Services Maxim Svistunov Red Hat Customer Content Services Radek Bíba Red Hat Customer Content Services David Ryan Red Hat Customer Content Services Cheryn Tan Red Hat Customer Content Services Lana Brindley Red Hat Customer Content Services Alison Young Red Hat Customer Content Services Legal Notice Copyright © 2020 Red Hat, Inc. This document is licensed by Red Hat under the Creative Commons Attribution-ShareAlike 3.0 Unported License. If you distribute this document, or a modified version of it, you must provide attribution to Red Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be removed. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. Linux ® is the registered trademark of Linus Torvalds in the United States and other countries. Java ® is a registered trademark of Oracle and/or its affiliates. XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries. MySQL ® is a registered trademark of MySQL AB in the United States, the European Union and other countries. Node.js ® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project. The OpenStack ® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community. All other trademarks are the property of their respective owners. Abstract This book contains advanced tuning procedures for Red Hat Enterprise Linux for Real Time. For installation instructions, see the Red Hat Enterprise Linux for Real Time Installation Guide . Table of Contents Table of Contents .P .R . E. .F . A. .C . E. 3. .C . H. .A . P. .T .E . R. 1.. .B . E. .F . O. .R . E. Y. .O . U. S. .T . A. .R .T . .T . U. .N . I.N . .G . .Y . O. .U . .R . R. .E . D. H. .A . T. E. .N . T. .E .R . P. .R . I.S . E. L. I.N . .U . X. F. O. R. R. .E .A . L. T. .I M. E. .S . Y. .S . T. E. .M . 4 1.1. RUNNING LATENCY TESTS AND INTERPRETING THEIR RESULTS 5 .C . H. .A . P. .T .E . R. 2. G. E. N. E. R. .A . L. S. .Y . S. T. .E . M. .T .U . N. I.N . G. 9. 2.1. USING THE TUNA INTERFACE 9 2.2. SETTING PERSISTENT TUNING PARAMETERS 9 2.3. SETTING BIOS PARAMETERS 10 2.4. INTERRUPT AND PROCESS BINDING 11 2.5. FILE SYSTEM DETERMINISM TIPS 14 2.6. USING HARDWARE CLOCKS FOR SYSTEM TIMESTAMPING 15 2.7. AVOID RUNNING EXTRA APPLICATIONS 17 2.8. SWAPPING AND OUT OF MEMORY TIPS 18 2.9. NETWORK DETERMINISM TIPS 20 2.10. SYSLOG TUNING TIPS 21 2.11. THE PC CARD DAEMON 22 2.12. REDUCE TCP PERFORMANCE SPIKES 22 2.13. SYSTEM PARTITIONING 23 2.14. REDUCE CPU PERFORMANCE SPIKES 24 .C . H. .A . P. .T .E . R. 3. R. .E . A. .L . T. I. M. .E . -. S. .P . E. C. I.F .I .C . .T .U . N. I.N . G. .2 . 6. 3.1. SETTING SCHEDULER PRIORITIES 26 3.2. USING KDUMP AND KEXEC WITH THE RED HAT ENTERPRISE LINUX FOR REAL TIME KERNEL 29 3.3. TSC TIMER SYNCHRONIZATION ON OPTERON CPUS 32 3.4. INFINIBAND 33 3.5. ROCEE AND HIGH PERFORMANCE NETWORKING 33 3.6. NON-UNIFORM MEMORY ACCESS 33 3.7. REDUCING THE TCP DELAYED ACK TIMEOUT 34 3.8. USING DEBUGFS 35 3.9. USING THE FTRACE UTILITY FOR TRACING LATENCIES 35 3.10. LATENCY TRACING USING TRACE-CMD 39 3.11. USING SCHED_NR_MIGRATE TO LIMIT SCHED_OTHER TASK MIGRATION. 40 3.12. REAL TIME THROTTLING 40 3.13. ISOLATING CPUS USING TUNED-PROFILES-REALTIME 42 3.14. OFFLOADING RCU CALLBACKS 45 .C . H. .A . P. .T .E . R. 4. .A . P. .P .L . I.C . A. .T . I.O . .N . .T . U. .N . I.N . G. .A . N. .D . .D . E. .P . L. .O . Y. .M . .E .N . .T . .4 . 7. 4.1. SIGNAL PROCESSING IN REAL-TIME APPLICATIONS 47 4.2. USING SCHED_YIELD AND OTHER SYNCHRONIZATION MECHANISMS 47 4.3. MUTEX OPTIONS 48 4.4. TCP_NODELAY AND SMALL BUFFER WRITES 50 4.5. SETTING REAL-TIME SCHEDULER PRIORITIES 51 4.6. LOADING DYNAMIC LIBRARIES 51 4.7. USING _COARSE POSIX CLOCKS FOR APPLICATION TIMESTAMPING 52 4.8. ABOUT PERF 53 .C . H. .A . P. .T .E . R. 5. M. O. R. .E . I.N . .F .O . .R . M. .A . T. .I O. N. .5 . 7. 5.1. REPORTING BUGS 57 .A .P . P. .E . N. .D . I. X. A . .E .V . E. .N . T. T. .R .A . C. .I .N . G. .5 . 8. 1 Tuning Guide .A .P . P. .E . N. .D . I. X. B . D. E. T. .A . I.L . E. .D . .D . E. .S .C . R. .I .P .T . I.O . .N . .O . F. F. .T .R . A. .C . E. .5 . 9. .A .P . P. .E . N. .D . I. X. C . .R .E . V. .I S. .I O. N. H. .I S. .T . O. .R . Y. .1 .0 . 9. 2 PREFACE PREFACE This book details tuning information about Red Hat Enterprise Linux for Real Time. Many industries and organizations need extremely high performance computing and may require low and predictable latency, especially in the financial and telecommunications industries. Latency, or response time, is defined as the time between an event and system response and is generally measured in microseconds (μs). For most applications running under a Linux environment, basic performance tuning can improve latency sufficiently. For those industries where latency not only needs to be low, but also accountable and predictable, Red Hat has now developed a 'drop-in' kernel replacement that provides this. Red Hat Enterprise Linux for Real Time provides seamless integration with Red Hat Enterprise Linux 7 and offers clients the opportunity to measure, configure, and record latency times within their organization. You will need to have the Red Hat Enterprise Linux for Real Time kernel installed before you begin the tuning procedures in this book. If you have not yet installed the Red Hat Enterprise Linux for Real Time kernel, or need help with installation issues, read the Red Hat Enterprise Linux for Real Time Installation Guide. 3 Tuning Guide CHAPTER 1. BEFORE YOU START TUNING YOUR RED HAT ENTERPRISE LINUX FOR REAL TIME SYSTEM Red Hat Enterprise Linux for Real Time is designed to be used on well-tuned systems for applications with extremely high determinism requirements. Kernel system tuning offers the vast majority of the improvement in determinism. For example, in many workloads thorough system tuning improves consistency of results by around 90%. This is why we typically recommend that customers first perform the Chapter 2, General System Tuning of standard Red Hat Enterprise Linux before using Red Hat Enterprise Linux for Real Time. Things to Remember While You Are Tuning Your Red Hat Enterprise Linux for Real Time Kernel 1. Be Patient Real-time tuning is an iterative process; you will almost never be able to tweak a few variables and know that the change is the best that can be achieved. Be prepared to spend days or weeks narrowing down the set of tunings that work best for your system. Additionally, always make long test runs. Changing some tuning parameters then doing a five minute test run is not a good validation of a set of tunes. Make the length of your test runs adjustable and run them for longer than a.
Load more

Recommended publications
  • A Programmable Microkernel for Real-Time Systems∗
    A Programmable Microkernel for Real-Time Systems∗ Christoph M. Kirsch Marco A.A. Sanvido Thomas A. Henzinger University of Salzburg VMWare Inc. EPFL and UC Berkeley [email protected] tah@epfl.ch ABSTRACT Categories and Subject Descriptors We present a new software system architecture for the im- D.4.7 [Operating Systems]: Organization and Design— plementation of hard real-time applications. The core of the Real-time systems and embedded systems system is a microkernel whose reactivity (interrupt handling as in synchronous reactive programs) and proactivity (task General Terms scheduling as in traditional RTOSs) are fully programma- Languages ble. The microkernel, which we implemented on a Strong- ARM processor, consists of two interacting domain-specific Keywords virtual machines, a reactive E (Embedded) machine and a proactive S (Scheduling) machine. The microkernel code (or Real Time, Operating System, Virtual Machine microcode) that runs on the microkernel is partitioned into E and S code. E code manages the interaction of the system 1. INTRODUCTION with the physical environment: the execution of E code is In [9], we advocated the E (Embedded) machine as a triggered by environment interrupts, which signal external portable target for compiling hard real-time code, and in- events such as the arrival of a message or sensor value, and it troduced, in [11], the S (Scheduling) machine as a universal releases application tasks to the S machine. S code manages target for generating schedules according to arbitrary and the interaction of the system with the processor: the exe- possibly non-trivial strategies such as nonpreemptive and cution of S code is triggered by hardware interrupts, which multiprocessor scheduling.
    [Show full text]
  • Interrupt Handling in Linux
    Department Informatik Technical Reports / ISSN 2191-5008 Valentin Rothberg Interrupt Handling in Linux Technical Report CS-2015-07 November 2015 Please cite as: Valentin Rothberg, “Interrupt Handling in Linux,” Friedrich-Alexander-Universitat¨ Erlangen-Nurnberg,¨ Dept. of Computer Science, Technical Reports, CS-2015-07, November 2015. Friedrich-Alexander-Universitat¨ Erlangen-Nurnberg¨ Department Informatik Martensstr. 3 · 91058 Erlangen · Germany www.cs.fau.de Interrupt Handling in Linux Valentin Rothberg Distributed Systems and Operating Systems Dept. of Computer Science, University of Erlangen, Germany [email protected] November 8, 2015 An interrupt is an event that alters the sequence of instructions executed by a processor and requires immediate attention. When the processor receives an interrupt signal, it may temporarily switch control to an inter- rupt service routine (ISR) and the suspended process (i.e., the previously running program) will be resumed as soon as the interrupt is being served. The generic term interrupt is oftentimes used synonymously for two terms, interrupts and exceptions [2]. An exception is a synchronous event that occurs when the processor detects an error condition while executing an instruction. Such an error condition may be a devision by zero, a page fault, a protection violation, etc. An interrupt, on the other hand, is an asynchronous event that occurs at random times during execution of a pro- gram in response to a signal from hardware. A proper and timely handling of interrupts is critical to the performance, but also to the security of a computer system. In general, interrupts can be emitted by hardware as well as by software. Software interrupts (e.g., via the INT n instruction of the x86 instruction set architecture (ISA) [5]) are means to change the execution context of a program to a more privileged interrupt context in order to enter the kernel and, in contrast to hardware interrupts, occur synchronously to the currently running program.
    [Show full text]
  • Kernel Boot-Time Tracing
    Kernel Boot-time Tracing Linux Plumbers Conference 2019 - Tracing Track Masami Hiramatsu <[email protected]> Linaro, Ltd. Speaker Masami Hiramatsu - Working for Linaro and Linaro members - Tech Lead for a Landing team - Maintainer of Kprobes and related tracing features/tools Why Kernel Boot-time Tracing? Debug and analyze boot time errors and performance issues - Measure performance statistics of kernel boot - Analyze driver init failure - Debug boot up process - Continuously tracing from boot time etc. What We Have There are already many ftrace options on kernel command line ● Setup options (trace_options=) ● Output to printk (tp_printk) ● Enable events (trace_events=) ● Enable tracers (ftrace=) ● Filtering (ftrace_filter=,ftrace_notrace=,ftrace_graph_filter=,ftrace_graph_notrace=) ● Add kprobe events (kprobe_events=) ● And other options (alloc_snapshot, traceoff_on_warning, ...) See Documentation/admin-guide/kernel-parameters.txt Example of Kernel Cmdline Parameters In grub.conf linux /boot/vmlinuz-5.1 root=UUID=5a026bbb-6a58-4c23-9814-5b1c99b82338 ro quiet splash tp_printk trace_options=”sym-addr” trace_clock=global ftrace_dump_on_oops trace_buf_size=1M trace_event=”initcall:*,irq:*,exceptions:*” kprobe_event=”p:kprobes/myevent foofunction $arg1 $arg2;p:kprobes/myevent2 barfunction %ax” What Issues? Size limitation ● kernel cmdline size is small (< 256bytes) ● A half of the cmdline is used for normal boot Only partial features supported ● ftrace has too complex features for single command line ● per-event filters/actions, instances, histograms. Solutions? 1. Use initramfs - Too late for kernel boot time tracing 2. Expand kernel cmdline - It is not easy to write down complex tracing options on bootloader (Single line options is too simple) 3. Reuse structured boot time data (Devicetree) - Well documented, structured data -> V1 & V2 series based on this. Boot-time Trace: V1 and V2 series V1 and V2 series posted at June.
    [Show full text]
  • Additional Functions in HW-RTOS Offering the Low Interrupt Latency
    HW-RTOS Real Time OS in Hardware Additional Functions in HW-RTOS Offering the Low Interrupt Latency In this white paper, we introduce two HW-RTOS functions that offer the lowest interrupt latency available and help create a more software-friendly environment. One of these is ISR implemented in hardware, which improves responsiveness when activating a task from an interrupt and eliminates the need for developing a handler in software. The other is a function allowing the use of non-OS managed interrupt handlers in a multitasking environment. This makes it easier to migrate from a non-RTOS environment to a multitasking one. R70WP0003EJ0100 September, 2018 2 / 8 Multitasking Environment with Lowest Interrupt Latency Offered by HW-RTOS 1. Executive Summary In this white paper, we introduce two functions special to HW-RTOS that improve interrupt performance. The first is the HW ISR function. Renesas stylized the ISR (Interrupt Service Routine) process and implemented it in hardware to create their HW ISR. With this function, the task corresponding to the interrupt signal can be activated directly and in real time. And, since the ISR is implemented in the hardware, application software engineers are relieved of the burden of developing a handler. The second is called Direct Interrupt Service. This function is equivalent to allowing a non-OS managed interrupt handler to invoke an API. This function %" "$# $""%!$ $ $""%!$!" enables synchronization and communication "$ "$ between the non-OS managed interrupt handler and $($ $($ '$ '$ tasks, a benefit not available in conventional $ $ software. In other words, it allows the use of non-OS # $ % " "$) ) managed interrupt handlers in a multitasking $($ '$ environment.
    [Show full text]
  • Hiding Process Memory Via Anti-Forensic Techniques
    DIGITAL FORENSIC RESEARCH CONFERENCE Hiding Process Memory via Anti-Forensic Techniques By: Frank Block (Friedrich-Alexander Universität Erlangen-Nürnberg (FAU) and ERNW Research GmbH) and Ralph Palutke (Friedrich-Alexander Universität Erlangen-Nürnberg) From the proceedings of The Digital Forensic Research Conference DFRWS USA 2020 July 20 - 24, 2020 DFRWS is dedicated to the sharing of knowledge and ideas about digital forensics research. Ever since it organized the first open workshop devoted to digital forensics in 2001, DFRWS continues to bring academics and practitioners together in an informal environment. As a non-profit, volunteer organization, DFRWS sponsors technical working groups, annual conferences and challenges to help drive the direction of research and development. https://dfrws.org Forensic Science International: Digital Investigation 33 (2020) 301012 Contents lists available at ScienceDirect Forensic Science International: Digital Investigation journal homepage: www.elsevier.com/locate/fsidi DFRWS 2020 USA d Proceedings of the Twentieth Annual DFRWS USA Hiding Process Memory Via Anti-Forensic Techniques Ralph Palutke a, **, 1, Frank Block a, b, *, 1, Patrick Reichenberger a, Dominik Stripeika a a Friedrich-Alexander Universitat€ Erlangen-Nürnberg (FAU), Germany b ERNW Research GmbH, Heidelberg, Germany article info abstract Article history: Nowadays, security practitioners typically use memory acquisition or live forensics to detect and analyze sophisticated malware samples. Subsequently, malware authors began to incorporate anti-forensic techniques that subvert the analysis process by hiding malicious memory areas. Those techniques Keywords: typically modify characteristics, such as access permissions, or place malicious data near legitimate one, Memory subversion in order to prevent the memory from being identified by analysis tools while still remaining accessible.
    [Show full text]
  • Demarinis Kent Williams-King Di Jin Rodrigo Fonseca Vasileios P
    sysfilter: Automated System Call Filtering for Commodity Software Nicholas DeMarinis Kent Williams-King Di Jin Rodrigo Fonseca Vasileios P. Kemerlis Department of Computer Science Brown University Abstract This constant stream of additional functionality integrated Modern OSes provide a rich set of services to applications, into modern applications, i.e., feature creep, not only has primarily accessible via the system call API, to support the dire effects in terms of security and protection [1, 71], but ever growing functionality of contemporary software. How- also necessitates a rich set of OS services: applications need ever, despite the fact that applications require access to part of to interact with the OS kernel—and, primarily, they do so the system call API (to function properly), OS kernels allow via the system call (syscall) API [52]—in order to perform full and unrestricted use of the entire system call set. This not useful tasks, such as acquiring or releasing memory, spawning only violates the principle of least privilege, but also enables and terminating additional processes and execution threads, attackers to utilize extra OS services, after seizing control communicating with other programs on the same or remote of vulnerable applications, or escalate privileges further via hosts, interacting with the filesystem, and performing I/O and exploiting vulnerabilities in less-stressed kernel interfaces. process introspection. To tackle this problem, we present sysfilter: a binary Indicatively, at the time of writing, the Linux
    [Show full text]
  • Improving the Performance of Hybrid Main Memory Through System Aware Management of Heterogeneous Resources
    IMPROVING THE PERFORMANCE OF HYBRID MAIN MEMORY THROUGH SYSTEM AWARE MANAGEMENT OF HETEROGENEOUS RESOURCES by Juyoung Jung B.S. in Information Engineering, Korea University, 2000 Master in Computer Science, University of Pittsburgh, 2013 Submitted to the Graduate Faculty of the Kenneth P. Dietrich School of Arts and Sciences in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science University of Pittsburgh 2016 UNIVERSITY OF PITTSBURGH KENNETH P. DIETRICH SCHOOL OF ARTS AND SCIENCES This dissertation was presented by Juyoung Jung It was defended on December 7, 2016 and approved by Rami Melhem, Ph.D., Professor at Department of Computer Science Bruce Childers, Ph.D., Professor at Department of Computer Science Daniel Mosse, Ph.D., Professor at Department of Computer Science Jun Yang, Ph.D., Associate Professor at Electrical and Computer Engineering Dissertation Director: Rami Melhem, Ph.D., Professor at Department of Computer Science ii IMPROVING THE PERFORMANCE OF HYBRID MAIN MEMORY THROUGH SYSTEM AWARE MANAGEMENT OF HETEROGENEOUS RESOURCES Juyoung Jung, PhD University of Pittsburgh, 2016 Modern computer systems feature memory hierarchies which typically include DRAM as the main memory and HDD as the secondary storage. DRAM and HDD have been extensively used for the past several decades because of their high performance and low cost per bit at their level of hierarchy. Unfortunately, DRAM is facing serious scaling and power consumption problems, while HDD has suffered from stagnant performance improvement and poor energy efficiency. After all, computer system architects have an implicit consensus that there is no hope to improve future system’s performance and power consumption unless something fundamentally changes.
    [Show full text]
  • Review Der Linux Kernel Sourcen Von 4.9 Auf 4.10
    Review der Linux Kernel Sourcen von 4.9 auf 4.10 Reviewed by: Tested by: stecan stecan Period of Review: Period of Test: From: Thursday, 11 January 2018 07:26:18 o'clock +01: From: Thursday, 11 January 2018 07:26:18 o'clock +01: To: Thursday, 11 January 2018 07:44:27 o'clock +01: To: Thursday, 11 January 2018 07:44:27 o'clock +01: Report automatically generated with: LxrDifferenceTable, V0.9.2.548 Provided by: Certified by: Approved by: Account: stecan Name / Department: Date: Friday, 4 May 2018 13:43:07 o'clock CEST Signature: Review_4.10_0_to_1000.pdf Page 1 of 793 May 04, 2018 Review der Linux Kernel Sourcen von 4.9 auf 4.10 Line Link NR. Descriptions 1 .mailmap#0140 Repo: 9ebf73b275f0 Stephen Tue Jan 10 16:57:57 2017 -0800 Description: mailmap: add codeaurora.org names for nameless email commits ----------- Some codeaurora.org emails have crept in but the names don't exist for them. Add the names for the emails so git can match everyone up. Link: http://lkml.kernel.org/r/[email protected] 2 .mailmap#0154 3 .mailmap#0160 4 CREDITS#2481 Repo: 0c59d28121b9 Arnaldo Mon Feb 13 14:15:44 2017 -0300 Description: MAINTAINERS: Remove old e-mail address ----------- The ghostprotocols.net domain is not working, remove it from CREDITS and MAINTAINERS, and change the status to "Odd fixes", and since I haven't been maintaining those, remove my address from there. CREDITS: Remove outdated address information ----------- This address hasn't been accurate for several years now.
    [Show full text]
  • Thread Scheduling in Multi-Core Operating Systems Redha Gouicem
    Thread Scheduling in Multi-core Operating Systems Redha Gouicem To cite this version: Redha Gouicem. Thread Scheduling in Multi-core Operating Systems. Computer Science [cs]. Sor- bonne Université, 2020. English. tel-02977242 HAL Id: tel-02977242 https://hal.archives-ouvertes.fr/tel-02977242 Submitted on 24 Oct 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Ph.D thesis in Computer Science Thread Scheduling in Multi-core Operating Systems How to Understand, Improve and Fix your Scheduler Redha GOUICEM Sorbonne Université Laboratoire d’Informatique de Paris 6 Inria Whisper Team PH.D.DEFENSE: 23 October 2020, Paris, France JURYMEMBERS: Mr. Pascal Felber, Full Professor, Université de Neuchâtel Reviewer Mr. Vivien Quéma, Full Professor, Grenoble INP (ENSIMAG) Reviewer Mr. Rachid Guerraoui, Full Professor, École Polytechnique Fédérale de Lausanne Examiner Ms. Karine Heydemann, Associate Professor, Sorbonne Université Examiner Mr. Etienne Rivière, Full Professor, University of Louvain Examiner Mr. Gilles Muller, Senior Research Scientist, Inria Advisor Mr. Julien Sopena, Associate Professor, Sorbonne Université Advisor ABSTRACT In this thesis, we address the problem of schedulers for multi-core architectures from several perspectives: design (simplicity and correct- ness), performance improvement and the development of application- specific schedulers.
    [Show full text]
  • Embedded Linux Conference Europe 2019
    Embedded Linux Conference Europe 2019 Linux kernel debugging: going beyond printk messages Embedded Labworks By Sergio Prado. São Paulo, October 2019 ® Copyright Embedded Labworks 2004-2019. All rights reserved. Embedded Labworks ABOUT THIS DOCUMENT ✗ This document is available under Creative Commons BY- SA 4.0. https://creativecommons.org/licenses/by-sa/4.0/ ✗ The source code of this document is available at: https://e-labworks.com/talks/elce2019 Embedded Labworks $ WHOAMI ✗ Embedded software developer for more than 20 years. ✗ Principal Engineer of Embedded Labworks, a company specialized in the development of software projects and BSPs for embedded systems. https://e-labworks.com/en/ ✗ Active in the embedded systems community in Brazil, creator of the website Embarcados and blogger (Portuguese language). https://sergioprado.org ✗ Contributor of several open source projects, including Buildroot, Yocto Project and the Linux kernel. Embedded Labworks THIS TALK IS NOT ABOUT... ✗ printk and all related functions and features (pr_ and dev_ family of functions, dynamic debug, etc). ✗ Static analysis tools and fuzzing (sparse, smatch, coccinelle, coverity, trinity, syzkaller, syzbot, etc). ✗ User space debugging. ✗ This is also not a tutorial! We will talk about a lot of tools and techniches and have fun with some demos! Embedded Labworks DEBUGGING STEP-BY-STEP 1. Understand the problem. 2. Reproduce the problem. 3. Identify the source of the problem. 4. Fix the problem. 5. Fixed? If so, celebrate! If not, go back to step 1. Embedded Labworks TYPES OF PROBLEMS ✗ We can consider as the top 5 types of problems in software: ✗ Crash. ✗ Lockup. ✗ Logic/implementation error. ✗ Resource leak. ✗ Performance.
    [Show full text]
  • Thread Evolution Kit for Optimizing Thread Operations on CE/Iot Devices
    Thread Evolution Kit for Optimizing Thread Operations on CE/IoT Devices Geunsik Lim , Student Member, IEEE, Donghyun Kang , and Young Ik Eom Abstract—Most modern operating systems have adopted the the threads running on CE/IoT devices often unintentionally one-to-one thread model to support fast execution of threads spend a significant amount of time in taking the CPU resource in both multi-core and single-core systems. This thread model, and the frequency of context switch rapidly increases due to which maps the kernel-space and user-space threads in a one- to-one manner, supports quick thread creation and termination the limited system resources, degrading the performance of in high-performance server environments. However, the perfor- the system significantly. In addition, since CE/IoT devices mance of time-critical threads is degraded when multiple threads usually have limited memory space, they may suffer from the are being run in low-end CE devices with limited system re- segmentation fault [16] problem incurred by memory shortages sources. When a CE device runs many threads to support diverse as the number of threads increases and they remain running application functionalities, low-level hardware specifications often lead to significant resource contention among the threads trying for a long time. to obtain system resources. As a result, the operating system Some engineers have attempted to address the challenges encounters challenges, such as excessive thread context switching of IoT environments such as smart homes by using better overhead, execution delay of time-critical threads, and a lack of hardware specifications for CE/IoT devices [3], [17]–[21].
    [Show full text]
  • Walking the Linux Kernel
    Walking the Linux Kernel Stanislav Kozina Associate Manager April 2016 AGENDA Linux Kernel in general Debugging kernel issues without crash Prepare the system for crashing Crash it with systemtap See what we can get from the crash dump 2 Linux Kernel in general – boring ● Just software… ● Written in C & asm ● List of expected features ● Boot and initialization process ● Memory and process management ● Hardware abstraction ● Files, directories, sockets, … ● Resources abstraction ● CPU, memory, … ● POSIX 3 Linux Kernel in general – BUT! ● Quite big (20mil LOC) ● No libc (many other standards functions instead) ● Special environment ● Preemptive, shared memory space ● Early boot code is tricky ● No dynamic allocator, AP, scheduler, even locks! ● Special security requirements ● Kernel should not just die and/or leak anything 4 Linux Kernel in general – development system ● Open source ● List of maintainers in MAINTAINERS ● Patches posted via email ● Documentation/SubmittingPatches ● LKML ● Usually companies care about support of their stuff ● Hardware vendors… ● “We don't break userland” 5 DIGGING IN Observing kernel is not trivial ● Hard to get a consistent picture ● If we stop it, how we observe it? ● Using Vms? ● printk() ● Statistics, /proc, perf, strace, ftrace, ... 7 8 Debugging kernel issues ● Oops messages ● Message buffer, registers, stack/backtrace ● Oops leaves the system running, but unstable! ● Current task is killed ● Printk() ● Systemtap, ftrace ● crash ● Sysrq triggers 9 Kernel oops [[ 32.580355]32.580355] SysRqSysRq :: TriggerTrigger aa crashcrash [[ 32.581331]32.581331] BUG:BUG: unableunable toto handlehandle kernelkernel NULLNULL pointerpointer dereferencedereference atat [[ 32.582703]32.582703] IP:IP: [<ffffffff813b9716>][<ffffffff813b9716>] sysrq_handle_crash+0x16/0x20sysrq_handle_crash+0x16/0x20 [[ 32.583781]32.583781] PGDPGD 3b5030673b503067 PUDPUD 3b5020673b502067 PMDPMD 00 [[ 32.584609]32.584609] Oops:Oops: 00020002 [#1][#1] SMPSMP [[ 32.585210]32.585210] ModulesModules linkedlinked in:in: ip6t_rpfilterip6t_rpfilter (...(..
    [Show full text]