Red Hat Enterprise MRG 2 Realtime Tuning Guide Advanced tuning procedures for the Realtime component of Red Hat Enterprise MRG Lana Brindley Alison Young Cheryn Tan Red Hat Enterprise MRG 2 Realtime Tuning Guide Advanced tuning procedures for the Realtime component of Red Hat Enterprise MRG Lana Brindley Red Hat Engineering Content Services Alison Young Red Hat Engineering Content Services Cheryn Tan Red Hat Engineering Content Services [email protected] Legal Notice Copyright 2013 Red Hat, Inc. The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC- BY-SA is available at . In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. Linux is the registered trademark of Linus Torvalds in the United States and other countries. Java is a registered trademark of Oracle and/or its affiliates. XFS is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries. MySQL is a registered trademark of MySQL AB in the United States, the European Union and other countries. All other trademarks are the property of their respective owners. 1801 Varsity Drive Raleigh, NC 27606-2072 USA Phone: +1 919 754 3700 Phone: 888 733 4281 Fax: +1 919 754 3701 Keywords Abstract This book contains advanced tuning procedures for the MRG Realtime component of the Red Hat Enterprise MRG distributed computing platform. For installation instructions, see the MRG Realtime Installation Guide. Red Hat Enterprise MRG 2 Realtime Tuning Guide Table of Contents .P . r.e . f.a . c. e. 5 1. Document Conventions 5 1.1. Typographic Conventions 5 1.2. Pull-quote Conventions 7 1.3. Notes and Warnings 7 2. Getting Help and Giving Feedback 8 2.1. Do You Need Help? 8 2.2. We Need Feedback! 8 .C . h. a. .p . t.e . r. .1 .. B. .e .f .o .r .e . .y .o . u. .s . t.a . r. t. t. u. .n .i n. .g . .y .o . u. r. .M . R. .G . R. .e .a . l.t .i m. .e . .s .y . s. t. e. m. 9 .C . h. a. .p . t.e . r. .2 .. G. .e . n. e. .r a. .l .S . y. s. .t e. .m . T. u. .n . i.n .g . 11 2.1. Using the Tuna interface 11 2.2. Setting persistent tuning parameters 11 2.3. Setting BIOS parameters 12 2.4. Interrupt and process binding 13 2.5. File system determinism tips 16 2.6. Using hardware clocks for system timestamping 17 2.7. Avoid running extra applications 19 2.8. Swapping and out of memory tips 20 2.9. Network determinism tips 21 2.10. syslog tuning tips 22 2.11. The PC card daemon 24 2.12. Reduce TCP performance spikes 24 2.13. Reducing the TCP delayed ack timeout 24 .C . h. a. .p . t.e . r. .3 .. .R . e. .a .l t. i.m . .e .- .S . p. e. .c .i f. i.c . T. .u . n. i. n. g. 26 3.1. Setting scheduler priorities 26 3.2. Using kdump and kexec with the MRG Realtime kernel 28 3.3. TSC timer synchronization on Opteron CPUs 33 3.4. Infiniband 34 3.5. RoCEE and High Performance Networking 34 3.6. Non-Uniform Memory Access 34 3.7. Mount debugfs 35 3.8. Using the ftrace utility for tracing latencies 35 3.9. Latency tracing using trace-cmd 38 3.10. Using sched_nr_migrate to limit SCHED_OTHER task migration. 39 .C . h. a. .p . t.e . r. .4 . .. A. .p .p . l.i c. a. .t .i o. n. T. .u . n. i.n . g. a. n. .d . D. e. p. .l o. y. .m . e. .n .t . 4 1 4.1. Signal processing in Realtime applications 41 4.2. Using sched_yield and other synchronization mechanisms 41 4.3. Mutex options 42 4.4. TCP_NODELAY and small buffer writes 44 4.5. Setting Realtime scheduler priorities 45 4.6. Loading dynamic libraries 45 4.7. Using _COARSE POSIX clocks for application timestamping 46 .C . h. a. .p . t.e . r. .5 .. M. .o . r.e . .I n. .f o. .r m. a. t. i.o . n. 4 8 5.1. Reporting Bugs 48 5.2. Further Reading 48 .E . v. e. n. .t . T. .r a. .c .i n. .g . 4 9 6 Preface .F .u . n. c. .t i. o. n. T. .r a. .c .e . r. 55 .R . e. v. i. s. i.o . n. .H . i.s . t.o . r. y. 95 7 Red Hat Enterprise MRG 2 Realtime Tuning Guide 8 Chapter 1. Before you start tuning your MRG Realtime system Preface Red Hat Enterprise MRG This book contains basic installation and tuning information for the MRG Realtime component of Red Hat Enterprise MRG. Red Hat Enterprise MRG is a high performance distributed computing platform consisting of three components: 1. Messaging — Cross platform, high performance, reliable messaging using the Advanced Message Queuing Protocol (AMQP) standard. 2. Realtime — Consistent low-latency and predictable response times for applications that require microsecond latency. 3. Grid — Distributed High Throughput (HTC) and High Performance Computing (HPC). All three components of Red Hat Enterprise MRG are designed to be used as part of the platform, but can also be used separately. MRG Realtime Many industries and organizations need extremely high performance computing and require low and predictable latency, especially in the financial and telecommunications industries. Latency, or response time, is defined as the time between an event and system response and is generally measured in microseconds (μs). For most applications running under a Linux environment, basic performance tuning can improve latency sufficiently. For those industries where latency not only needs to be low, but also accountable and predictable, Red Hat have now developed a 'drop-in' kernel replacement that provides this. MRG Realtime is distributed as part of Red Hat Enterprise MRG and provides seamless integration with Red Hat Enterprise Linux 6. MRG Realtime offers clients the opportunity to measure, configure and record latency times within their organization. About The MRG Realtime Tuning Guide This book is laid out in three main sections: General system tuning, which can be performed on a Red Hat Enterprise Linux 6 kernel and MRG Realtime specific tuning, which should be performed on a MRG Realtime kernel in addition to the standard Red Hat Enterprise Linux 6 tunes. The third section is for developing and deploying your own MRG Realtime programs. You will need to have the MRG Realtime kernel installed before you begin the tuning procedures in this book. If you have not yet installed the MRG Realtime kernel, or need help with installation issues, read the MRG Realtime Installation Guide. 1. Document Conventions This manual uses several conventions to highlight certain words and phrases and draw attention to specific pieces of information. In PDF and paper editions, this manual uses typefaces drawn from the Liberation Fonts set. The Liberation Fonts set is also used in HTML editions if the set is installed on your system. If not, alternative but equivalent typefaces are displayed. Note: Red Hat Enterprise Linux 5 and later includes the Liberation Fonts set by default. 1.1. Typographic Conventions Four typographic conventions are used to call attention to specific words and phrases. These conventions, and the circumstances they apply to, are as follows. 9 Red Hat Enterprise MRG 2 Realtime Tuning Guide Mono-spaced Bold Used to highlight system input, including shell commands, file names and paths. Also used to highlight keys and key combinations. For example: To see the contents of the file my_next_bestselling_novel in your current working directory, enter the cat my_next_bestselling_novel command at the shell prompt and press Enter to execute the command. The above includes a file name, a shell command and a key, all presented in mono-spaced bold and all distinguishable thanks to context. Key combinations can be distinguished from an individual key by the plus sign that connects each part of a key combination. For example: Press Enter to execute the command. Press Ctrl+Alt+F2 to switch to a virtual terminal. The first example highlights a particular key to press. The second example highlights a key combination: a set of three keys pressed simultaneously. If source code is discussed, class names, methods, functions, variable names and returned values mentioned within a paragraph will be presented as above, in mono-spaced bold. For example: File-related classes include filesystem for file systems, file for files, and dir for directories. Each class has its own associated set of permissions. Proportional Bold This denotes words or phrases encountered on a system, including application names; dialog box text; labeled buttons; check-box and radio button labels; menu titles and sub-menu titles. For example: Choose System → Preferences → Mouse from the main menu bar to launch Mouse Preferences. In the Buttons tab, click the Left-handed mouse check box and click Close to switch the primary mouse button from the left to the right (making the mouse suitable for use in the left hand). To insert a special character into a gedit file, choose Applications → Accessories → Character Map from the main menu bar.