Front cover Linux Performance and Tuning Guidelines Operating system tuning methods Performance monitoring tools Performance analysis Eduardo Ciliendo Takechika Kunimasa ibm.com/redbooks Redpaper International Technical Support Organization Linux Performance and Tuning Guidelines July 2007 REDP-4285-00 Note: Before using this information and the product it supports, read the information in “Notices” on page vii. First Edition (July 2007) This edition applies to kernel 2.6 Linux distributions. This paper was updated on April 25, 2008. © Copyright International Business Machines Corporation 2007. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . vii Trademarks . viii Preface . ix How this paper is structured. ix The team that wrote this paper . .x Become a published author . xi Comments welcome. xii Chapter 1. Understanding the Linux operating system. 1 1.1 Linux process management . 2 1.1.1 What is a process? . 2 1.1.2 Life cycle of a process . 3 1.1.3 Thread. 4 1.1.4 Process priority and nice level . 5 1.1.5 Context switching . 5 1.1.6 Interrupt handling . 6 1.1.7 Process state . 6 1.1.8 Process memory segments. 8 1.1.9 Linux CPU scheduler . 9 1.2 Linux memory architecture . 10 1.2.1 Physical and virtual memory . 10 1.2.2 Virtual memory manager. 12 1.3 Linux file systems . 15 1.3.1 Virtual file system . 15 1.3.2 Journaling . 16 1.3.3 Ext2. 17 1.3.4 Ext3. 18 1.3.5 ReiserFS . 19 1.3.6 Journal File System . 19 1.3.7 XFS . 19 1.4 Disk I/O subsystem . 19 1.4.1 I/O subsystem architecture . 20 1.4.2 Cache . 21 1.4.3 Block layer . 23 1.4.4 I/O device driver . 24 1.4.5 RAID and storage system . 25 1.5 Network subsystem. 26 1.5.1 Networking implementation. 26 1.5.2 TCP/IP . 30 1.5.3 Offload . 33 1.5.4 Bonding module . 34 1.6 Understanding Linux performance metrics . 34 1.6.1 Processor metrics . 34 1.6.2 Memory metrics. 35 1.6.3 Network interface metrics . 36 1.6.4 Block device metrics . 36 Chapter 2. Monitoring and benchmark tools . 39 © Copyright IBM Corp. 2007. All rights reserved. iii 2.1 Introduction . 40 2.2 Overview of tool functions . 40 2.3 Monitoring tools. 41 2.3.1 top . 41 2.3.2 vmstat . 42 2.3.3 uptime . 43 2.3.4 ps and pstree . 44 2.3.5 free . 46 2.3.6 iostat . 48 2.3.7 sar . 50 2.3.8 mpstat . 51 2.3.9 numastat . 52 2.3.10 pmap . 52 2.3.11 netstat . 53 2.3.12 iptraf . 54 2.3.13 tcpdump / ethereal . 55 2.3.14 nmon . ..