Solaris 10 Performance, Observability and Debugging Richard McDougall Jim Mauro Distinguished Engineer Senior Staff Engineer Sun Microsystems, Inc Sun Microsystems, Inc [email protected] [email protected] This tutorial is copyright © 2006 by Richard McDougall and James Mauro. It may not be used in whole or part for commercial purposes without the express written consent of Richard McDougall and James Mauro About The Instructors Richard McDougall, had he lived 100 years ago, would have had the hood open on the first four-stroke internal combustion powered vehicle, exploring new techniques for making improvements. He would be looking for simple ways to solve complex problems and helping pioneering owners understand how the technology worked to get the most from their new experience. these days, McDougall uses technology to satisfy his curiosity. He is a Distinguished Engineer at Sun Microsystems, specializing in operating systems technology and systems performance. Jim Mauro is a Senior Staff Engineer in Sun's Performance, Architecture and Applications Engineering group, where his most recent efforts have been Solaris performance on Opteron platforms, specifcally in the area of file system and raw disk IO performance. Jim's interests include operating systems scheduling and thread support, threaded applications, file systems and operating system tools for observability. Outside interests include reading and music; Jim proudly keeps his turntable in top working order, and still purchases and plays 12 inch vinyl LPs. Jim lives in New Jersey with his wife and two Sons. When he's not writing or working, he's handling trouble tickets generated by his family on issues they're having with home networking and getting the printer to print. Richard and Jim authored Solaris Internals: Solaris 10 and Open Solaris Kernel Architecture. Prentice Hall, 2006. ISBN 0-13-148209-2 Richard and Jim (with Brendan Gregg) authored Solaris Performance and Tools: DTrace and MDB Techniques for Solaris 10 and Open Solaris Prentice Hall, 2006. ISBN 0-13-156819-1 Richard and Jim authored Solaris Internals:Core Kernel Architecture, Prentice Hall, 2001. ISBN 0-13-022496-0 [email protected] [email protected] Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 2 Agenda • Session 1 - 9:00AM to 10:30PM > Goals, non goals and assumptions > OpenSolaris > Solaris 10 Kernel Overview > Solaris 10 Features > The Tools of the Trade • Session 2 - 11:00PM to 12:30PM > Memory > Virtual Memory > Physical Memory > Memory dynamics > Performance and Observability > Memory Resource Management Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 3 Agenda • Session 3 - 2:00PM to 3:30PM > Processes, Threads & Scheduling > Processes, Threads, Priorities & Scheduling > Performance & Observability – Load, apps & the kernel > Processor Controls and Binding > Resource Pools, Projects & Zones • Session 4 - 4:00PM to 5:30PM > File Systems and I/O > I/O Overview > The Solaris VFS/Vnode Model > UFS – The Solaris Unix File System > Performance & Observability > Network & Miscellanea Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 4 Session 1 Intro, Tools, Stuff Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 5 Goals, Non-goals & Assumptions • Goals > Architectural overview of the Solaris kernel > The tools – what they are, what they do, when and how to use them > Correlate performance & observability to key functions > Resource control & management framework • Non-goals > Detailed look at core kernel algorithms > Networking internals • Assumptions > General familiarity with the Solaris environment > General familiarity with operating systems concepts Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 6 OpenSolaris • An open source operating system providing for community collaboration and development • Source code released under the Common Development & Distribution License (CDDL – pronounced “cuddle”) • Based on “Nevada” Solaris code-base (Solaris 10+) • Core components initially, other systems will follow over time > ZFS! • Communities, discussion groups, tools, documentation, etc Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 7 Why Performance, Observability & Debugging? • Reality, what a concept > Chasing performance problems > Sometimes they are even well defined > Chasing pathological behaviour > My app should be doing X, but it's doing Y – It's only doing it sometimes > Understand utilization > Resource consumption – CPU, Memory, IO > Capacity planning > In general, attaining a good understanding of the system, the workload, and how they interact • 90% of system activity falls into one of the above categories, for a variety of roles > Admins, DBA's, Developers, etc... Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 8 Before You Begin... “Would you tell me, please, which way I ought to go from here?” asked Alice “That depends a good deal on where you want to get to” said the Cat “I don't much care where...” said Alice “Then it doesn't matter which way you go” said the Cat Lewis Carroll Alice's Adventures in Wonderland Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 9 General Methods & Approaches • Define the problem > In terms of a business metric > Something measurable • System View > Resource usage/utilization > CPU, Memory, Network, IO • Process View > Execution profile > Where's the time being spent > May lead to a thread view • Drill down depends on observations & goals > The path to root-cause has many forks > “bottlenecks” move > Moving to the next knee-in-the-curve Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 10 Amdahl's Law • In general terms, defines the expected speedup of a system when part of the system is improved • As applied to multiprocessor systems, describes the expected speedup when a unit of work is parallelized > Factors in degree of parallelization S is the speedup 1 S= 1−F F F is the fraction of the work that is serialized N N is the number of processors 1 S= 1−0.5 S = 1.6 4 processors, ½ of the work is serialized 0.5 4 1 S= 1−0.25 S = 2.3 4 processors, ¼ of the work is serialized 0.25 4 Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 11 Solaris Kernel Features • Dynamic • Multithreaded • Preemptive • Multithreaded Process Model • Multiple Scheduling Classes > Including realtime support, fixed priority and fair-share scheduling • Tightly Integrated File System & Virtual Memory • Virtual File System • 64-bit kernel > 32-bit and 64-bit application support • Resource Management • Service Management & Fault Handling • Integrated Networking Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 12 The 64-bit Revolution Solaris 2.6 Solaris 7, 8, 9, 10, ... ILP32 Apps ILP32 Apps LP64 Apps ILP32 Libs ILP32 Libs LP64 Libs ILP32 Kernel ILP32 Kernel LP64 Kernel ILP32 Drivers ILP32 Drivers LP64 Drivers 64-bit HW 32-bit HW 32-bit HW (SPARC, X64) Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 13 Solaris 8 – A Few Selected Highlights • A new 1:1 threads implementation > /usr/lib/lwp/libthread.so • Page cache enhancements (segmap) > Cyclic page cache • /dev/poll for scalable I/O • Modular debugging with mdb(1) • You want statistics? – kstat(1M), prstat(1M), lockstat(1M), busstat(1M), cpustat(1M), ... • UFS Direct I/O Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 14 Threads Model Evolution Solaris 2.0 – Solaris 8 Solaris 8, 9, 10, ... dispatcher dispatcher processors processors process User thread LWP Kernel thread Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 15 Solaris 9 A Subset of the 300+ New Features Manageability Availability Security Scalability ● Solaris Containers ● Solaris Live Upgrade 2.0 ● IPSec v4 and v6 ● IPv6 TM ● Solaris 9 ● Dynamic Reconfiguration ● SunScreen Firewall ● Thread enhancements TM Resource Manager ● Sun StorEdge Traffic ● Enhanced RBAC ● Memory optimization ● IPQoS Manager Software ● Kerberos V5 ● Advanced page coloring TM ● Mem Placement Optimization ● Solaris Volume ● IP Multipathing ● IKE ● Multi Page Size Support Manager (SVM) ● Reconfiguration ● PAM enhancements ● Hotspot JVM tuning ● Soft Disk Partitions Coordination Manager ● Secure sockets layer (SSL) TM ● NFS performance increase ● Filesystem for DBMS ● Driver Fault Injection ● Solaris Secure Shell ● UFS Direct I/O ● Framework ● Extensible password UFS Snapshots ● TM Dynamic System Domains ● ● Mobile IP encryption Solaris Flash ● TM ● TM Enhanced DNLC ● Reliable NFS ● Solaris Solaris Live Upgrade 2.0 ● RSM API ● ● TCP timers Security Toolkit TM Patch Manager ● J2SE 1.4 software with ● ● PCI and cPCI hot-swap ● TCP Wrappers Product Registry 64-bit and IPv6 ● Sun ONE DS integration ● Kernel and user-level ● NCA enhancements ● Legacy directory proxy encryption frameworks ● Secure LDAP client ● Random number generator ● Solaris WBEM Services ● SmartCard APIs . and more: ● Solaris instrumentation ● FRU ID ● Compatibility Guarantee TM ● Sun Management Center ● Java Support ● Linux Compatibility ● Network Services ● G11N and Accessibility ● GNOME Desktop Copyright © 2006 Richard McDougall & James Mauro Usenix '06 – Boston, Massachusetts 16 Solaris 10 The Headline Grabbers • Solaris Containers (Zones) • Solaris