SOLARISTM Kernel Performance, Observability & Debugging Richard McDougall James Mauro Distinguished Engineer Senior Staff Engineer Performance and Performance and Availability Engineering Availability Engineering Sun Microsystems, Inc. Sun Microsystems, Inc. [email protected] [email protected] This tutorial is copyright © 2005 by Richard McDougall and James Mauro. It may not be used in whole or part for commercial purposes without the express written consent of Richard McDougall and James Mauro About The Instructors Richard McDougall is a Distinguished Engineer in the Performance and Availability Engineering group at Sun Microsystems, where he focuses on large systems architecture, performance, measurement and observability. Richard's interests and expertise include the development of tools for measuring and Sizing Solaris systems. Among his numerous contributions, Richard designed and implemented numerous enhancements to the Solaris kernel virtual memory subsystem, and file system IO layer. Jim Mauro is a Senior Staff Engineer in the Performance and Availability Engineering group at Sun Microsystems, where he focuses on availability benchmarking and system performance tuning. Jim's past efforts include developing a framework for measuring system availability, individual availability benchmarks, improving SunCluster availability and establishing company-wide metrics for assessing system availability. Richard and Jim authored Solaris Internals: Core Kernel Architecture, Prentice Hall, 2001. ISBN 0-13-022496-0 [email protected] [email protected] Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. Credits Phil Harman: Multi-threading diagrams and Solaris introduction Bryan Cantril, Mike Shapiro, Adam Leventhal: Solaris dtrace tutorial Scott Fehrman: Virtual memory graphics Kevin Sheehan: IO Topology slides Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. Agenda – Day 1 ● Session 1 - 9:00AM to 10:30PM – Goals, non goals and assumptions – Solaris Kernel Overview & Features – Observability & Tracing Tools & Utilities ● Session 2 - 11:00PM to 12:30PM – Memory ● Virtual Memory ● Physical Memory ● Memory dynamics ● Performance and Observability ● Memory Resource Management Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. Agenda – Day 1 (cont) ● Session 3 - 2:00PM to 3:30PM – Processes, threads & scheduling ● The Solaris Multhreaded Process Model ● The Dispatcher & Scheduling Classes ● Performance & Observability ● Processor Controls and Binding ● Session 4 - 4:00PM to 5:30PM – File Systems and I/O ● I/O Overview ● The Solaris VFS/Vnode Model ● UFS – The Solaris Unix File System ● Performance & Observability Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. Agenda – Day 2 ● Session 1 - 9:00AM to 10:30PM – DTrace ● A Deeper Dive – A System View – Traps & Interrupts ● Session 2 - 11:00PM to 12:30PM – Advanced Memory Topics ● Memory monitoring and measuring ● Utilizing and tuning large memory Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. Agenda – Day 2 (cont) ● Session 3 - 2:00PM to 3:30PM – Processes, threads & scheduling ● A Deeping Dive ● The Runtime Linker ● Watching Processes with Dtrace ● Process/Thread Lab ● Session 4 - 4:00PM to 5:30PM – Disk I/O Performance – File System Performance – Network Attached Storage – File System Performance Characterization – ZFS – Resource Management – Large System Performance Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. Goals, Non-goals & Assumptions ● Goals – Architectural overview of the Solaris kernel – Drill down into key subsystems – The tools – what they are, what they do, when and how to use them – Correlate performance & observability to key functions – Resource control & management framework ● Non-goals – Detailed look at core kernel algorithms – Networking internals ● Assumptions – General familiarity with the Solaris environment – General familiarity with operating systems concepts Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. Why Performance, Observability & Debugging? ● Reality, what a concept – Chasing performance problems ● Sometimes they are even well defined – Chasing pathological behaviour ● My app should be doing X, but it's doing Y – It's only doing it sometimes – Understand utilization ● Resource consumption – CPU, Memory, IO ● Capacity planning – In general, attaining a good understanding of the system, the workload, and how they interact ● 90% of system activity falls into one of the above categories, for a variety of roles Copyright © 2005 Richard McDo–ugalAdml & James Maiurons, DBA's, Developers, etc... USENIX '05 – Anaheim, Ca. Before You Begin... “Would you tell me, please, which way I ought to go from here?” asked Alice “That depends a good deal on where you want to get to” said the Cat “I don't much care where...” said Alice “Then it doesn't matter which way you go” said the Cat Lewis Carroll Alice's Adventures in Wonderland Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. General Methods & Approaches ● Define the problem – In terms of a business metric – Something measurable ● System View – Resource usage ● CPU, Memory, Network, IO ● Process View – Execution profile ● Where's the time being spent – May lead to a thread view ● Drill down depends on observations & goals – The path to root-cause has many forks – “bottlenecks” move ● Moving to the next knee-in-the-curve Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. Solaris Kernel Features ● Dynamic ● Multithreaded ● Preemptive ● Multithreaded Process Model ● Multiple Scheduling Classes – Including realtime support ● Tightly Integrated File System & Virtual Memory ● Virtual File System ● 64-bit kernel – 32-bit and 64-bit application support ● Resource Management ● Service Management & Fault Handling ● Integrated Networking Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. The 64-bit Revolution Solaris 2.6 Solaris 7, 8, 9, 10, ... ILP32 Apps ILP32 Apps LP64 Apps ILP32 Libs ILP32 Libs LP64 Libs ILP32 Kernel ILP32 Kernel* LP64 Kernel ILP32 Drivers ILP32 Drivers* LP64 Drivers 32-bit HW 32-bit HW* 64-bit HW * Solaris 10: 64-bit kernel only on SPARC Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. Solaris 8 A Few Selected Highlights ● A new 1:1 threads implementation – /usr/lib/lwp/libthread.so ● Page cache enhancements (segmap) – Cyclic page cache ● /dev/poll for scalable I/O ● Modular debugging with mdb(1) ● You want statistics? – kstat(1M), prstat(1M), lockstat(1M), busstat(1M), cpustat(1M), ... ● UFS Direct I/O Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. The Multithreading Revolution Solaris 2.0 – Solaris 8 Solaris 8, 9, 10, ... dispatcher dispatcher processors processors process User thread LWP Kernel thread Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. Solaris 9 A Subset of the 300+ New Features Manageability Availability Security Scalability ● Solaris Containers ● Solaris Live Upgrade 2.0 ● IPSec v4 and v6 ● IPv6 TM ● Solaris 9 ● Dynamic Reconfiguration ● SunScreen Firewall ● Thread enhancements TM Resource Manager ● Sun StorEdge Traffic ● Enhanced RBAC ● Memory optimization ● IPQoS Manager Software ● Kerberos V5 ● Advanced page coloring TM ● Mem Placement Optimization ● Solaris Volume ● IP Multipathing ● IKE ● Multi Page Size Support Manager (SVM) ● Reconfiguration ● PAM enhancements ● Hotspot JVM tuning ● Soft Disk Partitions Coordination Manager ● Secure sockets layer (SSL) TM ● NFS performance increase ● Filesystem for DBMS ● Driver Fault Injection ● Solaris Secure Shell ● UFS Direct I/O ● UFS Snapshots Framework ● Extensible password ● TM Dynamic System Domains ● Solaris Flash ● Mobile IP encryption ● TM TM Enhanced DNLC ● Solaris Live Upgrade 2.0 ● Reliable NFS ● Solaris ● RSM API ● ● Patch Manager TCP timers Security Toolkit TM ● J2SE 1.4 software with ● Product Registry ● TCP Wrappers 64-bit and IPv6 ● Sun ONE DS integration ● Kernel and user-level ● NCA enhancements ● Legacy directory proxy encryption frameworks . and more: ● Secure LDAP client ● Random number generator ● Compatibility Guarantee ● Solaris WBEM Services ● SmartCard APIs ● Java Support ● Solaris instrumentation ● Linux Compatibility ● FRU ID TM ● ● Sun Management Center Network Services ● G11N and Accessibility Copyright © 2005 Richard McDougall & James Mauro ● GNOME DUSeENskIXtop'05 – Anaheim, Ca. Solaris 10 The Headline Grabbers ● Solaris Containers (Zones) ● Solaris Dynamic Tracing (dtrace) ● Predictive Self Healing – System Management Framework – Fault Management Architecture ● Process Rights Management ● Premier x86 support ● Optimized 64-bit Opteron support (x64) ● Zetabyte Filesystem (ZFS) ... and much, much more! Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. Solaris Kernel Overview System Call Interface Networking Kernel Services Virtual File System framework & Processes Clocks & Timers Framework services & Threads etc UFS NFS ProcFS . .Spe. cFS TCP/IP Scheduler ty i r e e e o v i r r ti c pr ha me ha a s s ti ResourResource cemanagemmanagementent& cont& controlrsols r d l r e a i me x e nte ti i r fa fi Memory Management Virtual Memory Kernel Memory Allocation Bus & Nexus Drivers Hardware Address Translation (HAT) sd ssd . Hardware – SPARC / x86 / x64 Copyright © 2005 Richard McDougall & James Mauro USENIX '05 – Anaheim, Ca. Introduction To Performance & Observability Tools