arXiv:1111.0594v1 [cs.DB] 2 Nov 2011 iecl.LthsaeOracle microsecond are at Latches processes contem- timescale. synchronize in to used DBMS widely porary are the Spinlocks in synchronization to interprocess engines. of database again problem bring the technologies plan cores first hundreds of Rise Abstract h ffc ftuning of effect analytically the estimate to developed model mathematical A analyses strategies, counters. statistic latch spinning-blocking corresponding contemporary the and explores and It realization diagnose latches. to Oracle possibilities tune the investigates work This applications. and user tracing and OS allows both DTrace Solaris profiling of framework. emergence Tracing be- the This namic with spinlocks. user possible meth- of came direct effectiveness no measure were to user there ods recently Such and Until preemption context. context multitasking. by sys- user influenced operating are in spinlocks in work level used latches spinlocks kernels, system tems to in contrast observe In to common environments. is OLTP concurrency contention high latch contemporary The locks. codn olts Oracle latest to According Introduction 1 Spin-Blocking Time, Keywords: est hs tutrssnhoie yLcs Latches Mutexes: Locks, by KGX synchronized ac- and structures processes these Simultaneous to cess structures. result meta- cache, and millions data the of ”System consist called and is (SGA) accessed Area” memory Global shared processes This memory. thousands shared architecture” contains ”dedicated RDBMS instance Oracle OLTP Area”. Huge Global System the in protect structures to data mechanism shared serialization low-level simple, ”A rceig fMDA 01Cneec.1-4My21.Limass 2011. May 10-14 Conference. 2011 MEDIAS of Proceedings xlrn Oracle Exploring rce pnok ac,Drc,Spin DTrace, Latch, Spinlock, Oracle, SPIN

R COUNT ouetto 1 ac is latch [1] documentation

R DM pcfi spin- specific RDBMS

R http://andreynikolaev.wordpress.com value. nryNkle DE T,Russia, LTD, RDTEX Nikolaev Andrey DM ace sn Solaris using latches RDBMS TM [email protected], 0Dy- 10 hr xs w eea pnoktypes: spinlock [3] general in two found exist be There may contem- algorithms The such of evaluated. review and porary proposed mutual were of re- that field spinlock alizations Since the sophisticated Various in done algorithms. [2]. were exclusion researches in of lot Dijkstra a Edsger time, by introduced were waiting” first synchronization busy multiprocessor for of spinlocks kind of use Use a the is re- task, lock thread useful a the a such As performing of isn’t but available. becomes active lock mains check- the repeatedly until (”spins”) loop ing a in waits simply thread Wikipedia 1 Table only. mechanisms. synchronization Cache lat- these Library in compares spin- inside appeared used versions Mutexes commonly Oracle latches. est most — the Oracle of explore inside goal lock to The is work concept. realizations this spinlock Oracle spin-blocking the general are of mutexes KGX and Latches • • rqec n hrdbsutilization. are bus shared spinlocks and References) frequency Memory system Remote (or optimize operations atomic to block. cannot metrics threads Major OS Kernel spinlock. System e.I smr ffiin opl h ac o several for latch the poll to efficient more is It mu- and tex. latch Oracle like spinlocks application User Fig.1. enstesilc as spinlock the defines Oracle l yrs SN978-5-88835-032-4 ISBN Cyprus. ol,

R DM architecture RDBMS TM DTrace alc hr the where lock ”a . usec rather than pre-empt the thread doing 1 ms 1.1 Oracle R RDBMS Performance Tuning context switch. Metrics are latch acquisition CPU overview and elapsed times. During the last 30 years, Oracle developed from the first The latch is a hybrid user level spinlock. The documen- tiny one-user SQL database to the most advanced con- tation named subsequent latch acquisition phases as: temporary RDBMS engine. Each version introduced new performance and concurrency advances. The fol- lowing timeline is the excerpt from this evolution: • Atomic Immediate Get. • v. 2 (1979): the first commercial SQL RDBMS • If missed, latch spins by polling location nonatom- • ically during Spin Get. v. 3 (1983): the first database to support SMP • v. 4 (1984): read-consistency, Database Buffer Cache • In spin get not succeed, latch sleeps for Wait Get. • v. 5 (1986): Client-Server, Clustering, Distributing Database, SGA According to Anderson classification [4] the latch spin • v. 6 (1988): procedural language (PL/SQL), is one of the simplest spinlocks - TTS (”test-and-test- undo/redo, latches and-set”). • v. 7 (1992): Library Cache, Shared SQL, Stored pro- Frequently spinlocks use more complex structures than cedures, 64bit TTS. Such algorithms, like famous MCS spinlocks [5] • v. 8/8i (1999): Object types, Java, XML were designed and benchmarked to work in the con- • v. 9i (2000): Dynamic SGA, Real Application Clusters ditions of 100% latch utilization and may be heavily affected by OS preemption. For the current state of • v. 10g (2003): Enterprise Grid Computing, Self- spinlock theory see [6]. Tuning, mutexes • v. 11g (2008): Results Cache, SQL Plan Management, If user spinlocks are holding for long, for example due Exadata to OS preemption, pure spinning becomes ineffective. • To overcome this problem, after predefined number of v. 12c (2011): Cloud. Not yet released at the time of writing spin cycles latch waits (blocks) in a queue. Such spin- blocking was first introduced in [8] to achieve balance As of now, Oracle R is the most complex and widely between CPU time lost to spinning and context switch used SQL RDBMS. However, quick search finds more overhead. Optimal strategies how long to spin before then 100 books devoted to Oracle performance tuning blocking were explored in [9, 10, 11]. Robustness of spin- on Amazon [13, 14, 15]. Dozens conferences covered this blocking in contemporary environments was recently in- topic every year. Why Oracle needs such tuning? vestigated in [12] Main reason of this is complex and variable workloads. Contemporary servers having hundreds CPU cores bring Oracle is working in so different environments ranging to the first plan the problem of spinlock SMP scalabil- from huge OLTPs, petabyte OLAPs to hundreds of tiny ity. Spinlock utilization increases almost linearly with instances running on one server. Every database has its number of processors [23]. One percent spinlock utiliza- unique features, concurrency and scalability issues. tion of Dual Core development computer is negligible and may be easily overlooked. However, it may scales To provide Oracle RDBMS ability to work in such di- upto 50% on 96 cores production server and completely verse environments, it has complex internals. Last Ora- hang the 256 core machine. This phenomenon is also cle version 11.2 has 344 ”Standard” and 2665 ”Hidden” known as ”Software lockout”. tunable parameters to adjust and customize its behav- ior. Database administrator’s education is crucial to Table 1. Serialization mechanisms in Oracle adjust these parameters correctly. Locks Latches Mutexes Working at Support, I cannot underestimate the impor- Access Several Types and Operations tance of developer’s education. During design phases, Modes Modes developers need to make complicated algorithmic, phys- Acquisition FIFO SIRO (spin) SIRO ical database and schema design decisions. Design mis- + FIFO takes and ”temporary” workarounds may results in mil- SMP Atom- No Yes Yes icity lion dollars losses in production. Many ”Database Inde- Timescale Milli- Microseconds SubMicro- pendence” tricks also results in performance problems. seconds seconds Another flavor of performance problems come from self- Life cycle Dynamic Static Dynamic tuning and SQL plan instabilities, OS and Hardware issues. One need take into account also more than 10 kslgetl (get exclusive latch), the probe description will million bug reports on MyOracleSupport. It is crucial be pid16444:oracle:kslgetl:entry. Predicate and ac- to diagnose the bug correctly. tion of probe will filter, aggregate and print out the data. All the scripts used in this work are the collec- Historically latch contention issues were hard to diag- tions of such triggers. nose and resolve. Support engineers definitely need more mainstream science support. This work summa- Unlike standard tracing tools, DTrace works in Solaris rizes author investigations in this field. kernel. When oracle process entered probe function, the execution went to Solaris kernel and the DTrace To allow diagnostics of performance problems Oracle filled buffers with the data. The dtrace program instrumented his software well. Every Oracle session printed out these buffers. keeps many statistics counters. These counters describe ”what sessions have done”. There are 628 statistics in Kernel based tracing is more stable and have less over- 11.2.0.2. head then userland. DTrace sees all the system activity and can take into account the ”unaccounted for” user- Oracle Wait Interface events complements the statistics. land tracing time associated with kernel calls, schedul- This instrumentation describes ”why Oracle sessions ing, etc. have waited”. Latest 11.2.0.2 version of Oracle accounts 1142 distinct wait events. Statistics and Wait Interface DTrace allowed this work to investigate how Oracle data used by Oracle R AWR/ASH/ADDM tools, Tun- latches perform in real time: ing Advisors, MyOracleSupport diagnostics and tuning tools. More than 2000 internal ”dynamic performance” • Count the latch spins X$ tables provide additional data for diagnostics. Ora- cle performance data are visualized by Oracle Enterprise • Trace how the latch waits Manager and other specialized tools. • Measure times and distributions This is the traditional framework of Oracle performance tuning. However, it was not effective enough in spin- • Compute additional latch statistics locks troubleshooting. The following next sections describe Oracle performance tuning and database administrator’s related results. 1.2 The Tool Reader interested in mathematical estimations may pro- ceed directly to section 3 To discover how the Oracle latch works, we need the tool. Oracle Wait Interface allows us to explore the waits only. Oracle X$/V$ tables instrument the latch 2 Oracle latch instrumentation acquisition and give us performance counters. To see how latch works through time and to observe short du- It was known that the Oracle server uses kslgetl - Ker- ration events, we need something like stroboscope in nel Service Lock Management Get Latch function to ac- physics. Likely, such tool exists in Oracle SolarisTM. quire the latch. DTrace reveals other latch interface This is DTrace, Solaris 10 Dynamic Tracing framework routines: [16].

DTrace is event-driven, kernel-based instrumentation • kslgetl(laddr, wait, why, where) – get exclusive that can see and measure all OS activity. It allows latch defining the probes (triggers) to trap and write the handlers (actions) using dynamically interpreted C-like • kslg2c(l1,l2,trc,why, where) – get two excl. language. No application changes needed to use DTrace. child latches This is very similar to triggers in database technologies. • kslgetsl(laddr,wait,why,where,mode) DTrace provides more than 40000 probes in Solaris ker- – get shared latch. In Oracle 11g – nel and ability to instrument every user instruction. It ksl get shared latch() describes the triggering probe in a four-field format: provider:module:function:name. • kslg2cs(l1,l2,mode,trc,why, where)) – get two shared child latches A provider is a methodology for instrumenting the sys- tem: pid, fbt, syscall, sysinfo, vminfo ... • kslgpl(laddr,comment,why,where) – get par- ent and all childs If one need to set trigger inside the oracle process with Solaris spid 16444, to fire on entry to function • kslfre(laddr) – free the latch Fortunately Oracle gave us possibility to do the same When Oracle process waits (sleeps) for the latch, it puts using oradebug call utility. It is possible to acquire latch address into ksllawat, ”where” and ”why” val- the latch manually. This is very useful to simulate latch ues into ksllawer and ksllawhy columns of correspond- related hangs and contention. ing x$ksupr row. This is the fixed table behind the v$process view. These columns are extremely useful SQL>oradebug call kslgetl when exploring why the processes contend for the latch. This is illustrated on Figure 2. DTrace scripts also demonstrated the meaning of argu- ments:

• laddres – address of latch in SGA

• wait – flag for no-wait or wait latch acquisition

• where – integer code for location from where the latch is acquired.

• why — integer context of why the latch is acquiring Fig.2. Latch is holding by process, not session at this where.

• mode – requesting state for shared lathes. 8 - In summary, Oracle instruments the latch acquisition in SHARED mode. 16 - EXCLUSIVE mode x$ksupr fields:

”Where” and ”why” parameters are using for the in- • ksllalaq – address of latch acquiring. Populated strumentation of latch get. during immediate get (and spin before 11g) Integer ”where” value is the reason for latch acqui- • ksllawat — latch being waited for. sition. This is the index in an array of ”locations” • ksllawhy – why for the latch being waited for strings that literally describes ”where”. Oracle exter- nalizes this array to SQL in x$ksllw fixed table. These • ksllawere – where for the latch being waited for strings the database administrators are commonly see in v$latch misses and AWR/Statspack reports. • ksllalow – bit array of levels of currently holding latches Fixed view v$latch misses is based on x$kslwsc fixed table. In this table Oracle maintains an array of coun- • ksllaspn — latch this process is spinning on. Not ters for latch misses by ”where” location. populated since 8.1 ”Why” parameter is named ”Context saved from • ksllaps% — inter-process post statistics call” in dumps. It specifies why the latch is acquired at this ”where”. 2.1 The latch structure - ksllt ”Where” and ”why” parameters instrument the latch get. When the latch will be acquired, Oracle Latch structure is named ksllt in Oracle fixed tables. It saves these values into the latch structure. Oracle contains the latch location itself, ”where” and ”why” 11g externalizes latch structures in x$kslltr parent values, latch level, latch number, class, statistics, wait and x$kslltr children fixed tables for parent and list header and other attributes. child latches respectively. Versions 10g and be- Table 2.1. Latch size by Oracle version fore used x$ksllt table. Fixed views v$latch and v$latch children were created on these tables. Version Unix 32bit Unix 64bit Windows 32bit ”Where” and ”why” parameters for last latch ac- 7.3.4 92 – 120 quisition may be seen in kslltwhr and kslltwhy 8.0.6 104 – 104 columns of these tables. Fixed table x$ksuprlat shows 8.1.7 104 144 104 latches that processes are currently holding. View 9.0.1 ? 200 160 v$latchholder created on it. Again, ”where” and 9.2.0 196 240 200 ”why” parameters of latch get present in ksulawhr 10.1.0 ? 256 208 and ksulawhy columns. 10.2.0-11.2.0.2 100 160 104 Contrary to popular believe Oracle latches were signif- • Class. 0-7. Spin and wait class assigned to the icantly evolved through the last decade. Not only ad- latch. Oracle 9.2 and above. ditional statistics appeared (and disappeared) and new (shared) latch type was introduced, the latch itself was Evolution of Oracle latches is summarized in table 2.2. changed. Table 2.1 shows how the latch structure size changed by Oracle version. The ksllt size decreased in Table 2.2. Latch attributes by Oracle version 10.2 because Oracle made obsolete many latch statistics.

Oracle latch is not just a single memory location. Before Oracle Number of PAR G2C LNG UFS SHR Oracle 11g the value of first latch byte (word for shared version latches latches) was used to determine latch state: 7.3.4.0 53 14 2 3 — — 8.0.6.3 80 21 7 3 — 3 • 0x00 – latch is free. 8.1.7.4 152 48 19 4 — 9 9.2.0.8 242 79 37 — — 19 • 0xFF – exclusive latch is busy. Was 0x01 in Oracle 10.2.0.2 385 114 55 — 4 47 7. 10.2.0.3 388 117 58 — 4 48 10.2.0.4 394 117 59 — 4 50 • 0x01,0x02,etc. — shared latch holding by 1,2, 11.1.0.6 496 145 67 — 6 81 etc. processes simultaneously. 11.1.0.7 502 145 67 — 6 83 11.2.0.1 535 149 70 — 6 86 • 0x20000000 | pid — shared latch holding exclu- sively. To prevent deadlocks Oracle process can acquire latches In Oracle 11g the first exclusive latch word represents only with level higher than it currently holding. At the the Oracle pid of the latch holder: same level, the process can request the second G2C latch child X in wait mode after obtaining child Y, if • 0x00 – latch free. and only if the child number of X < child number of Y. If these rules are broken, the Oracle process raises • 0x12 – Oracle process with pid 18 holds the ex- ORA-600 errors. clusive latch. ”Rising level” rule leads to ”trees” of processes waiting for and holding the latches. Due to this rule the con- 2.2 Latch attributes tention for higher level latches frequently exacerbates contention for lower level latches. These trees can be According to Oracle Documentation and DTrace traces, seen by direct SGA access programs. each latch has at least the following flags and attributes: Each latch can be assigned to one of 8 classes with dif- ferent spin and wait policies. By default, all the latches • Name — Latch name as appeared in V$ views belong to class 0. The only exception is ”process allo- • SHR — Is the latch Shared? Shared latch is Read- cation latch”, which belongs to class 2. Latch assign- Write spinlock. ment to classes is controlled by initialization parameter LATCH CLASSES. Latch class spinning and wait- • PAR — Is the latch Solitary or Parent for the fam- ing policies can be adjusted by 8 parameters named ily of child latches? Both parent and child latches LATCH CLASS 0 to LATCH CLASS 7. share the same latch name. The parent latch can be gotten independently, but may act as a master latch when acquired in special mode in kslgpl(). 2.3 Latch Acquisition in Wait Mode

• G2C — Can two child latches be simultaneously According to contemporary Oracle 11.2 Documenta- requested in wait mode? tion, latch wait get (kslgetl(laddress,1,. . . )) proceeds • LNG — Is wait posting used for this latch? Obso- through the following phases: lete since Oracle 9.2. • One fast Immediate get, no spin. • UFS — Is the latch Ultrafast? It will not increment miss statistics when STATIS- • Spin get: check the latch upto SPIN COUNT TICS LEVEL=BASIC. 10.2 and above times. • Level. 0-14. To prevent deadlocks latches can be • Sleep on ”latch free” wait event with exponential requested only in increasing level order. backoff. • Repeat. Note the semop() operating system call. This is infinite wait until posted. This operating system call will block It occurs that such algorithm was really used ten years the process until another process posts it during latch ago in Oracle versions 7.3-8.1. For example, look at release. Oracle 8i latch get code flow using Dtrace: Therefore, in Oracle 9.2-11.2, all the latches in default kslgetl(0x200058F8,1,2,3) -KSL GET exclusive Latch class 0 rely on wait posting. Latch is sleeping without kslges(0x200058F8, ...) -wait get any timeout. This is more efficient than previous algo- skgsltst(0x200058F8) ... call repeated 2000 times rithm. Contemporary latch statistics shows that most pollsys(...,timeout=10 ms)- Sleep 1 latch waits is less then 1 ms now. In addition, spinning skgsltst(0x200058F8) ... call repeated 2000 times once reduce CPU consumption. pollsys(...,timeout=10 ms)- Sleep 2 skgsltst(0x200058F8) ... call repeated 2000 times However, this introduces a problem. If wakeup post pollsys(...,timeout=10 ms)- Sleep 3 is lost in OS, waiters will sleep infinitely. This was skgsltst(0x200058F8) ... call repeated 2000 times common problem in earlier 2.6.9 Linux kernels. Such pollsys(...,timeout=30 ms)- Sleep 4 ... losses can lead to instance hang because the process will never be woken up. Oracle solves this problem by The 2000 cycles is the value of SPIN COUNT initial- ENABLE RELIABLE LATCH WAITS parame- ization parameter. This value could be changed dynam- ter. It changes the semop() system call to semtime- ically without Oracle instance restart. dop() call with 0.3 sec timeout. Corresponding Oracle event 10046 trace [14] is: Latches assigned to non-default class wait until time- out. Number of spins and duration of sleeps for class X WAIT #0: nam=’latch free’ ela=1 p1=536893688 p2=29 p3=0 are determined by corresponding LATCH CLASS X WAIT #0: nam=’latch free’ ela=1 p1=536893688 p2=29 p3=1 parameter, which is a string of: WAIT #0: nam=’latch free’ ela=1 p1=536893688 p2=29 p3=2 WAIT #0: nam=’latch free’ ela=3 p1=536893688 p2=29 p3=2 ”Spin Yield Waittime Sleep0 Sleep1 ...Sleep7” The sleeps timeouts demonstrate the exponential back- off: Detailed description of non-default latch classes can be found in [21]. 0.01-0.01-0.01-0.03-0.03-0.07-0.07-0.15-0.23-0.39-0.39- DTrace demonstrated that by default the process spins 0.71-0.71-1.35-1.35-2.0-2.0-2.0-2.0...sec for exclusive latch for 20000 cycles. This is determined by static LATCH CLASS 0 initialization parameter. This sequence can be almost perfectly fitted by the fol- The SPIN COUNT parameter (by default 2000) is lowing formula. effectively static for exclusive latches [21]. Therefore timeout =2[(Nwait+1)/2] − 1 (1) spin count for exclusive latches can not be changed with- out instance restart. However, such sleep for predefined time was not effi- Further DTrace investigations showed that shared latch cient. Typical latch holding time is less then 10 mi- spin in Oracle 9.2-11g is governed by SPIN COUNT croseconds. Ten milliseconds sleep was too large. Most value and can be dynamically tuned. Experiments waits were for nothing, because latch already was free. demonstrated that X mode shared latch get spins by In addition, repeating sleeps resulted in many unneces- default up to 4000 cycles. S mode does not spin at sary spins, burned CPU and provokes CPU thrashing. all (or spins in unknown way). Discussion how Oracle It was not surprising that in Oracle 9.2-11g exclusive shared latch works can be found in [21]. The results are latch get was changed significantly. DTrace demon- summarized in table 2.3. strates its code flow: Table 2.3. Shared latch acquisition kslgetl(0x50006318, 1) S mode get X mode get sskgslgf(0x50006318)= 0 -Immediate latch get Held in S mode Compatible 2* SPIN COUNT kslges(0x50006318, ...) -Wait latch get Held in X mode 0 2* SPIN COUNT skgslsgts(...,0x50006318) -Spin latch get Blocking mode 0 2* SPIN COUNT sskgslspin(0x50006318)... - repeated 20000 cycles kskthbwt(0x0) kslwlmod() - set up Wait List 2.3.1 Latch Release sskgslgf(0x50006318)= 0 -Immediate latch get skgpwwait -Sleep latch get Oracle process releases the latch in kslfre(laddr). To semop(11, {17,-1,0}, 1) deal with invalidation storms [4], the process releases the latch nonatomically. Then it sets up memory barrier Since version 10.2 many previously collected latch using atomic operation on address individual to each statistics have been deprecated. We have lost important process. This requires less bus invalidation and ensures additional information about latch performance. Here I propagation of latch release to other local caches. will discuss the remaining statistics set. This is not fair policy. Latch spinners on the local CPU As was demonstrated in previous chapter, since version board have the preference. However, this is more effi- 9.2 Oracle uses completely new latch acquisition algo- cient then atomic release. Finally the process posts first rithm: process in the list of waiters. Immediate latch get Spin latch get 3 The latch contention Add the process to waiters queue Sleep until posted 3.1 Raw latch statistic counters GETS, MISSES, etc. are the integral statistics counted Latch statistics is the tool to estimate whether the latch from the startup of the instance. These values depend acquisition works efficiently or we need to tune it. Ora- on complete workload history. AWR and Statspack re- cle counts a broad range of latch related statistics. Table ports show changes of integral statistics per snapshot 3.1 contains description of v$latch statistics columns interval. Usually these values are ”averaged by hour”, from contemporary Oracle documentation [1]. which is much longer then typical latch performance Oracle collects more statistics then are usually con- spike. sumed by classic queuing models. Another problem with AWR/Statspack report is aver- Table 3.1. Latch statistics aging over child latches. By default AWR gathers only summary data from v$latch. This greatly distorts latch efficiency coefficients. The latch statistics should not be Statistic: Documentation de- When and how averaged over child latches. scription: it is changed: To avoid averaging distortions the following analy- GETS Number of times the Incremented by sis uses the latch statistics from v$latch parent and latch was requested in one after latch v$latch children (or x$ksllt in Oracle version less willing-to-wait mode acquisition then 11g) MISSES Number of times the Incremented by latch was requested in one after latch The current workload is characterized by differential willing-to-wait mode acquisition if miss latch statistics and ratios. and the requestor had occurred to wait Table 3.2 Differential (point in time) latch statistics SLEEPS Number of times a Incremented by willing-to-wait latch number of times request resulted in a process slept Description: Definition: AWR equivalent: session sleeping while during latch waiting for the latch acquisition ′′ ′′ SPIN- Willing-to-wait latch re- Incremented by ∆GETS Get Requests Arrival λ = ′′ ′′ GETS quests, which missed one after latch ∆time Snap T ime (Elapsed) the first try but suc- acquisition if miss rate ceeded while spinning but not sleep occured. Counts Gets effi- ∆MISSES ′′ ′′ only the first spin ρ = ∆GETS Pct Get Miss /100 WAIT- Elapsed time spent Incremented by ciency TIME waiting for the latch (in wait time spent Sleeps ra- ∆SLEEPS ′′ ′′ microseconds) during latch κ = ∆MISSES Avg Slps /Miss acquisition. tio IMMED- Number of times a latch Incremented by

IATE- was requested in no- one after each ′′ ′′ ∆W AIT TIME W ait T ime (s) GETS wait mode no-wait latch get Wait time W = 106 ∆time ′′Snap T ime (Elapsed)′′ IMMED- Number of times a no- Incremented by per second

IATE- wait latch request did one after unsuc- ′′ ′′ ∆SPIN GETS Spin Gets MISSES not succeed cessful no-wait Spin σ = ∆MISSES ′′Misses′′ latch get efficiency There exist several ways to choose the basic set of dif- Here I introduced the the ferential statistics. I will use the most close to AWR/ min(N ,N ) Statspack way containing ”Arrival rate”, ”Gets effi- η = CP U proc ciency”, ”Spin efficiency”, ”Sleeps ratio” and ”Wait min(NCP U ,Nproc) − 1 time per second”. Table 3.2 defines these quantities. multiplier to correct naive utilization estimation. This work analyzes only wait latch gets. The no-wait Clearly, the η multiplier confirms that the entire ap- (IMMEDIATE ...) gets add some complexity only for proach is inapplicable to single CPU machine. Really several latches. I will also assume ∆time to be small η significantly differs from one only during demonstra- enough that workload do not change significantly. tions on my Dual-Core notebook. For servers its im- Other statistics reported by AWR depend on these key pact is below precision of my estimates. For example statistics: for small 8 CPU server the η multiplier adds only 14% correction. ∆MISSES • Latch miss rate is ∆time = ρλ. We can experimentally check the accuracy of these formulas and, therefore, Poisson arrivals approxima- • Latch waits (sleeps) rate is ∆SLEEPS = κρλ. ∆time tion. U can be independently measured by sampling of v$latchholder. The latchprofx.sql script by Tanel From the queuing theory point of view, the latch is Poder [18] did this at high frequency. Within our accu- G/G/1/(SIRO+FIFO) system with interesting queue racy we can expect that ρ and U should be at least of discipline including Serve In Random Order spin and the same order. First In First Out sleep. Using the latch statistics, I can roughly estimate queuing characteristics of latch. I We know that U = λS, where S is average service (latch expect that the accuracy of such estimations is about holding) time. This allows us to estimate the latch hold- 20-30%. ing time as: ηρ As a first approximation, I will assume that incoming S = (4) λ latch requests stream is Poisson and latch holding (ser- vice) times are exponentially distributed. Therefore, our first latch model will be M/M/1/(SIRO+FIFO). This is interesting. We obtained the first estimation of latch holding time directly from statistics. In AWR Oracle measures more statistics then usually consumed terms this formula looks like by classic queuing models. It is interesting what these additional statistics can be used for. ′′Pct Get Miss′′ × ′′Snap Time′′ S = η 100 × ′′Get Requests′′ 3.2 Average service time: 3.3 Wait time: The PASTA (Poisson Arrivals See Time Averages) [20] property connects ρ ratio with the latch utilization. For Poisson streams the latch gets efficiency should be equal Look more closely on the summary wait time per to utilization: second W . Each latch acquisition increments the WAIT TIME statistics by amount of time it waited ∆misses ∆latch hold time ρ = ≈ U = (2) for the latch. According to the Little law, average latch ∆gets ∆time sleeping time is related the length of wait (sleep) queue: However, this is not exact for server with finite number of processors. The Oracle process occupies the CPU L = λwaits×haverage wait timei = λρκ×δ(Wait Time) while acquiring the latch. As a result, the latch get see the utilization induced by other NCP U − 1 processors The right hand side of this identity is exactly the ”wait only. Compare this with MVA [17] arrival theorem. In time per second” statistic. Therefore, actually: some benchmarks there may be only Nproc ≤ NCP U Oracle shadow processes that generates the latch load. W ≡ L (5) In such case we should substitute Nproc instead NCP U in the following estimate: We can experimentally confirm this conclusion be- 1 1 cause L can be independently measured by sampling ρ ≃ 1 − U = U (3) min(N ,N ) η of v$process.latchwait column.  CP U proc  3.4 Recurrent sleeps: Note that according to general queuing theory the ”Serve In Random Order” discipline of latch spin does In ideal situation, the process spins and sleeps only once. not affect average latch acquisition time. It is indepen- Consequently, the latch statistics should satisfy the fol- dent on queuing discipline. In steady state, the number lowing identity: of processes served during the passage of incoming re- quest through the system should be equal to the number MISSES = SPIN GETS + SLEEPS (6) of spinning and waiting processes.

Or, equivalently: In Oracle 11g the latch spin is no longer instrumented 1= σ + κ (7) due to a bug. The 11g spin is invisible for SQL. This do not allow us to estimate Ns and related quantities. In reality, some processes had to sleep for the latch sev- eral times. This occurred when the sleeping process was posted, but another process got the latch before the first 3.6 Comparison of results process received the CPU. The awakened process spins and sleeps again. As a results the previously equality Let me compare the results of DTrace measurements became invalid. and latch statistics. Typical demonstration results for our 2 CPU X86 server are: Before version 10.2 Oracle directly counted these se- quential waits in separate SLEEP1-SLEEP3 statis- /usr/sbin/dtrace -s latch_times.d -p 17242 0x5B7C75F8 tics. Since 10.2 these statistics became obsolete. How- ... ever, we can estimate the rate of such ”sleep misses” latch gets traced: 165180 from other basic statistics. The recurrent sleep incre- ’’Library cache latch’’, address=5b7c75f8 ments only the SLEEPS counter. The SPIN GETS Acquisition time: statistics not changed. The σ +κ−1 is the ratio of inef- value ------Distribution ------count ficient latch sleeps to misses. The ratio of ”unsuccessful 4096 | 0 8192 |@@ 7324 sleep” to ”sleeps” is given by: 16384 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 151748 σ + κ − 1 32768 |@ 4493 Recurrent sleeps ratio = (8) 65536 | 1676 κ 131072 | 988 Normally this ratio should be close to ρ. Frequent ”un- 262144 | 464 successful sleeps” are inefficient and may be a symptom 524288 | 225 of OS waits posting problems or bursty workload. 1048576 | 211 2097152 | 53 4194304 | 21 3.5 Latch acquisition time: 8388608 | 1 16777216 | 1 33554432 | 0 Average latch acquisition time is the sum of spin time and wait time. Oracle does not directly measure the spin Holding time: time. However, we can measure it on Solaris platform value ------Distribution ------count using DTrace. 8192 | 0 16384 |@@@@@@@@@@@@@@@@@@@@@@@@@ 105976 On other platforms, we should rely on statistics. 32768 |@@@@@@@@@@@@ 50877 Fortunately in Oracle 9.2-10.2 one can count the 65536 |@@ 6962 average number of spinning processes by sampling 131072 | 1986 x$ksupr.ksllalaq. The process set this column 262144 | 829 equal to address of acquired latch during active phase 524288 | 330 of latch get. Oracle 8i and before even fill the 1048576 | 205 v$process.latchspin during latch spinning. 2097152 | 34 4194304 | 6 Little law allows us to connect average number of spin- 8388608 | 0 ning processes with the spinning time: Average acquisition time =26 us Ns = λTs (9) Average holding time =37 us

As a result the average latch acquisition time is: The above histograms show latch acquisition and hold- ing time distributions in logarithmic scale. Values are −1 Ta = λ (Ns + W ) (10) in nanoseconds. Compare the above average times with the results of latch statistics analysis under the same To achieve this one need to tune the SQL opera- conditions: tors, use bind variables, change the physical schema, etc...Classic Oracle Performance books explore these Latch statistics for 0x5B7C75F8 topics [13, 14, 15]. ’’library cache’’ level#=5 child#=1 However, this tuning methodology may be too expen- Requests rate: lambda= 20812.2 Hz Miss /get: rho= 0.078 sive and even require complete application rewrite. This Est. Utilization: eta*rho= 0.156 work explores complementary possibility of changing Sampled Utilization: U= 0.143 SPIN COUNT. This commonly treated as old style Slps /Miss: kappa= 0.013 tuning, which should be avoided at any means. Increas- Wait_time/sec: W= 0.025 ing of spin count may leed to waste of CPU. However, Sampled queue length L= 0.043 nowadays the CPU power is cheap. We may already Spin_gets/miss: sigma= 0.987 have enough free resources. We need to find conditions Sampled spinnning: Ns= 0.123 when the spin count tuning may be beneficial. Derived statistics: Secondary sleeps ratio = 0.01 Processes spin for exclusive latch spin upto 20000 cy- Avg latch holding time = 7.5 us cles, for shared latch upto 4000 cycles and infinitely for sleeping time = 1.2 us mutex. Tuning may find more optimal values for your acquisition time = 7.2 us application. Oracle does not explicitly forbid spin count tuning. We can see that ηρ and W are close to sampled U and However, change of undocumented parameter should be L respectively. The holding and acquisition times from discussed with Support. both methods are of the same order. Since both meth- ods are intrusive, this is remarkable agreement. Mea- surements of latch times and distributions for demo and 4.1 Spin count adjustment production workloads conclude that: The latch holding time for the contemporary servers Spin count tuning depends on latch type. For shared should be normally in microseconds range. latches:

• Spin count can be adjusted dynamically by 4 Latch contention in Oracle 9.2- SPIN COUNT parameter. 11g • Good starting point is the multiple of default 2000 value.

Latch contention should be suspected if the latch wait • Setting SPIN COUNT parameter in ini- events are observed in Top 5 Timed Events AWR sec- tialization file, should be accompanied by tion. Look for the latches with highest W . Symptoms LATCH CLASS 0=”20000”. Otherwise of contention for the latch are highly variable. Most spin for exclusive latches will be greatly affected commonly observed include: by next instance restart.

• W > 0.1 sec/sec On the other hand if contention is for exclusive latches then: • Utilization > 10% • Spin count adjustment by LATCH CLASS 0 • Acquisition (or sleeping) time significantly greater parameter needs the instance restart. then holding time • Good starting point is the multiple of default 20000 V$latch misses fixed view and latchprofx.sql script value. by Tanel Poder [18] reveal ”where” the contention arise. • It may be preferable to increase the number of One should always take into account that contention for ”yields” for class 0 latches. a high-level latch frequently exacerbates contention for lower-level latches [13]. In order to tune spin count efficiently the root cause How treat the latch contention? During the last 15 of latch contention must be diagnosed. Obviously spin years, the latch performance tuning was focused on count tuning will only be effective if the latch holding application tuning and reducing the latch demand. time S is in its normal microseconds range. At any time the number of spinning processes should remain and probability of not releasing the latch during time t less then the number of CPUs. is Ql(τ ≥ t)=1−Pl(τ

Tk+1 + ∆ >Tk + hk Oracle allows measuring the average number of spinning The process will sleep for the latch if Tk+1+∆

Γsg = tpsg(t) dt = tpl(t) dt + ∆(1 − Pl(∆)) (16) latch miss: Tk+1

To reflect this, I will add the subscript l to all residual σ =1 − Ql(∆) distributions. In addition, I will omit subscript k for ∞ (18) the stationary state.  Γsg = htli− Ql dt  ∆ Let me denote the probability that missed process see R  latch release at time less then t as: According to classic considerations from the re- newal theory [20], the distribution of residual time P (τ

2 ing time distribution im- ht i The average residual latch holding time is htli = h2ti . pact spinning efficiency Incorporating this into previous formulas for spin effi- and CPU consumption only through the average holding ciency and CPU time results in: time hti. This allows to estimate how these quantities depend upon SPIN COUNT (∆) change. ∆ 1 σ = hti Q(t)dt If processes never release latch immediately (p(0) = 0)  0 (19) R ∆ ∞ then  1 ∆ 3  Γsg = hti dt Q(z) dz σ = hti + O(∆ ) 0 t 2 (22) Γ = ∆ − ∆ + O(∆4)  R R ( sg 2hti These nice formulas encourage us that observables ex- plored are not artifacts: For Oracle performance tuning purpose we need to know

∆ ∞ what happens if we double spin count: σ = 1 dt p(z) dz hti In low efficiency region doubling the spin count will dou-  0 t (20) R ∆ R ∞ ∞ ble ”spin efficiency” and also double the CPU consump-  1  Γsg = hti dt dz p(x) dx tion. 0 t z  R R R These estimations especially useful in the case of se- Assuming existence of second moments for latch holding vere latch contention and for the another type of Oracle time distribution we can proceed further. It is possible spinlocks — the mutex. to change the integration orders using:

∞ ∞ ∞ ∞ dz p(x)dx = zp(z)dz − t p(z)dz 5.2 Spin count tuning when efficiency is Zt Zz Zt Zt high

Utilizing this identity twice, we arrive to the following In high efficiency region, the sleep cuts off the tail of expression: latch holding time distribution: ∆ ∞ ∞ 1 2 ∆ ∆ 1 Γsg = t p(t) dt + (t − )p(t) dt σ =1 − hti (t − ∆)p(t) dt 2hti hti 2 ∆ Z0 ∆Z  2 ∞  ht i R 1 2  Γsg = 2hti − 2hti (t − ∆) p(t) dt I will focus on two regions where analytical estimations ∆ R possible. To estimate the effect of spin count tuning,  Oracle normally operates in this region of small latch we need the approximate scaling rules depending on the sleeps ratio. Here the spin count is greater than number value of ”spin efficiency” σ =”Spin gets/Miss”. of instructions protected by latch ∆ ≫ hti. From the above it is 5.1 Spin count tuning when spin efficiency clear that the spin time is low is bounded by both the ”residual latch holding time” and the spin count: The spin may be inefficient σ ≪ 1. In this low efficiency region, the (20) can be rewritten in form:

∆ 2 ∆ 1 ht i σ = − (∆ − t)p(t) dt Γsg < min( , ∆) hti hti 2hti  0 (21) 2 ∆  ∆R 1 2  Γsg = ∆ − 2hti + 2hti (∆ − t) p(t) dt Sleep prevents process from waste CPU for spinning for 0 heavy tail of latch holding time distribution  R  Normally latch holding time distribution has exponen- Using DTrace, it explored how the contemporary latch tial tail: works, its spinning-blocking strategies, corresponding Q(t) ∼ C exp(−t/τ) parameters and statistics. The mathematical model was κ =1 − σ ∼ C exp(−t/τ) 2 developed to estimate the effect of tuning the spin count. ht i Γsg ∼ − Cτ exp(−t/τ) 2hti The results are important for precise performance tun- It is easy to see that if ”sleep ratio” is small κ =1−σ ≪ ing of highly loaded Oracle OLTP databases. 1 then Doubling the spin count will square the sleep ratio coef- ficient. This will only add part of order κ to spin CPU 7 Acknowledgements consumption. I would like to paraphrase this for Oracle performance Thanks to Professor S.V. Klimenko for kindly inviting tuning purpose as: me to MEDIAS 2011 conference If ”sleep ratio” for exclusive latch is 10% than increase of spin count to 40000 may results in 10 times decrease Thanks to RDTEX CEO I.G. Kunitsky for financial of ”latch free” wait events, and only 10% increase of support. Thanks to RDTEX Technical Support Centre CPU consumption. Director S.P. Misiura for years of encouragement and support of my investigations. In other words, if the spin is already efficient, it is worth to increase the spin count. This exponential law can be compared to Guy Harrison experimental data [24]. References 5.3 Long distribution tails: CPU thrashing [1] Oracle R Database Concepts 11g Release 2 (11.2). 2010. Frequent origin of long latch holding time distribution tails is so-called CPU thrashing. The latch contention [2] E. W. Dijkstra. 1965. Solution of a prob- itself can cause CPU starvation. Processes contending lem in concurrent programming control. Com- for a latch also contend for CPU. Vise versa, lack of mun. ACM 8, 9 (September 1965), 569-. CPU power caused latch contention. DOI=10.1145/365559.365617 Once CPU starves, the operating system runqueue length increases and loadaverage exceeds the number [3] J.H. Anderson, Yong-Jik Kim, ”Shared-memory of CPUs. Some OS may shrink the time quanta un- Mutual Exclusion: Major Research Trends Since der such conditions. As a result, latch holders may not 1986”. 2003. receive enough time to release the latch. [4] T. E. Anderson. 1990. The Performance of Spin The latch acquirers preempt latch holders. The Lock Alternatives for Shared-Memory Multiproces- throughput falls because latch holders not receive CPU sors. IEEE Trans. Parallel Distrib. Syst. 1, 1 (Jan- to complete their work. However, overall CPU con- uary 1990), 6-16. DOI=10.1109/71.80120 sumption remains high. This seems to be metastable state, observed while server workload approaches 100% [5] John M. Mellor-Crummey and Michael L. Scott. CPU. The KGX mutexes are even more prone to this 1991. Algorithms for scalable synchronization transition. on shared-memory multiprocessors. ACM Trans. Due to OS preemption, residual latch holding time will Comput. Syst. 9, 1 (February 1991), 21-65. raise to the CPU scheduling scale – upto milliseconds DOI=10.1145/103727.103729 and more. Spin count tuning is useless in this case. [6] M. Herlihy and N. Shavit. 2008. The Art of Mul- Common advice to prevent CPU thrashing is to tune tiprocessor Programming. Morgan Kaufmann Pub- SQL in order to reduce CPU consumption. Fixed pri- lishers Inc., San Francisco, CA, USA. ISBN:978- ority OS scheduling classes also will be helpful. Future 0123705914. ”Chapter 07 Spin Locks and Con- works will explore this phenomenon. tention.”

[7] T. E. Anderson, D. D. Lazowska, and H. M. Levy. 6 Conclusions 1989. The performance implications of thread man- agement alternatives for shared-memory multipro- This work investigated the possibilities to diagnose and cessors. SIGMETRICS Perform. Eval. Rev. 17, 1 tune latches, the most commonly used Oracle spinlocks. (April 1989), 49-60. DOI=10.1145/75372.75378 [8] J. K. Ousterhout. Scheduling techniques for con- [19] Anna R. Karlin, Kai Li, Mark S. Manasse, and current systems. In Proc. Conf. on Dist. Comput- Susan Owicki. 1991. Empirical studies of com- ing Systems, 1982. petitve spinning for a shared-memory multiproces- sor. SIGOPS Oper. Syst. Rev. 25, 5 (September [9] Beng-Hong Lim and Anant Agarwal. 1993. Waiting 1991), 41-55. DOI=10.1145/121133.286599 algorithms for synchronization in large-scale multi- processors. ACM Trans. Comput. Syst. 11, 3 (Au- [20] L. Kleinrock, Queueing Systems, Theory, Volume gust 1993), 253-294. DOI=10.1145/152864.152869 I, ISBN 0471491101. Wiley-Interscience, 1975.

[10] Anna R. Karlin, Mark S. Manasse, Lyle A. Mc- [21] Andrey Nikolaev blog, Latch, mutex and beyond. Geoch, and Susan Owicki. 1990. Competitive ran- http://andreynikolaev.wordpress.com domized algorithms for non-uniform problems. In [22] F. Zernike. 1929. ”Weglangenparadoxon”. Hand- Proceedings of the first annual ACM-SIAM sympo- buch der Physik 4. Geiger and Scheel eds., Springer, sium on Discrete algorithms (SODA ’90). Society Berlin 1929 p. 440. for Industrial and Applied Mathematics, Philadel- phia, PA, USA, 301-309. [23] B. Sinharoy, et al. 1996. Improving Software MP Efficiency for Shared Memory Systems. Proc. of the [11] L. Boguslavsky, K. Harzallah, A. Kreinen, K. 29th Annual Hawaii International Conference on Sevcik, and A. Vainshtein. 1994. Optimal strate- System Sciences gies for spinning and blocking. J. Parallel Dis- trib. Comput. 21, 2 (May 1994), 246-254. [24] Guy Harrison. 2008. Using spin count to reduce DOI=10.1006/jpdc.1994.1056 latch contention in 11g. Yet Another Database Blog. http://guyharrison.squarespace.com/ [12] Ryan Johnson, Manos Athanassoulis, Radu Sto- ica, and Anastasia Ailamaki. 2009. A new look at the roles of spinning and blocking. In Pro- About the author ceedings of the Fifth International Workshop on Data Management on New Hardware (Da- MoN ’09). ACM, New York, NY, USA, 21-26. Andrey Nikolaev is an expert at RDTEX First Line DOI=10.1145/1565694.1565700 Oracle Support Center, Moscow. His contact email is [email protected]. [13] Steve Adams. 1999. Oracle8i Internal Services for Waits, Latches, Locks, and Memory. O’Reilly Me- dia. ISBN: 978-1565925984

[14] Millsap C., Holt J. 2003. Optimizing Oracle per- formance. O’Reilly & Associates, ISBN: 978- 0596005276.

[15] Richmond Shee, Kirtikumar Deshpande, K. Gopalakrishnan. 2004. Oracle Wait Interface: A Practical Guide to Performance Diagnostics & Tuning. McGraw-Hill Osborne Media. ISBN: 978- 0072227291

[16] Bryan M. Cantrill, Michael W. Shapiro, and Adam H. Leventhal. 2004. Dynamic instrumentation of production systems. In Proceedings of the annual conference on USENIX Annual Technical Confer- ence (ATEC ’04). USENIX Association, Berkeley, CA, USA, 2-2.

[17] Reiser, M. and S. Lavenberg, ”Mean Value Analysis of Closed Multichain Queueing Networks,” JACM 27 (1980) pp. 313-322

[18] Tanel Poder blog. Core IT for Geeks and Pros http://blog.tanelpoder.com