ImplementingLightweight Threads D. Stein,D. Shah- SunSoft,Inc. ABSTRACT We describean implementationof a threadslibrary that providesextremely lightweight threads within a single uNrx processwhile allowing fully concurrentaccess to system resources.The threadsare lighnveightenough so that they can be createdquickly, there can be thousandspresent, and synchronizationcan be accomplishedrapidly. These goals are achievedby providinguser threads which multiplexon a pool of ketnel-supportedthreads of control. This pool is managedby the library and will automaticallygrow or shrink as requiredto ensurethat the processwill makeprogress while not usingan excessiveamount of kernel resources.The programmercan also tune the relationshipbetween threads and kernel supportedthreads of control.This paperfocuses on schedulingand synchronizinguser threads,and their interactionwith uNlx sigrralsin a multiplexingthreads library. Introduction resourcesthat may be appliedto solvingthe problem at hand. This paper describesa threadslibrary imple- mentation that provides extremely lightweight Threadsshare the processinstructions and most threadswithin a singleuNIx processwhile allowing of its data.A changein shareddata by one thread fully concurrentaccess to system resources. The can be seen by the other threadsin the process. threadsare lightweight enoughso that they can be Threads also share most of the operating system createdquickly, there can be thousandspresent, and stateof a process.For example,if one threadopens synchronizationcan be accomplishedrapidly. These a file, another thread can read it. There is no goalsare achievedby providinguser threads which system-enforcedprotection between threads. multiplex on a pool of kernel-supportedthreads of There are a variety of synchronizationfacilities control. The implementationconsists of a threads to allow threads to cooperatein accessingshared library and a kernel that has been modified to sup- data.The synchronizationfacilities include mutual port multiple threadsof control within a singleuttx exclusion(mutex) locks, condition variables,sema- process[Eykholt 1992]. phores and multiple readers, single writer The paper containsseven sections:The first (readers/writer)locks. The synchronizationvariables sectionis an overviewof the threadsmodel exported are allocatedby the applicationin ordinarymemory. by the library. The secondsection is an overviewof Threadsin different processescan also synchronize the threads library architecture.Readers familiar with eachother via synchronizationvariables placed with the SunOSMulti+hread Architecture [Powell in sharedmemory or mappedfiles, even thoughthe 19911can skip the first two sections.The third sec- threadsin different processesare generallyinvisible tion details the interfacessupplied by the kernel to to eachother. Such synchronizationvariables must support multiple threadsof control within a single be markedas being process-sharedwhen they are process.The fourth section details how the threads initialized.Synchronization variables may also have library schedulesthreads on kernel-supportedthreads different variants that can, for example,have dif- of control. The fifth section details how the library ferent blocking behavioreven though the samesyn- implementsthread synchronization.The sixth sec- chronization semantic (e.g., mutual exclusion) is tion details how the threadslibrary implementssig- maintained. nals even though threadsare not known to the ker- Each threadhas a programcounter and a stack nel. The final section briefly describesa way to to maintainof local variablesand returnaddresses. implementa debuggerthat understandsthreads. Each thread can make arbitrary system calls and interact with other processesin the usual ways. Threads Model Some operationsaffect all the threadsin a process. A traditionaluNx processhas a singlethread For example,if onethread calls exit ( ), all threads of control. In the SunOS Multi.thread Architecture aredestroyed. [Powell1991], there can be morethan one threadof Each threadhas its own signal mask.This per- control, or simply more threads, that execute mits a threadto block asynchronouslygenerated sig- independentlywithin a process. In general, the nals while it accessesstate that is also modifiedby a number or identities of threadsthat an application signal handler.Signals that are synchronouslygen- processapplies to a problem are invisible from out- erated (e.g., sIcsEGv) are sent to the thread that sidethe process.Threads can be viewedas execution causedthem. signals that are generatedexternally Summer '92 USENIX - June 8-June 12, L992- San Antonio, TX ImplementingLightweight Threads D. Stein,D. Shah are sent to one of the threadswithin a processthat independentpage faults, and may run in parallel on has it unmasked.If all threadsmask a signal, it is a multiprocessor.All the twps in the system are set pending on the processuntil a thread unmasks scheduledby the kernel onto the availableCpUs that signal. Signalscan also be sent to particular accordingto their schedulingclass and priority. threadswithin the same process. As in single- The threadslibrary schedules threads on a pool threadedprocesses, the number of sigrrals received of lwps in the process,in much the sameway by the processis less than as the or equal to the number kernel schedulesLw?s pool sent. on a of processors3. Threadsthemselves are simply data structuresand All threadswithin a processshare the set of stacks that are maintainedby the library and are signalhandlersr. If the handleris set to SIc IGN unknownto the kernel.The threadlibrary can cause any receivedsignals are discarded.If the handTeris an LwP to stop executingits currentthread and start set to SIG_DFL the defaultaction (e.g., stop, ' con- executinga new threadwithout involving the operat- tinue, exit)ãpplies to the processas a *:nble. ing system.Similarly, threads may also be crèated, A processcan fork in one of two ways: The destroyed,blocked, âctivated, etc., without operating first way clones the entire processand all of its systeminvolvement. L\ryps are relatively much more threads.The secondway only reproducesthe calling expensivethan threadssince each one usesdedicated threadin the new process.This last is usefulwhen kernel resources.If the threadslibrary dedicatedan the processis simplygoing to call exec ( ). LwP to each threadas in [Cooper1990] then many applicationssuch as databases or window systems Threads Library Architecture could not use threadsfreely (e.g. one or more per client) Threads are the programmer's interface for or they would be inefficient. Although ihe window systemmay multi+hreading.Threads are implementedby a be best expressedas a large number dynamicallylinked library thar utilizes underlying of threads,only a few of the threadsevèr needto be active(i.e., kernel-supportedthread! of control, called light- requirekernel resources, other - thanvirtual memory) weight processes(rwrsf, as shownin Figure 1. at the sameinstant. Sometimesa particularthread must be visible = to the system.For example,when a threadmust sup- { Thread =rwn Q Q=cpu port real+ime response(e.g., mousetracking thread) and thus be scheduledwith respectto all the other executionentities in the system(i.e., global schedul- ing scope).The library accommodatesthis by allow- ing a thread to be permanentlybound to an lwp. Even when a threadis bound to an Lwp. it is still a thread and can interact or synchronizewith other threadsin the process,bound or unbound. By defining Kernel both levels of interface in the architecture,we make clear the distinctionbetween what the programmersees and what the kernel pro--and Hardware vides. Most programmersprogram using threads do not think aboutrwps. When it is appropriateto Figure 1: Multi-threadArchitecture Examples optimize the behaviorof the program,the program- mer hasthe ability to tune the relationshipbenveen Each lwp can be thought of as a virtual CpU threadsand Lwps. This allows programmersto struc- which is available for executing code or system turetheir applicationassuming extremely lightweight -of calls. Each Lwp is separarely dispatched by the ker- threadswhile bringing the appropriatedegree nel, may perform independent system calls, incur kernel-supportedconcurrency to bear on the compu- tation. A threads programmercan think of rwps used by the applicationas the degreeof real its own vector con- @ave of signal currencythat the applicationrequires. handlers was rejected becauseit would add a non-t¡ivial amount of' storage to each thread and it allowed the LIVP interfaces possibility of conflicting requestsby different threads(e.g., SIG_DFr/ and a handler) that would make both Lwps are like threads.They sharemost of the implementation and use difficult. In addition, per-thread processresources. Each rwp has a private set of canbe programmedon top of a sharedvector of handle¡s, registers and a signal mask. Lwps also have if zThe^required. Lwps in this document are fundamentally different JAchrally, thal th9 lwr library in SunOS4.0. Lack of imagination the SunOS5.0 kernelschedules kemel th¡eads and a desi¡e to conform to generally acceptedterminology on top of processors.Kemel th¡eads are used for the leadus to use the sâñe name. executioncontext underlying Lwrs. See[Eykholt 1992]. '92 Summer USENIX- June8-June 12,lgg} - SanAntonio, TX D. Stein,D. Shah Implementing Lightweight Threads attributesthat are unavailableto threads. For exam- initialization.When a Lwp synchronizationprimitive ple they have a kernel-supportedscheduling class, causesthe