Multithreading the Sunos Kernel

BeyondMultiprocessing: Multithreadingthe SUnOSKernel t. 4. Fyt4olg S, R. Kleimaq S. Børton,R. Faulkner,A. Shivalingiah, M. Smith,D. Stein,J. Voll, M. Weeks,D. Williams- SunSoft,[nc. ABSTRACT Preparingthe SUnOSiSVR4kernel for today'schallenges: symmetric multiprocessing, multi-th¡eadedapplications, real-time, and multimedia,led to the incorporationof several innovativetechniques. In particular,the kernelwas re-structuredaround threads. Threads are usedfor most asynchronousprocessing, including interrupts. The resultingkernel is fully preemptibleand capableof real-timeresponse. The combinationprovides a robustbase for highly concurrent,responsive operation. Introduction having only a small data structure and a stack. Switchingbetween When we started to investigateenhancements kernelthreads does not requirea changeof virtual memory to the SunOSkernel to supportmultiprocessors, we addressspace information, so it is relatively realized that we wanted to go further than merely inexpensive.Kernel threadsare fully preemptible addinglocks to the kernel and keepingthe userpro- and may be scheduledby any of the scheduling cessmodel unchanged.It was importantfor the ker- classesin the system,including the real-time(fixed priority) nel to be capableof a high degreeof concurrencyon class. Sinceall other exe- cution entities tightly coupled symmetricmultiprocessors, but it are built using kernel th¡eads,they representa fully preemptible, was also a goal to supportmore thanone threadof real-time"nucleus" within the kernel. controlwithin a userprocess. These threads must be capableof executingsystem calls and handlingpage Kernel th¡eads use synchronizationprimitives faults independently. On multiprocessorsystems, that supportprotocols for preventingpriority inver- theseth¡eads of control must be capableof running sion, so a thread'spriority is determinedby which concurently on different processors.[Powell 1991] activitiesit is impedingby holdinglocks as well as describedthe user-visiblethread architecture. by the serviceit is performing[Khanna 1992]. We also wanted the kernel to be capableof SUnOSuses kernel tlueads to provideasynchro- bounded dispatch latency for real-time threads nous kernel activity, such as asynchronouswrites to [Khanna1992]. Real-time response requires absolute disk, servicingSTREAMS queues, and callouts. This control over scheduling, requiring preemption at removesvarious diversions in the idle loop and trap almostany point in the kernel,and eliminationof code and replaces them with independently unboundedpriority inversionswherever possible. scheduledthreads. Not only does this increase potential (these The kernel itself is a very complex multi- concurrency activities can be handledby otherCPUs), it gives th¡eadedprogram. Th¡eads can be used by user but also eachasynchro- nous activity a priority applicationsas a structuring techniqueto manage so that it can be appropri- atelyscheduled. multiple asynchronousactivities; the kernel benefrts from a th¡eadfacility thatis essentiallythe same. Even interruptsare handledby kernel threads. The kernel synchronizeswith The resultingSunOS 5.0 kernel, the central interrupt handlersvia normal thread operatingsystem component of Solaris 2.0, is fully synch¡onizationprimitives. If an interrupt preemptible,has real-timescheduling, symmetrically threadencounters a locked synchronization variable,it supports multiprocessors,and supports userJevel blocks and allows the critical section to clear. multithreading.Several of the locking strategies used in this kernel were describedin [Kleiman A major featureof the new kernelis its support 1992), In this paper we'll describesome of the of multiple kernel-supportedth¡eads of control, implementation features that make this kernel called lightleight processes(t-wts), in any userpro- unique. cess,sharing the addressspace of the processand other resources,such as open files. The kernel sup- Overview of the Kernel Architecture ports the executionof user LWPs by associatinga A kernel threadis the fundamentalentitv that is kernel threadwith eachLWp, as shown in Figure 1. scheduledand dispatchedonto one of the ôpUs of While all LWPshave a kernel thread,not all kernel the system. A kernel thread is very lightweight, th¡eadshave an LWP. '92 Summer USENIX- June8-June 12,lgg2 - SanAntonio, TX 11 BeyondMultiprocessing: Multithreading the SUnOSKernel J. R. Eykholt,... = VM addressspace [ Thread C=t*t Q="tu tl LWP data _L_YP_{ala_ r---------l thread Hardware Figure 1: Multi+hreadarchitecture examoles A user-levellibrary uses LWps to implement nl*p user-levelthreads lstein 1992]. These threadsare scheduledat user-leveland switchedbv the librarv E to any of the LrJi,rPsbelonging to the piocess. User L--_____J threads can also be bound to a particular LWp. Figure 2: MT Data Structuresfor a Process Separatinguser-level threads from the LWp allows the user thread library to quickly switch between The kernel threadstructure contains the kernel userthreads without entering the kernel. In addition, registers,scheduling class, dispatch queue links, and it allows a userprocess to havethousands of threads, pointersto the stack and the associatedLwp, pro- without overwhelmingkernel resources. cess,and CPUstructures. The threadstructure is not Data Structures swapped,so it also containssome data associated with the LWP that is neededeven when the LWp In the traditionalkemel, the user and proc structure is swapped out. Thread structures are structurescontained all kernel data for the process. linked on a list of threadsfor the process,and also Processordata was held in global variablesand data on a list of all existingthreads in the system. structures. The per-process data was divided betweennon-swappable data in the proc structure, Per-processordata is kept in the cpu structure, pointers andswappable data in the user structure.The ker- which has to the cunently executingthread, nel stack of the process,which is also swappable, the idle threadfor that CPU,and cunent dispatching was allocatedwith the user structurein the user and interrupt handlinginformation. There is a sub- area,usually one or two pageslong. structureof the cpu structurethat can be architecture dependent,but the main body is intendedto be The restructuredkemel must separatethis data applicableto mostmultiprocessing architectures. into data associatedwith each Lwp and its kernel thread,the data associatedwith each process.and To speedaccess to the thread,LWp, process, the data associatedwith each pto"".sot. Figure 2 and CPUstructures, the SPARCimplementation uses global showsthe relationshipof theseðata structuresln the a register,Vog7, to point to the currentthread restructuredkemel. structure. A C-preprocessormacro, curthread, allows accessto fields in the currentthread struc¡ure The per-processdata is contained in the proc with a singleinstruction. The currentLWp, process, structure. It containsa list of kernel threadsassoci- and CPU structuresare quickly accessiblethrough ated with the process, a pointer to the process pointers in the thread structure. In the future we addressspace, user credentials, and the list of signal may dedicateadditional global registersfor other handlers.The proc structurealso containsthe ves- frequentlyaccessed structures. tigial user structure,which is now much smaller thana page,and is no longerpractical to swap. Kernel Thread Scheduling SunOS5.0 provides The LWP structurecontains the per-LWp data severalscheduling classes. such as the process-control-block(pcU¡ for A schedulingclass determines the relativepriority of storing processeswithin user-levelprocessor registers, system call arguments, the class,and convertsthat priority to a global priority. With signal handling masks, resourceusage information, the addition of multithreading,the scheduling and profiling pointers. It also containspointers to classesand dispatcher operate on threads the associatedkemel threadand Drocessstructures. instead of processes. The schedulingclasses The kernel stack of the threadis ãilocatedwith the cunently supportedare system, timesharing, LWPstructure inside a swappablearea. andreal-time (fixed-priority). '92 t2 Summer USENIX - June 8-June 12, L992- San Antonio, TX J. R Eykholt,... BeyondMultiprocessing: Multithreading the SunOSKernel The dispatcherchooses the thread with the semaghores,and multiple readers, single writer greatestglobal priority to run on the CpU. If more (readen/writer)locks. The interfacesare shown in than one thread has the same priority, they are Figure4r. dispatchedin round-robinorder. /* Mutual excluglon tocke */ The kernel hasbeen made preemptible to better vold mutex_enter(kmutex_t *Ip) t support the real-time class and interrupt threads. void nutex_exit(kmutex_t *lp); Preemptionis disabledonly in a small number of *Ip, boundedsections of code. This meansthat a runn- void nutex_init(k¡nutex_t cha! *name, knrutex_type_t type, void *arg); able thread runs as soon as is practical after its vold . mutex_destroy ( k¡rutex_t * Ip ) i priority becomeshigh enough. For example,when int mutex_tryênter(kmutex_t *lp); thread A releasesa lock on which higher priority threadB is sleeping,the running threadA immedi- /* conditfon varfablea */ . vold cv_waltlkcondvar_t *cp, kmutex_t ately puts itself back on the run queue and allows int cv_wait_aig(kcondvar_t *cp, "lp); the CPU to run thread B. On a multiprocessor,if kmutex_t *lp) i threadA hasbetter priority than threadB, but thread int cv_timedwaít(kcondvar_t icvp, B has better priority than the current th¡ead on

Load more