STREAMS Overview STREAMS is a exible communication sub- system framework { Originally develop ed by Dennis Ritchie for Research UNIX Network Programming UNIX STREAMS provides a uniform infrastruc- Overview of STREAMS ture for developing and con guring character- based I/O { e.g., networks, terminals, lo cal IPC Douglas C. Schmidt STREAMS supp orts the addition and re- moval of pro cessing comp onents at installation- time or run-time { Via user-level orkernel-level commands 1 2 STREAMS Overview cont'd STREAMS Bene ts The STREAMS paradigm is data-driven, not demand-driven STREAMS provides an integrated environ- ment for developing kernel-resident network- { i.e., asychronous in the kernel, yet synchronous at the application ing services STREAMS promotes de nition of standard Supp orts b oth immediate and deferred pro- service interfaces cessing { e.g., TPI and DLPI Internally, data are transferred by passing p ointers to messages STREAMS supp orts dynamic \service sub- stitution" controlled by user-level commands { Goal is to reduce memory-to-memory copying over- head 4 3 A Simple Stream Bene ts cont'd STREAMS APPLICATION PROCESS fd = open ("/dev/dev0"); USER Message-based interfaces enable o -b oard KERNEL proto col migration WRITE STREAM READ QUEUE HEAD QUEUE Permits layered and de-layered multiplexi ng More recent implementations take advan- WRITE- SIDE READ- SIDE of paralleli sm in the op erating system tage ( DOWNSTREAM) ( UPSTREAM) and hardware "dev0" WRITE STREAM READ QUEUE DRIVER QUEUE 6 5 A Mo dule on a Stream The Stream Head A \stream head" exp orts a uniform service to other layers of the UNIX kernel APPLICATION PROCESS interface ioctl (fd, PROCESSI_PUSH, "modx"); { Including the application \layer" running in user- USER space KERNEL General stream head services include WRITE STREAM READ QUEUE HEAD QUEUE 1. Queueing a Provides a synchronous interface to asychronous STREAM devices "modx" WRITE READ QUEUE MODULE QUEUE 2. Datagram- and stream-oriented data transfer 3. Segmentation and reassembly of messages "dev0" WRITE STREAM READ Event propagation QUEUE DRIVER QUEUE 4. { i.e., signals 8 7 The Stream Head cont'd The Stream Head cont'd Stream head op erations include PROCESS { stropen USER KERNEL . called from le system layer to op en a Stream sd_pollist strclean stdata_t { . called from le system layer to remove event sd_wrq sd_siglist cells from Stream Head when a le is closed STREAM strevent strclose queue_t queue_t { Head strwsrv() strrput() strevent . called from the le system layer to dismantle a Stream q_next q_next strevent Intermediate strread STREAM Modules { and STREAM called from the le system layer to retrieve data Driver . messages coming upstream 10 9 Stream Head op erations cont'd Messages { strwrite In STREAMS, all information is exchanged . called from the le system layer to send data via messages messages downstream { i.e., b oth data and control messages of various priorities { strioctl . called from the le system layer to p erform con- trol op erations A multi-comp onent message structure is used to reduce the overhead of { strgetmsg 1. Memory-to-memory copying . called from the system call layer to get a proto col or data message coming upstream { i.e., via \reference counting" 2. Encapsulation/de-encapsulation { strputmsg { i.e., via \comp osite messages" . called from the system call layer to send a pro- to col or data message downstream { strpoll Messages may b e queued at STREAM mo d- . called from the le system layer to check if p ol- ules lable events are satis ed 11 12 Comp osite Message Message Structure mblk_t mblk_t b_band b_next b_cont b_prev b_cont b_rptr b_datap b_datap b_wptr b_datap db_ref DYNAMICALLY ALLOCATED db_base db_base db_type DATA mblk_t db_lim BUFFER db_base dblk_t dblk_t queue_t dblk_t DATA DATA BUFFER BUFFER 13 14 Message Bu er Sharing Message Typ es mblk_t STREAMS M DATA user data M PROTO proto col information M PASSFP pass le p ointer IOCTL user io ctl request mblk_t M BREAK request line break b_datap M b_rptr M SIG signal pro cess group b_wptr dblk_t M DELAY request transmit delay M CTL mo dule-sp eci c control message SETOPTS set Stream head options b_datap M RSE reserved for RSE use b_rptr M db_base b_wptr db_ref = 2 Normal priority messages { M DATA, M PROTO, M PASSFP, and M IOCTL may b e generated from user-level Typically subject to ow control DATA { BUFFER 16 15 STREAMS Message Typ es Queue Structure cont'd M PCPROTO proto col information M FLUSH ush queues IOCACK acknowledge io ctl request M qband IOCNAK fail io ctl request M qb_hiwat qb_last qband COPYIN request to copyin io ctl data M qb_lowat qb_first module_stat qb_count qb_next M COPYOUT request to copyout io ctl data M IOCDATA reply to M COPYIN and M COPYOUT WRITE READ PCSIG signal pro cess group M STREAM QUEUE QUEUE q_qinfo q_qinfo HEAD READ read noti cation M queue_t queue_t qinit M HANGUP line disconnect q_next q_next open() ERROR fatal error M close() WRITE READ put() STOP stop output immediately M STREAM QUEUE QUEUE q_qinfo DRIVER service() START restart output M q_qinfo queue_t queue_t STOPI stop input immediately M module_info STARTI restart input M q_link q_maxpsz qband qband PCRSE reserved for RSE use M q_flag q_minpsz q_hiwat q_last q_lowat q_ptr q_count q_first High priority messages *Typically not ow controlled *M PCPROTO may b e generated from user-level * Others passed b etween STREAM comp onents 17 18 Queue Structure cont'd Queue and Message Linkage t queue queue_t qband { Primary data structure BAND 1 BAND 2 . Each mo dule contains a write queue and a read queue q_nband q_band qb_next qb_next { Stores information on ow control, scheduling, max/min interface sizes, linked messages, private data q_first qb_first qb_first q_last qb_last qb_last qinit { Contains put, service, open, close routines sub NORMAL ( BAND 0) MESSAGES PRIORITY ( BAND 1) qband b_next MESSAGES b_prev PRIORITY Contains information on each additional message { b_next ( BAND 2) MESSAGES > 0 and < 256 band b_prev HIGH b_next PRIORITY b_prev MESSAGES module info { Stores default ow control information 19 20 Queue Subroutines cont'd Queue Subroutines Standard subroutines cont'd Four standard subroutines are asso ciated with queues in a mo dule or driver, e.g., { putqueue t *q, mblk t *mp { openqueue t *q, dev t *devp, int o ag, int s ag, cred t *cred p; . Performs immediate pro cessing . Called when Stream is rst op ened and on any . Supp orts synchronous communciation services subsequent op ens i.e., further queue pro cessing is blo cked until . Passed a p ointer to the new read queue put returns . Also called any time a mo dule is \pushed" onto { servicequeue t *q the Stream . Performs deferred pro cessing { closedev t dev, int ag, int otyp, cred t *cred p; . Supp orts asynchronous communication services . Called when last reference to a Stream is closed Uses the message queue available in a queue . Also called when a mo dule is \p opp ed" o the Stream . Runs as a \weightless" pro cess::: 21 22 Flow Control and the service Queue Flow Control Pro cedure Typical non-multiplexed example WRITE() SLEEPS UNTIL STREAM IS APPLICATION " BACK- ENABLED" PROCESS USER service queue t *q KERNEL int TYPICALLY NO QUEUEING ON WRITE f STREAM HEAD STREAM QUEUE HEAD t *mp; WRITE QUEUE mblk q_next while mp = getq q != 0 WRITE queclass mp == QPCTL jj STREAM if QUEUE MODULE PRESSURE - canputnext q f BACK q_flag == QFULL /* Pro cess message */ q_next putnext q, mp; WRITE STREAM g QUEUE DRIVER else f putb q q, mp; return 0; g g put and service work together to supp ort ad- Flow control is more complex with multi- visory ow control plexers and concurrency 24 23 Flow Control and the canput Pro cedure Flow Control and the put canputnext is used by put and service Pro cedure routines to test advisory ow control condi- tions Typical put example int put queue t *q, mblk t *mp e.g., f if queclass mp == QPCTL jj int canputnext queue t *q canputnext q f f /* Pro cess message */ nd closest queue with a service pro cedure putnext q, mp; if queue is full f else set ag for \back-enabling" putq q, mp; return 0; /* Enables service routine */ g return 0; return 1; g g Note that non-MP systems may use canput:: : 25 26 getq putq The mblk t *getqqueue t * function dequeues a message from a queue The int putqqueue t *, mblk t * function enqueues a message on a queue { It is typically called by a queue's service pro- cedure { It is typically called by a queue's put pro cedure when it can no longer pro ceed::: Messages are dequeued in priority order putq automatically handles { i.e., higher priority messages are stored rst in the queue! 1. priority-band allo cation 2. priority-band message insertion getq handles 3. ow control 1. Flow control 2. Back-enabling Enqueueing a high priority message auto- matically schedules the queue's service pro cedure to run at some p oint getq returns 0 when there are no available { Di ers on MP vs. non-MP system messages 28 27 Multiplexin g STREAMS provides rudimentary supp ort for Multiplexor Links b efore multiplexing via multiplexor drivers { Unlike mo dules, multiplexors o ccupy a le-system no de that can b e \op ened" fd1 = open ("/dev/ip"); APPLICATION fd2 = open ("/dev/eth"); . Rather than \pushed" USER KERNEL STREAM WRITEWRITE READREAD WRITEWRITE READREAD STREAM HEAD QUEUEQUEUE QUEUEQUEUE QUEUEQUEUE QUEUEQUEUE HEAD Multiplexors may contain one ormore upp er and/orlower connections to other STREAM and/or multiplexors mo dules STREAM WRITEWRITE READREAD WRITEWRITE READREAD STREAM QUEUE QUEUE QUEUE QUEUE MULTIPLEXOR QUEUE QUEUE QUEUE QUEUE DRIVER "/dev/ip" "/dev/eth" { Enables supp ort forlayered network proto col suites .