IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. COM-28, NO. 2, FEBRUARY 1980 27 1 preference for throughput over delay. This parameter can be stochastic fluctuationof the system load. Numerical results are presented considered as a further control tool to limit the throughput, and it is shown that the throughput-delay performance of a network can be thedelay, the average and/orthe maximum number of improved by proper selection of the design parmeters, such as the window tokens, and to limit the destination buffer utilization (Table size, the timeout period, etc. I?) ; The numerical examples indicate that the performance of INTRODUCTION a network when token limits are selected according to the A computer network may be thought of as a collection of suboptimalpolicy determined by our heuristic solution is resources to beused by a competing population of users. very close to the performance of an optimal policy (Fig. 3). Networkresources i~cludebuffers, transmission bandwidth, Thisfigure further indicates that the performance of a processortime, name space, table entries, logical channels, periodicdecision, is very close to the optimal as long as etc.;the user population includes any source of datawhich the decision period is chosen properly. This property is of requires transmission through the network. The collection of significant importance in practical application, as a periodic resourceshas a limited capacity which causes conflicts to decision reduces the overhead due to the signaling of control occuramong the users of thesystem. These conflicts may packets. resultin a degradation of systemperformance to the point that the system becomes clogged and the throughput goes to REFERENCES .~ zero [ 11.This behavior is typical of “contention”systems L. Kleinrock and p. Kermani,“Static flow control in store-and-forward in which the throughput will increasewith the applied load computer networks,” IEEE Trans. Cornmun., this issue,pp. 271-279. P.Kermani, “Switching and flow control techniques in computer up to some optimum value, beyond which, due to unpredict- communicationnetworks,” Comput. Sci. Dep., Univ. Calif., Los ablebehavior by users and servers and additional user-user Angeles,UCLA-ENG-7802, Feb. 1978 ($so publishedas Ph.D. and user-server interactionand overhead, more load causes dissertation, Dec. 1977). areduction in throughput [2] .I Networkscannot afford to [31 A. Giessler, J. Haenle, A. Koenig, andE. Pade, “Packet networks with accept all thetraffic that isoffered without control; there deadlock-freebuffer allocation, an investigation by simulation,” Compur. Neworks, vol. 2, pp. 191-208, July 1978. mustbe rules which govern the acceptance of traffic from [41 L. Kleinrock, “Power and deterministicrules of thumb for probabilistic outsideand coordinate the flow inside the network. These problemsin computer communications,” in Proc. Inr. Cortf: on rules are commonly known as fZow.contro2 procedures. More Cotnmun., Boston, MA, June 1979, pp. 43.I. 143.I. 10. preciselyflow control is the set of mechanismswhereby a R. Howard, DynamicProgrumming and Markov Processes. Cambridge,MA: MIT, 1960. flow of data can be maintained within limits ‘compatible with B. L. Miller,“Dispatching from depot repair in a recoverableitem the amount of availableresources [ 31 . ’ inventory system: On the optimality of a heuristic rule,” Munagemenr In order to keep the network traffic within desirable limits, Sci., vol. 21, pp. 316325, Nov. 1974. flow control procedures, among other things, must be equipped A. F. Veinott, Jr.,’ “Thestatus of mathematicalinventory theory,” Munugernent Sci., vol. 2, pp. 745-777, July 1966. quippedwith throttling mechanisms. These mechanisms D. Blackwell, “Discrete dynamic programming,” Ann. Math. Sruris., include: credit (or tokens), which give permission for message VOI. 33, pp. 719-726, 1962. flow; a rate at which a given flow may proceed; a stop-andgo M. J. Sobel, “Optimal operation of queues,” in Marhemaricul Methods procedure which turns a flow on and off according to some in Queueing. Berlin: Springer-Verlag, 1974, pp. 231-261. criteria;the introduction of delay, so asto slowdown the flow,etc. [3]. A windowmechanism is an example of the credit scheme. Existing control methods in store-and-forward communica- tionnets can be classifiedas eitherlocal control or global control.Local control is appliedby a communication pro- Static Flow Control in Store-and-Forward ComputerNetworks cessor within the subnet OD the basis of its own as well as its immediateneighbors’ traffic data and resource utilization. Due tosome limitations [4], [ 51,local control is not,by LEONARD KLEINROCK, FELLOW, IEEE, AND PARVIZ KERMANI itself,sufficient to prevent congestion, and global control is necessary in order to further stop the input to the network Abstract-In this paper we develop an analytic model for end-to-end well before the network is loaded to saturation.’%his control communication protocols and studythe window mechanism for flow canbe accomplished by limiting the number of packets control in store-and-forward (inparticular message-switching) computer- simultaneouslycontained in the network. Examples of the based communication networks.We develop a static flow control model in existingmethods ~f globalflow control are: isarithmic flow which the parameters of the system are not dynamically adjusted to the control[4] studied for the NPL network;and end-toend flowcontrol ([6] andthe references therein] ) used inthe ARPANET,where, basically, the total number of credits Paper approved by the Editor for Computer Communication of the IEEE Communications Society for publication without oral presenta- between two users are limited. tion. Manuscript received June 13, 1978; revised June 27, 1979. This Most end-tolend flow control mechanisms use a variant of work was supported by the Advanced Research Projects Agency of the thecredit throttling technique and are usually described in Department of Defense under Contract MDA 903-77-CO272. terms of awindow mechanism [7], wherethe unacknowl- P. Kermani was with the Deqartment of Computer Science, Univer- edged messages (or packets) are limited to lie within a sliding sityof California, Los Angeles, CA 90024. Heis now with the IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598. window. L. Kleinrock is with the Department of Computer Science, Univer- End-to-endflow control is accomplished through inter- sity of California, Los Angeles, CA 90024. process communication protocols and any attempt to quanti-
0090-6778/80/0200-0271$00.75 @ 1980 IEEE tatiVelY studythe former should start with the development A ? ofan analytic model for the latter. Due to thecomplex DESTINATION CONTROLLER multivariate environment of these distributed dynamic control procedures,to date little work has been done in evaluating NETWORK the performance of flow' control procedures in terms of their 1 efficiencyand freedom from deadlock and degradation. In I I I I NETWORK NETWORK view of this, the recent work in the field has been extremely BOUNDARY BOUNDARY welcome [ 81 -[ 151. The purpose of this paper is to develop an analytic model Fig. 1. Structure of a network. forend-to-end communication protocols and to study the tokenmechanism forflow control store-and-forwardin computer based communication networks (we will be mainly concernedwith message- and packet-switching systems). We first develop a static flow control model in which the param- eters of the system are not dynamically adjusted to the sto- chastic fluctuations of the system load. We then present nu- merical results to show the effect of different parameters on the overall performance of the system. We use the static 'flow controlresults in a subsequent paper to develop a dynamic flow control system. Theanalysis carried out in thispaper is basedon many simplifyingassumptions. We considerthe calculations as an initial step towards a better understanding of more complex and more elaborate systems. It is believed that such a study is necessary even if it is based on idealized assumptions. RECEIVER (SUB) & 11. THE MODEL AND THE ASSUMPTIONS In this section we develop an analytic model for a network inwhich messages are flow cpntrolled with a token mecha- nism. We begin by describing the structure of the network and itscomponents. We thenelaborate on the assumptions in- volved in our analysis, and finally we present the analysis.
A. Structure of' The Network regardingthe implications encountered in theexact analysis With a credit mechanism using tokens, the total number of the interested reader is referred to [ 131 ). messagesin the network between a source-destination pair A detailed specification of the structure of the subnetwork is restrictedto a maximum value. We modelthe network as is not the object of our study; we view the subnetwork as a in Fig. 1, in whichthe traffic controller (TC) is responsible source of randomdelay which we choose torepresent by a forkeeping the number of unacknowledgedmessages below properdistribution function (which will bediscussed in w, the limit on the number of tokens (we occasionally refer Section 11-B below). to w as the token limit). Fig.2(b) shows the structure of the -destination node. The structure of the TC is further elaborated in Fig. 2(a). Be'cause ofour requirement that the TC must receive an We maythink of a circulating pool of w "tokens,"each of ACK forthe last (re)transmitted message, tGe. destination which permits exactly one message to be accepted to the net- node may receive multiple copies of the same message. Clearly, work and to be sent toward the destination node; a copy of onlythe first received copy should be. delivered tothe re- each message is retained in the buffer space (labeled timeout ceivinguser and the later copies' must be discarded. The boxin Fig. 2(a)) provided by the TC. If an ACK forthis "duplicateremover" inthe destination node (Fig. 2(b)) message is receivedwithin T seconds(the timeout period). checks all the received messages and discards all the previously The TC discards its stored copy of the message and the token accepted ones, but sends an ACK for these duplicate messages is returnedback to the pool. This token may then be used backto the sending TC; only the first copy of a message is toaccept a new message from the source. Thus, we allow passed to the buffer storage. If there is space available at the up to w outstandingmessages between a source-destination buffer, the message is admitted and is put in the queue to be pair. If, however, the ACK isnot received within T seconds, delivered to the 'receiving user and an ACK is sent back to the themessage is retransmitted and no token is released. In TC. If, however, there is no buffer to store the message, it'is our anaylsis we required that the ACK for the last (re)trans- rejected,no ACK is sentback, and later copies will beac- mitted message be receive$%o that the TC can accept a new cepted. message to network. This"means, gnce an ACK is not received withinthe timeout period, the'TC assumes the transmission B. Assunlptions has failed and any ACK received after the timeout period is Measurementson existing networks, in particular on the ignored.The reason for this simplifying assumption is that ARPANET [ 161, showthat the average number of packets modeling of theexact operation is extremelydifficult and per message is very close to one(1.2 packets per message in the theanalysis is unnecessarilycomplicated (for adiscussion ARPANET); hence. we make the simplifying assumption that IEEE TRANSACTIONS ON COMMIINICATIONS. VOL. COM-28. NO. 2. I‘tUI
FT;(f)= 1 - e-(Yl-he) - (yl - h,)re-(Yl-he>t C. The Arlui.vsis We startour analysis by derivingthe ACK delaydistribu- where 7,. is arandom variable representing the round-trip- tion. Based onassumption 4 above,the distribution of the delay, X, is the(sub)network traffic rate and y1 = pCl, Cl acknowledgment delay is being the network channel capacity. 5) Thedestination node has a finite number of buffers; FTu(r) = (I - PI)[1 - - Are- (1) each buffer can store one (and only one) message. 6) Negligible transmission error rate. and its density function will be 7) Atthe destination node only the first successful copy of amessage is accepted;however, ACK’s forlater copies f- (r) = (1 - P,)A2re-* aresent back. After the removal of the duplicate messages, tU (2) thecounting process of message arrivals tothe destination where buffer is also assumed to be.Poisson. A = 71 - X,. 1 8) Nomessage sequencing. (3 Some Rrmurks: 1) The first three assumptions are made to In the above equations F,, is a random variable representing the substantiateassumption number 4. Withthese three assump- acknowledgment delay and PI is the probability that an ACK is tionsthe (sub)network can essentialy bemodeled asa notreceived. We note that Tu is a “degenerate” random vari- Markoviannetwork of queues. Simulation and measurement able. studies of existing networks [ 181, [ 161 and [ 191 show that a multistage Erlangian distribution is a good approximation for theround-trip delay distribution. Ideally, an Erlangian dis- 1 It should be emphasized that the first 4 assumptions (including as- tribution of a degree larger than 10 is the best choice; how- sumption 7) lose their consistency when w is small and/or the (subhet- ever,any degree larger than 2 makesthe derivations and work is lightly loaded. In fact it can be easily shown that for w = 1 and theintegrations extremely complicated. Considering the fact when Pl = 0 the arrival process to the (sub)network becomes Erlangian of the second degree. Only when a high traffic is offered from TC to that the protocol performance is relativelyinsensitive to the (sub)network the above assumptions become more consistent. The fact degree of the distribution([ 1 11, also our analysis with degrees= that we study the system under heavy traffic justifies these assumptions 1 and2), we choose to usean Erlangian distribution of the to some degroe. 274 ll~~l To calculate the retransmission probability we note that a message must be retransmitted when either the message is lost at the destination buffer, or the transmission is successful but theacknowledgment delay is greaterthan the timeout (7); therefore, we have The unknowns in the above equation are A,* and PI,the probability that an ACK is not returned. By virtue of assump- P, = Pr [retransmission] tion 7, the destination buffer behaves like an MIMIlIB queue- ing system where B is the buffer size at the destination node. = Pl + (1 - Pf)Pr [Fa > T I no loss] To calculatethe loss probability PI, we notethat at equi- libriumthe input rate of messages tothe network is or (A*) equalto the output rate of messagesfrom it. Hence id*, Y, = Pl-Pl)(l + (1 + A)e-A7 (4) the total arrival rate of messages to the buffer (the accepted messages plus the rejected ones),will be The effective network traffic, A,, is related to the input rate x* of messages to the network by the following: Ad* = - 1 -P, 1 = ---A, and by (5) we have 1 -P, 1 -P, We now consider the length of the time a message occupies Ad* = -he * a buffer in the TC. Using a renewal theory argument 1201 we 1 -- P, have [ 131 : and so we have [ 21 ] where To, = E [time of occupancy] where y2 = pC2 and C2 is the capacity of the output chan- nel. The above equation can be written as and TI = E[Fa 17, < TI. The analytic results presented so far depend upon the solution Calculation of TI is fairly easy. It can be shown that [ 131 of the two equations (9) and (1 1). For a given value of the parameters (7,M!, yl,yz, and B) the above system of equations 2 Ar2e-A7 TI =-- - can be solved through an iterative procedure [221. A 1 -[ 1 + AT]~-"~ Having foundthe network traffic he*, wecan use (5) to find the maximum input rate (or the maximum throughput) Finally,we find the valueof To, as follows: ofthe network. The total end-to-end delay consists of two components:the network delay and the destination buffer delay. In calculating the network delay we should notice that 2 Pf + (1 - P,)e-AT To, = - + (6) only the delay of the first successful message is of importance A (1 -Pl){ 1 - [ 1 + AT]^-/'^}' to us; further retransmissions only affect the network traffic (hknce have an indirect effect on the delay). Using a renewal Whenthe flow is cotrtrolledby the token mechanism theoryargument [20], the (maximum) averagenetwork describedin Section 11-A, w messagesare accepted to the delay can be found to be [ 131 networkon the average every To, seconds;therefore, the (maximum) input rate will be 1 Pl T* = +- 7. (12) W y1 - x,* 1 - P, A* = - (7) TO,* Theother component of the total deiay is thedestination where To,* is the occupancy time when the flow controlled bufferdelay. This is equivalent tothe system delay of the throughput is' maximum (with w tokens) and isgiven by (6) corresponding M/M/1 /B queue [ 2 1 ] (when X = A*). From (5) we have 1 W x,* = --, (8) 1 -pr TO,* where pd* = (ha*/pC,) is the utilization factor of the destina- We canuse the values of P, and To,* from (4) and (6) in tion buffer. Finally we have (8) to get Tee* = T* Tdst* (14) wA + x,* = 2(1-P,)[1-(1 fA~)e-~']+[P,+(l-P&-AT]A~ where Tcs6,*is the maximum end-to-end delay. This completes our analysis of the flow control mechanism. 1.0 As noted, the analysis does not provide us with a closed-form solution for the network throughput and delay; we can only solve the model through numerical techniques. In [ 131, based on some simpler and less realistic assumptions, other models for flow control have been developed for which closed-form solutions have been derived. These models, though very sim- plistic,have been used to develop a network algebra for a systematicstudy of series and parallel interconnections of networks and also to study the problem of the optimal alloca tionof tokens and channel capacity among anumber of parallel connected networks between a source and destinarion. The interested reader is referred to[ 131, [ 151. The analysis of this section has’been based on the equilib- 1 0.0 rium of the system for a selected set of parameters (7,w, y, 1.0 10.0 1w.o 1000.0 C1, Cp, B) whichare not dynamically adjusted to the sto- NORMALIZEDTIMEOUT IrlZll chastic fluctuations in the load of the system. We refer to the Fig. 3. Dependency of network throughput on timeout (B = 10, mechanism analyzed above as “static flow control.” C1IC2 = 1 ,Oh 111. NUMERICAL RESULTS Inthis section we present numerical examples for static IimT+- X* = 0. This fact is easily shown by using (7) .and (8). flowcontrol, and study the effect of different parameters Fig. 3 shows that this throughput degradation is more severe (h*,7, and E) on the throughput-delay performance of a net- for larger w. The reason for this degradation is that for a large work with y, C1,and C2 given. For a given set of parameters timeout the TC waits a long time for an ACK to come back; wecan find the maximum throughput (A*) andthe corre- however, because of the finite destination buffer size, there is sponding(therefore maximum) total average delay (T,,*). a nonzero probability that an ACK never comes back, hence Whenever there is no ambiguity, we will refer to these quan- the buffer of the traffic controller becomes full of unacknowl- titiessimply by throughput and delay. Throughout our pre- edgedmessages, and nofurther external traffic is accepted. sentation we consider the normalized values of certain quan- Clearly,this degradation is notpresent if thedestination tities;for measures of time, the normalization constant is buffersize is infinite;the throughput curve in Fig. 3 for = ( l/pCl), theaverage transmission time of amessage w = 10 andthe destination buffer size of infinity explicitly on a networkchannel. It also turns out that only the ratio showsthis fact. Fig. 3 shows that for a fixed w, there is a C1/C2 is of importance to us, and it is that which we use in value of timeout that optimizes (maximizes) the throughput. this section. We will designatethis optimal value by T~*(as opposed to We begin by studying the effect of the timeout period on 7d*, which is theoptimal value of timeout with respect to the network throughput. Fig. 3 shows the normalized through- theend-to-end delay; see below). This figure shows that for put (A*/yCl) as a function of the normalized timeout (7/X1) asmall w, thethroughput curve, after an initial increase, fordifferent w’s whenthe destination buffer size B = 10 remainsquite flat, and only for a large w is themaximal and C1/C2 = 1. It can be seen that when the timeout is very throughput sensitive to timeout. smallthe throughput decreases to zero. The reason for this Theeffect of the timeout period on delay is shownin degradation is thefollowing: when the timeout is small,the Fig. 4,-where thenormalized (maximum) end-to-end delay TC retransmits messages very often; hence the (subjnetwork (T,,*/Xl) is shown as a function of the normalized timeout channelutilization (AJyC1) approaches 1 andthe network for different w’s. As in Fig. 3, this figure also shows that for delay becomes unbounded; because the number of unacknowl- a fixed w, there is a value of timeout which optimizes (min- edged messages is limited, no new message is admitted to the imizes)the end-to-end delay; this optimal timeout (Td*, d networkand the throughput degrades to zero ((7)). It is for delay) is usually different from T~*.When the timeout is interestingto note that in order for the system to remain too small, like the throughput, the end-to-end delay becomes stable, the normalized timeout should be larger than w. This unbounded (theminimum allowable timeout is, of course, fact is intuitivelyclear in the deterministic case: the trans- wX1); asthe value of the timeout increases, the end-to-end missionof w messagestakes wX1 seconds (Xl beingthe delay first reaches a minimum and then becomes unbounded. transmission time of one message), and so a timeout of T < We now study the effect of the number of tokens on the wX1 causes retransmission of a message to start before trans- network performance. In Fig. 5 the normalized throughput is missionof the w messagescan possibly be completed. As a shown as a function of W. On the curve designated “optimal result,the network becomes saturated. This fact can be throughput,”for each w theoptimal value of timeout (T~*) establishedanalytically; the interested reader is referred to is chosen such that the throughput becomes maximal; on the [ 131. We should point’out that this behavior is a consequence curve designated “optimal delay,” the optimal value of time- of the requirement that the ACK for the last (re)transmitted out (T~*)is chosen so thatthe end-to-end delay becomes message should be received. In real life, only when the time- minimal.This figure also shows the throughput curves for out approaches0 does the network becomes saturated. severalconstant (normalized) timeouts. For afixed timeout Continuing our discussion, Fig. 3 also shows that when the the throughput curve is concave and the throughput reduces timeoutincreases, the network throughput first grows; how- tozero for w > T/.F1; thisagrees with our previous result. ever,after a certain value of 7, the throughput starts to de- Thethroughput curves for constant timeout display the crease.In fact, when the destination buffer size (B) is finite catastrophicbehavior of a contention system (described in 276 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. COM-28, NO. 2;FEBRUARY 1980 1 .o 12 0.9 W= OPTIMALTHROUGHPUT 11 10 0.8 >9 9 0s z P 0.7 g7 3 b I W 2 0.6 s I 8 N5 I w= 2 c ;4 I E 3 0 w= 1 23 0.5 2 2 E P 1 5 0.4 0 < 10.0 100.0 1.0 10.0 lwo.o B NORMALIZED TIMEOUT (7 1x1) 8 0.3 Fig. 4. End-to-end delay as a function of timeout (B = 10, C,/C2 = 1.0). 0.2 0.1 1.0 0.0 0.4 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 OPTIMAL THROUGHPUT NORMALIZED NETWORK TRAFFIC (XeP1) 0.8 l:ig. 6.Throughput versus network traffic (B = 10, C,/C2 = 1.0). \ OPTIMAL DELAY the opening remarks of this paper); here w represents the load 0.7 2 of the system. Note that the optimal throughput curve is the a n I maximumenvelope of these contours, and defines the tight W 3 upper houmf on the throughput-token performance. 8 0.6 I To show the effect of the contention more explicitly, we I- have plotted the network traffic versus the network throughput I 3 for(constant) normalized timeouts in Fig. 6. For a fixed T, 0.5 an increase in w results in an increase in the network traffic. E2 Initiallythe throughput increases asthe network traffic N9 grows; however, after a certain value of Xe*, the throughput 1 0.4 startsdeclining, and when thenetwork saturates (i.e., 4 E (h,*/pC,) = l), the throughput decreases to zero. In the same z figure we have also shown the contours of constant w. Note 0.3 thatfor a fixed w, when the timeout increases, throughput firstgrows and then levels off; however, the network traffic decreasescontinuously. Note also that all thew-contours 0.2 peeloff from a common upper bound curve, which is the maximal throughput curve (see Fig. 5). If any one of param- eters w and/or T is held fixed and the other one is increased, 0.1 thethroughput decreases to zero; only when both w and T are increased simultaneously (such that for each w the optimal rt* is chosen) will the throughput not degrade (see the optimal 5 10 15 20 25 throughput curve in Fig. 6). This figure shows that the (nor- T NUMBER OF TOKENS malized) throughput never reaches 1; even when MI and are chosen optimally; the throughput is limited by the destination 1:ig. 5. Network throughput versus number of tokens (B = 10, C,/C2 = node buffer size. 1 .O). Theprice for having more throughput is, of course, the delay which is shown in Fig. 7. This figure shows the through- put-delay equilibrium curves for fixed values of (normalized) IEEE TRANSACTIONSCOMMUNICATIONS, ON VOL.NO. COM-28, 2, FEBRUARY 1980 277 .-5--, 4 .-5--, 2 3 10 20m w=l I 2 I I 0 I 0.25 0.3 0.4 0.5 0.6 0.8 0.7 0.9 0.95 NORMALIZED THROUGHPUT Fig. 8. Throughput-delay for different destination buffer sizes (CI/Cp= 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.0). NORMALIZED THROUGHPUT Fig. 7. Throughput-delay tradeoff(B = 10, C,/C2 = 1.0). area);however, we havechosen the minimum buffer size in order to minimize the overall network cost. timeoutsand number of tokens. The minimum envelope of When thedestination buffer size and the capacities of these contours defines a tight lower bound on the throughput- channels are known, then it is important to know what (w,7) delayperformance and thus, can be viewed as the optinla1 combinationto use in order to provide a certaingrade of network pc’rf‘ortnance for our token mechanism model. As we service.In Fig. 9 we have plotted contours of constant (nor- pointed out earlier, the (normalized) throughput never reaches malized)throughput (the heavy solid curves), and (normal- 1 ; the maximum throughput (which is achieved when w --f 00 ized)end-to-end delay (the heavy dashed curves); we will and 7 -+ m such that 7 is optimal) is determined by the destina- refer theseto curves X*-contoursas and T*-contours, tionbuffer size and the destination channel capacity. For respectively.This figure displays most of the information example,in Fig. 7 thismaximum is achievedwhen the net- contained in theprevious figures; in particular we observe worktraffic approaches its maximum (X,,* = pC1), andfor the foliowing characteristics: thisnetwork traffic the (normalized) throughput is approxi- 1) The locus of minimum allowable (normalized) timeoutsis mately 0.909 (andthe end-to-end delay is, ofcourse, un- theh*(=O)-contour (which c:incides withthe T*(=W)-con- bounded). tour,designated by ‘‘w = 7/X1” inFig. 9. Alongthis curve InFig. 8 wehave shown the optimal throughput-delay the throughput is zero and the delay is unbounded (infinite). curves for different destination buffer sizes and a fixed desti- For small values of 7,the system is unstable. nationchannel capacity (Cl/Cz = 1). Alongeach curve the 2) A line of constantw (a vertical line) may intersecta speci- timeoutand w vary,and we have also shown the contours fiedA*-contour (say A* = X,*) at (a) twopoints; (b) one of constant w. This figure shows that for a fixed end-to-end point (tangent to the A*( = X1*)-contour); and (c) no point. delaythe network throughput increases rapidly as thedesti- In case (a) the two values of timeout give the same through- nation buffer size increases. (Note that for B = 20 the maxi- put (but the delays at these two points are different from each mumthroughput is almostachieved.) When the channel other). In case (b) the value of timeout at the tangent point capacities(network channels as well as thedestination node is rt*, the optimal timeout for maximal throughput (compare channel)are known, this figure can be used for guidance in withFig. 3). We haveindicated the loci of these points by determining a properdestination buffer size. For example, curve A in this figure (which we will refer to as the Tt*-curve). if we set a minimum level of throughput, say (A*/pC,) > 0.72, If the line of constant w does not intersect the A*( = A,*)- and a maximumend-to-end delay, say T,,*/X, < 10,then contour[case (c)], that indicates that a throughput of X1* thisfigure shows that the destination buffer size should be cannot be realized by this w. at least 5 (thepoint specified by the intersection of the 3) The above discussion can be carried out for the T*-con- lightdashed lines in Fig. 8). Notethat a destinationbuffer tours; the loci of points where the constantw lines are tangent size of B > 5 would also satisfy our requirements (the shaded to T*-contours (therd *-curve) is designated by curve B. 27 8 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. COM-28, NO, 2, FEBRUARY 1980 AA z Fig. 9 canbe used inan analogous way in situations where f-4.0 f=8 T-16 T=18.5 T = 32 the end-to-end delay is required to be no more than a given maximum value. Similar to the previous case, the best opera- ting (w,7) value (which results in a maximum throughput) is found by locatingthe intersection of theT*-contour and curve c. IV. CONCLUSION Based ona token mechanism, we developed a model for static flow control in store-and-forward computer communica- tion networks. It was shown that for a w,if the timeout period is too large and/ortoo small, the throughput degrades to zero. In fact, fora given w,there is avalue for timeout which optimizes thethroughput. The model explicitly shows the effect of contentionin that, when the timeout period isfixed, as w (orthe network traffic) increases, the throughput first in- creases and then degrades to zero.By studying the throughput- delay performance of networks (when traffic is controlled by a token mechanism), we found the optimal network perform nnce curve, namely, the minimum envelope of the throughput- delayequilibrium curves for fixed values of timeoutsand number of tokens.A number of design problems were pro- posed and, through a very useful graphical presentation, some guidelinesregarding the optimal selection of system param- eters were given. The development and the analysis of the model presented in this paper was based on a number of simplifying assump- tionsregarding the acknowledgment and timeout strategy, thesubnetwork delay distribution, and the destination node I 5 10 15 20 25 30 structure. We also assumed that the system is operating under NUMBER OF TOKENS heavytraffic condition. These simplifying assumptions, though to some extent idealistic, were necessary to lend the Fig. 9. Contours of throughput and end-to-end delay (E = 10, Cl/CZ = modelamenable to analysis. In view of these,the results 1.0). presented in this paper should be regarded as an approxima- tion to the behavior of the system in real life. Analysisof flow 4) We can use Fig.9 to find the maximum achievable through-control mechanisms is still in its infancy. Much work need to put (and the corresponding w) for a fixedtimeout. The.X*- bedone in this area to quantitatively understand implica- contour which is tangent to the constant 7 line (a horizontal tions of these mechanisms; hopefully, this paper has added to line with an intercept equal to the specified timeout) deter- this understanding. minesthe maximum throughput, and the value of w at the In a forthcoming paper, by using the results of this static tangentpoint is the w whichprovides this throughput. It is flow control analysis, we develop and study a dynamic flow interestingto note that this value of thethroughput is the control mechanism in which the number of tokens and tim- point where the 7-constant line intersects the envelope of the outare dynamically adjusted to the stochastic fluctuations throughput curves in Fig.3 (we have not drawn this curve). in the load on the system. 5) Fig. 9 can be used to find the best (w, 7) combination which guarantees a certain grade of service. Assume that for REFERENCES a given network ‘(for which the parameters 1-1, C2, and B Cl, [I] R. Kahn and W. Crowther, *,‘Rowcontrol in resource shanng computer areknown) we require at least a minimum throughput (say networks,” IEEE Trans. Commun.. vol. COM-20, pp. 546-550, June X*/c1> 0.85).In order to find the (w,7) whichresults in 1972. theminimum end-to-end delay, we tracethe X*( = 0.85)- [2] C. E. Agnew, “Dynamicmodeling and control of congestion-prone contour and find-the T*-contour that is tangent to it (in Fig. 9 systems,’’ Dep. Eng.-Econ. Syst., Stanford Univ., Stanford, CA, Tech. Rep. 10, Jan. 1974. the T*( = 18.5)-contour is such a contour, which is tangent [3] L. Pouzin, “Flow control in data networks-methods and tools,” in Conf. to the x*( = 0.85)-contour at w = 20 and 7/X1= 73). A.ny Proc.Int. Conf. Comput.Comm., Toronto, Canada, Aug. 1916,pp. (w, 7) other than 20,73) results in a higher delay; hence this 467414. pair is the best choice. The loci of these tangent points (or the [4] D. W. Davies, “Thecontrol of congestion inpacket’ switching networks,” in Proc. 2nd ACM-IEEE Symp.Problems of the best (w, 7) combinations) is shown in Fig. 9 as curve C. Note Optimization of Data Communication, Palo Alto, CA., Oct. 1971, pp. that we can use a smaller w and still have the same throughput 4649. (e.g., intersection of h*( = 0.85)-contour and curve A);how- [5] L. Kleinrock, QueueingSystems. Vol. I/: Computer .ever, the delay at this point is larger than18.5. Considering Applications. NewYork: Wiley-Interscience,1976. [6] J. M. McQuillan, W. R. Crowther, Cossell, C. Walden, andF. the ,fact thatw represents the buffer size of the traffic control- B. P. I>. E. Heart, “Improvements in the design and performance of the ARPA ler,this latter operating point (with a smaller w) maybe a network,” in 1972 Fall Joint Comput. Conf., AFIPS Canf. Proc., vol. better choice, as it results in a lower network (buffer) ‘cost. 41,Anaheim,CA, Dec. 1972, pp. 741-754. TRANSACTIONS ON COMMUNICATIONS, VOL. COM-28, NO. 2, FEBRUARY 1980 279 V. G. Cerfand R. E. Kahn. “A protocolfor packet network I. INTRODUCTION Interconnection,” IEEE Trans. Commun., vol. COM-22, pp. 637-648, May 1974. Packet switching, a new way to establish communications H. Rudin, “Chairman’s remarks: An introduction to flow control.” in between computer and computer and between user and com- Proc. Int. Conf. Commun.,Toronto, Canada, Aug. 1976, pp.463-466. puter, began with the development of the ARPANET [ 11441‘ S. S. Lam and M. Reiser. “Congestion control of store-and-forward in 1969.The ALOHA system at the University of Hawaii networks by input bufferlimits,” in P roc. Nat. Telecommun., Conf,,Los [ 51, [6] wasthe first network to employ radio links for Angeles, CA, Dec. 1977, pp. 12:1-1-12:1-6. M. C.Pennotti and M. Schwartz. “Congestion control in storeand packet communication. In the ALOHA system, user terminals forwardtandem links,” IEEE Trans. Commun., vol. COM-23, pp. are connected to a central computer via a single shared UHF 1434-1443, Dec. 1975. radio channel. The first analyses of packet radio performance C. A. Sunshine,,“Interprocess communication protocols for computer assumed that packet collisions were the major cause for loss networks,” Stanford Electronics Lab., Stanford Univ., Stanford, CA, Tech. Rep. 105, Dec. 1975. ofa packetand subsequent retransmission. More recently; A. Giessler, J. Haenle, A. Koenig, and E. Pade. “Packet networks with effortsto design packetradio systems to operate over de- deadlock-freebuffer allocation: An investigation by simulation,” gradedchannels have been undertaken [ 71. Thispaper ex- Comput. Networks, vol. 2, pp. 191-208. July 1978. aminesthe performance ofthe pure-ALOHA single-hop P.Kermani, “Switching and flow control techniques in computer packet radio technique over a slow Rayleigh fading channel. communicationnetworks,” Ph.D. dissertation, Comput. Sci. bp., Univ. Calif., Los Angeles, CA, Dec. 1977. Thereare several applications including ionospheric scintil- L. Kleinrock, “On flow control in computer networks,” in Proc. Int. lationsexperienced in satellite communications [ 81 , [91, ConJ Commun.,Toronto, Ont., Canada, June 1978, pp. 27.2.1-27.2.5. radiocommunications in an urban environment [ 101,and L. Kleinrock and P. Kermani. “A network algebra for the performance HF radio [ 1 11 . evaluationof interconnected computer networks,” in Proc.Nat. Previousefforts have addressed blocked data transmission Telecommun. Conf., Birmingham, AL,Dec 1978. L. Kleinrock and W. E. Naylor, “On measured behavior of the ARPA overrandom channels. Cavers [ 121 considersbuffer control network,” in 1974 Spring Joint Comput.Conf,, AFIPSConf. Proc.,vol. usingvariable-rate blocked-data transmission over slow Ray- 43, Chicago, IL,May 1974, pp. 767-780. leigh fading channels. The variable-rate techniques, as well as L. Kleinrock, CommunicationNets: Stochastic Message Flow And other variable communications parameter methods for mitiga- Delay. NewYork: McGraw-Hill, 1964. G. D.Cole, “Computer network measurements: Techniques and ting the effects of fading have received considerable attention experiments,”Comput. Sci. Dep., Univ. Calif., Los Angeles, CA, [ 131 -[ 181, but, with the exception of the paper by Cavers, UCLA-ENG-7165,Oct. 1971. none consider blocked data. Eaves and Levesque [ 191 analyze J. W. Forgie,“Speech transmission in packet-switchedstore-and- the probability of block error in slow Rayleigh fading. They forward networks,” in 1975 Spring Joint Comput. Conf. AFIPS Conf. Proc., Anaheim, CA, May 1975, pp. 137-142. develop a seriessolution for noncoherent frequency-shift- D.R. Cox, Renewal Theory. London: Butler, Tonner, 1967. keyingwhich is alsoderived here. Their final results are L.Kleinrock, QueueingSystems, Vol. I: Theory. New York: Wiley- obtained by numerical integration. Interscience, 1975. Thework described here differs from previous efforts in J. M. Ortega and W. C. Rheinbold. IterativeSolution of Nonlinear a number of significantways. First, we emphasize the per- Equations in Several Variables. New York: Academic, 1970. formance of a multiaccesstechnique, specifically pure- ALOHA, in a sbw Rayleigh fading environment. The goal is tocharacterize the performance of systems in fading which are not designed specifically to operate over fading channels. Anexample would be a satellitesystem for mobile users Packet Radio Performance OverSlow Rayleigh Fading Channels operating at UHF which is subject to ionospheric scintillation. Also, as opposed to the numerical results presented by Eaves JAMES A. ROBERTS, SENIOR MEMBER, IEEE, AND andLevesque [ 191,this paper derives closed-form expres- TIMOTHY J. HEALY, MEMBER, IEEE sionswhich are either exact results or excellent approxima- tionsfor the probability of block error in slow Rayleigh fading. The approximation technique is applicable in general Abstract-Expressions for the throughput andaverage packet delay for a to slow fading channel statistics and to arbitrary modulation pure-ALOHA single-hop packet radio system operating in slow Rayleigh formats. fading are derived. For noncoherent kequency-shift-keying (NCFSK),an exact closed form expression is presented. For coherent phase-shift-keying 11. PACKET RADIO SYSTEM MODELING FOR FADING (CPSK) an excellent approximation for large packet sizes is derived. This CHANNELS approximation techniqueis valid in general for other modulation schemes and for other fading channel statistical characterizations. Thepacket The channel throughput and packet delay of packet radio length which maximizes the useful data throughput in slow Rayleigh fading channelshave been determined for basic’ system concepts is found. The results of thisinvestigation indicate that a packet radio system suchaspure-ALOHA, slotted-ALOHA, and carrier sense can be.designedwith a modest link margin for fading and achieve identical multipleaccess (CSMA) [20]-[23]. However, none of these throughput performance over a nonfading channel and a fading channel fundamental calculations includes the effect of link errors due with only a small increase in average packet delay for thefading channel. to noiseand fading. In the absence of fadingthe noiseless assumption is quite good, but on a fading channei the signal- Paper approved by the Editor for Computer Communication of the to-noise ratio becomes a critical parameter. The approach in IEEE Communications Society for publication without oral presenta- thissection is to derivepure-ALOHA throughput and delay tion. Manuscript received January 19, 1979; revised August 16, 1979. J. A. Roberts is with ESL Incorporated, Sunnyvale, CA 94086. as a function of theprobability of a packeterror T. J. Healyis with the Department of Electrical Engineering and andthe probability of an acknowledgment error. We will Computer Science, University of Santa Clara, Santa Clara, CA 95053. assume that individual user transmissions are independent of 0090-6778/80/0200-0279$00.75 0 1980 IEEE