A Framework for Alternate Queueing: Towards Traffic Management by PC-UNIX Based Routers Kenjiro Cho Sony Computer Science Laboratory, Inc. Tokyo, Japan 1410022 [email protected] Abstract method to implement an alternative queueing discipline, which is the main obstacle to incorporating alternative Queueing is an essential element of traffic manage- queueing disciplines. ment, but the only queueing discipline used in traditional UNIX systems is simple FIFO queueing. This paper de- On the other hand, the rapidly increasing power of scribes ALTQ, a queueing framework that allows the use PCs, emerging high-speed network cards, and their drop- of a set of queueing disciplines. We investigate the is- ping costs make it an attractive choice to implement an sues involved in designing a generic queueing framework intelligent queueing on PC-based routers. Another driv- as well as the issues involved in implementing various ing force behind PC-based routers is flexibility in soft- queueing disciplines. ALTQ is implemented as simple ware development as the requirements for a router are extension to the FreeBSD kernel including minor fixes growing. to the device drivers. Several queueing disciplines in- In view of this situation, we have designed and built cluding CBQ, RED, and WFQ are implemented onto ALTQ, a framework for alternate queueing. ALTQ al- the framework to demonstrate the design of ALTQ. The lows implementors to implement various queueing dis- traffic management performance of a PC is presented to ciplines on PC-based UNIX systems. A set of queue- show the feasibility of traffic management by PC-UNIX ing disciplines are implemented to demonstrate the traf- based routers. fic management abilities of PC-UNIX based routers. ALTQ is designed to support a variety of queue- ing disciplines with different components: scheduling 1 Introduction strategies, packet drop strategies, buffer allocation strate- Traffic management is of great importance to today’s gies, multiple priority levels, and non-work conserving packet networks. Traffic management consists of a di- queues. Different queueing disciplines can share many verse set of mechanisms and policies, but the heart of parts: flow classification, packet handling, and device the technology is a packet scheduling mechanism, also driver support. Therefore, researchers will be able to im- known as queueing. Sophisticated queueing can pro- plement a new queueing discipline without knowing the vide performance bounds of bandwidth, delay, jitter, and details of the kernel implementations. loss, and thus, can meet the requirements of real-time Our framework is designed to support both research services. Queueing is also vital to best-effort services to and operation. Once such a framework is widely de- avoid congestion and to provide fairness and protection, ployed, research output by simulation can be easily im- which leads to more stable and predictable network be- plemented and tested with real machines and networks. havior. There has been a considerableamount of research Then, if proved useful, the discipline can be brought into related to traffic management and queueing over the last routers in practical service. Availability of a set of queue- several years. ing disciplines will raise public awareness of traffic man- Many queueing disciplines have been proposed and agement issues, which in turn raises research incentives studied to date by the research community, mostly by to attack hard problems. analysis and simulation. Such disciplines are not, how- Another important issue to consider is deployment of ever, widely used because there is no easy way to im- the framework. To this end just a framework is not plement them into the existing network equipment. In enough. In order to make people want to use it, we BSD UNIX, the only queueing discipline implemented have to have concrete queueing disciplines which have is a simple tail-drop FIFO queue. There is no general specific applications. Therefore, we have implemented 1 s = splimp(); ip_output if (IF_QFULL(&ifp->if_snd)) { IF_DROP(&ifp->if_snd); splx(s); m_freem(m); if_output return(ENOBUFS); } altq_enqueue IF_ENQUEUE IF_ENQUEUE(&ifp->if_snd, m); if ((ifp->if_flags & IFF_OACTIVE) == 0) if_snd alternative alternative discipline 1 discipline 2 (*ifp->if_start)(ifp); splx(s); IF_DEQUEUE altq_dequeue Figure 2: Enqueue Operation in if output if_start for the output queue of a network interface. The input Figure 1: Alternate Queueing Architecture queue has less importance because traffic control func- tions only at the entrance to a bottleneck link. In BSD UNIX, an output queue is implemented in the abstract Class-BasedQueueing (CBQ) [6] and RandomEarly De- interface structure ifnet. The queue structure, if snd, tection (RED) [5] targeting two potential user groups. is manipulated by IF ENQUEUE() and IF DEQUEUE() CBQ can be used for integrated services. RSVP [22] is macros. These macros are used by two functions regis- a resource reservation protocol for integrated services; tered in struct ifnet, if output and if start; if output de- RSVP itself is a signaling protocol to set up the traffic fined for each link type performs the enqueue operation, control module of the routers along a path. The RSVP and if start defined as part of a network device driver release from ISI [9] does not have a traffic control mod- performs the dequeue operation [13]. ule so that there are great demands for a queueing im- One might think that just replacing IF ENQUEUE() plementation capable of traffic control. Although CBQ and IF DEQUEUE() will suffice, but, unfortunately, it was not originally designed for use with RSVP, CBQ has is not the case. The problem is that the queueing op- been used by Sun as a traffic control module for RSVP erations used inside the kernel are not only enqueueing on Solaris and the source code has been available [20]. and dequeueing. In addition, surprisingly many parts of RED is for active queue management of best-effort the kernel code assume FIFO queueing and the ifqueue services [2]. Active queue management has been exten- structure. sively discussed to avoid congestion in the Internet. Al- To illustrate the problem, let’s take a look at the en- though CBQ can be used for this purpose, RED seems queue operation in a typical if output in Figure 2. The more popular since RED does not require flow states code performs three operations related to queueing. or class states, and thus, is simpler and more scalable. Again, no implementation is available at hand. 1. checkif the queueis full by IF QFULL(), andif so, Another important factor for widespread acceptance drop the arriving packet. is simplicity and stability of the implementation, which 2. enqueue the packet by IF ENQUEUE(). caused us to emphasize practicality instead of elegance while designing ALTQ. 3. call the device driver to send out the packet unless In summary, the goals of our prototype are three-fold: the driver is already busy. provide a queueing research platform. The code assumes the tail-drop policy, that is, the arriv- ing packet is dropped. But the decision to drop and the provide a traffic control kernel for RSVP. selection of a victim packet should be up to a queue- make active queue management available. ing discipline. Moreover, in a random drop policy, the drop operation often comes after enqueueing an arriv- In this paper, we will show the design and implemen- ing packet. The order of the two operations also de- tation of ALTQ, and report the traffic management per- pends on a queueing discipline. Furthermore, in a non- formance of the prototype on FreeBSD [7]. work conserving queue, enqueueing a packet does not mean the packet is sent out immediately, but rather, the 2 ALTQ driver should be invoked later at some scheduled timing. Hence, in order to implement a generic queueing inter- 2.1 ALTQ Design face, we have no choice but to replace part of the code in The basic design of ALTQ is quite simple; the queue- if output routines. ing interface is a switch to a set of queueing disciplines There are also problems in if start routines. Some as shown in Figure 1. Alternate queueing is used only drivers peek at the head of the queue to see if the driver 2 has enough buffer space and/or DMA descriptors for the the hook. This method has other advantages; the system next packet. Those drivers directly access if snd using can fall back to the original queueing if something goes different methods since no procedure is defined for a wrong. As a result, the system becomes more reliable, peek operation. A queueing discipline could have multi- easier to use, and easier to debug. In addition, it is com- ple queues, or could be about to dequeue a packet other patible with the existing user programs that refer to struct than the one at the head of the queue. Therefore, the ifnet (e.g., ifconfig and netstat). peek operation should be part of the generic queueing Queueing disciplines are controlled by ioctl system interface. Although it is possible in theory to rewrite all calls via a queueing device (e.g., /dev/cbq). ALTQ is de- drivers not to use peek operations, it is wise to support fined as a character device and each queueing discipline a peek operation, considering the labor required to mod- is defined as a minor device of ALTQ. To activate an al- ify the existing drivers. A discipline must guarantee that ternative queueing discipline, a privileged user program the peeked packet will be returned by the next dequeue opens the queue device associated with the discipline, operation. then, attaches the discipline to an interface and enables IF PREPEND() is defined in BSD UNIX to add a it via the corresponding ioctl system calls. When the al- packet at the head of the queue, but the prepend oper- ternative queueing is disabled or closed, the system falls ation is intended for a FIFO queue and should not be back to the original FIFO queueing.