Fiber Distributed Data Interface

Digital Technical Journal Digital Equipment Corporation

Volume 3 Number 2

Spring 1991 Editorial Jane C. Blake, Editor Kathleen M. Stetson, Associate Editor Circulation Catherine M. Phillips, Administrator Suzanne]. Babineau, Secretary Production Helen L. Patterson, Production Editor Nancy Jones, Typographer Peter Woodbury, Illustrator Advisory Board Samuel H. Fuller, Chairman Richard W Beane Robert M. Glorioso Richard]. Hollingsworth John W McCredie Alan G. Nemeth Mahendra R. Patel F. Grant Saviers Robert K. Spitz Victor A. Vyssotsky Gayn B. Winters

The Digital Technicaljournal is published quarterly by Digital Equipment Corporation, 146 Main Street ML01-3/B68, Maynard, Massachusetts 01754-2571. Subscriptions to the Journal are $40.00 for four issues and must be prepaid in U. S. funds. University and college professors and Ph.D. students in the electrical engineering and computer science fields receive complimentary subscriptions upon request. Orders, inquiries, and address changes should be sent to The Digital Technicaljournal at the published-by address. Inquiries can also be sent electronically to Dl]®CRL.DEC.COM. Single copies and back issues are avai!able for $16.00 each from Digital Press of Digital Equipment Corporation, 12 Crosby Drive, Bedford, MA017 30-1493. Digital employees may send subscription orders on the ENE T to RDVAX::JOURNAL or by interoffice mail to mailstop ML01- 3/B68. Orders should include badge number, cost center, site location code and address. All employees must advise of changes of address. Comments on the content of any paper are welcomed and may be sent to the editor at the published-by or network address. Copyright© 1991 Digital Equipment Corporation. Copying without fee is permitted provided that such copies are made for use in educational institutions by faculty members and are not distributed for commercial advantage. Abstracting with credit of Digital Equipment Corporation's authorship is permitted. All rights reserved. The information in this Journal is subject to change without notice and should not be construed as a commitment by Digital Equipment Corporation. Digital Equipment Corporation assumes no responsibility for any errors that may appear in this Journal. ISSN 0898-901X Documentation NumberEY-H876E-DP The following are trademarks of Digital Equipment Corporation: CI, DDCMP, DEC, DECbridge, DECconcentrator, DEC FDDicontroller, DECnet, DECmcc, DECstation, DECWORLD, Digital, the Digital logo, DNA, I.AT, ThinWire, TURBOchannel, ULTRIX, VAX, VAXcluster, VMS. Cover Design AppleTalk is a registered trademark of Apple Computer, Inc. Our cover depicts the dual fiber-optic ring that isthe physical Cis a registered trademark of Microsoft Corporation. medium for high-speed transmission of data in Digital's new Motorola is a registered trademark of Motorola, Inc. FDD/local area network. High-performance workstations, SUN is a registered trademark and NFS is a trademark of Sun systems, servers, and other LANs can be easily connected to the Microsystems, Inc. ring through new bridge and wiring concentrator products, X Window System is a trademark of the Massachusetts Institute which are among the topics featured in this issue on FDDI. of Technology. UNIXis a registered trademark of UNIX System Laboratories, Inc. The cover was designed byDave Bryant of Digital's Media Book production was done by Digital's Media Communications

Communications Group. Group in Bedford, MA. Contents

8 Foreword Mark F. Kempf

Fiber Distributed Data Interface

10 Fiber Distributed Data Interface Overview William R. Hawe, Ri chard Graham, and Peter C. Hayden

19 Development of the FDDI Physical Layer jerry D. Hutchison, Christopher Baldwin, and Bruce W Thompso n

31 FDDI Data Link Development Henry S. Yang, Barry A. Spinney, and Stephen To wning

42 An Overview of the Common Node Software Paul W Ciarfclla, David Benson, and David S. Sawyer

53 Development of the DECbridge 500 Product Ro bert C. Kochem, James S. Hiscock, and BrianT. Mayo

64 The DECconcentrator 500 Product William]. Tiffany, G. Paul Ko ning, and james E. Kuenzel

76 DECelms-Managing Digital's FDDI and Ethernet Extended Local Area Networks Bruce E. Sweet

85 ULTRIX Fiber Distributed Data Interface Networking Subsystem Implementation Ursula Sinkewicz, Chran-Ham Chang, Lawrence G. Palmer, Craig Smelser, and Fred L. Templin I Editor'sIntroduction

Consistent behavior in the physical and data link layers is managed by a set of reusable soft­ ware libraries called Common Node Software. Paul Ciarfella, Dave Benson, and Dave Sawyer relate the events that led to the development of CNS and describe its functions. CNS implements the proto­ cols defined by the FDDI station management stan­ dard and manages the fOf)[ chip set. CNS is included in the DECbridge and DECconcentrator products. The DECbridge '500 product is the bridge for traffic between the high-speed FDD! LAN and the slower Ethernet H02.::'> LANs. Design considerations Jane C. Blake and DECbridge functions are presented by Bob Editor Kochem, jim Hiscock, and Brian Mayo. Their discus­ Digital's fiber distributed data interface, FDDI, is a sion otJcrs insight into the complexities involved high-speed LAN that links workstations, systems, in connecting fANs with different data rates, frame and other local area networks such as Ethernet. formats, and frame sizes. This new !.AN offers distance as well as speed, and Bi ll Tiffany, Paul Koning , and jim Kuenzel then extends connections beyond the limits of a build­ describe the DECconcentrator '500 product. The ing to cover wider areas such as a campus. Papers DECconcentrator, the cornerstone of the FD])] LAN, in this issue of the Dip,ittt! Technical joumaf pro­ provides additional ports to which stations can be vide insights into the tech nology choices made connected by radially wired cables. The authors during FDDI development and describe the design examine the significance of Digital's choice of a dual of the layers and products that make up this 100- ring of trees topology, which led to the need for a megabit/second LA.J'\1 system. concentrator, and give detai Is of DECconcentrator Among the reasons Digital's engineers chose FDDI development. technology were its high speed, high throughput, Remote management of the DEC:concentrator and consistency with existing and evolving stan­ and DECbridge products, as well as of Ethernet dards. In this issue's opening paper, .Bill Hawe. Rich bridges, is provided by DECelms software. Bruce Graham , and Peter Hayden review these selection Sweet outlines the challenges DECelms developers criteria and discuss tbe reasoning behind the final faced, including an evolving ANSI FDDr standard choices. The authors additionally present an over­ and differences between FDDI and Ethernet tech­ vinv of FDDI layers and product operations, which nologies. He then describes the network nunage­ establishes a context for the papers that follow. ment architecture and gives details of features that The lowest FDDI layer provides the physical con­ benefit the network manager. nections for data transmission on the fiber-optic The final paper in this issue addresses the devel­ ring. In their paper, Jerry Hutchison , Chris Baldwin, opment of an adapter that allows high-performance and Bruce Thompson describe the operation of the RISC workstations to connect to FOOl. Ursula physical layer, the .functiona I partition in g. and the Sinkewicz, Chran-Ham Chang , Larry Palmer, Craig choice of chip set technologies. They then exam­ Smelser, and Fred Templin review ULTRIX support ine the distributed clocking scbeme ami present for the FDDr system and then give detai Is of the DEC the methods used in the design of the optical link FDDlcontroller 700 adapter, which provides a single (methods later adopted by the Physical Layer FDOJ attachment for DECstation '5000 workstations. 1Yiedium Dependent Working Group of the FDDI The authors have included discussions of relevant committee). performance data. Both the development of the physical layer and The editors thank Mark Kempf for his help in the next higher layer, the data link layer, resulted in initiating and preparing this issue, and for kind ly contributions to the ANSI f'DDI standard. Digital's agreeing to write the issue's Foreword. implementation of the data link layer is the subject of the next paper by Henry Yang, Barry Spinney. and Steve Towni ng. In acltl ition to presenting several key algorithms, the authors review the functions of the three data link chips. They conclude their paper with a discussion of chip simulation and test.

2 Biographies I

Christopher Baldwin As a principal engineer in the Network Systems Engineering Group, Christopher Baldwin is responsible fo r the development of the fiber-optic physical layer standard fo r FDDI (I'M D) and helped develop the fiber-optic hardware used in FOOl products. He also worked on fiber-optic transmission fo r the LANBridge 200. Before joining Digital in 1986, Chris was a senior engineer at Polaroid Corporation. He holds a H.A. ( 1983, honors) in elec­ trical engineering from Brown University and an M.S. (1984)from the Institute of Optics at the University of Rochester.

David Benson Software principal engineer Dave Benson is a member of the Communications Systems Engineering Group and the project leader for the Common Node Software. His next responsibility will be as a firmware project leader in the area of fu ture FOOl products. Previously, Dave designed and imple­ mented the physical layer portion of CNS, codeveloped the design verification test (OVT) monitor tool, and was a member of the hardware design team fo r the FODI tester. Dave came to Digital in 1986 with ten years' engineering experience from Honeywell Information Systems.

Chran-Ham Chang Chran-Ham Chang is a member of the ULTRJX Network Engineering Group. As a senior software engineer, he is responsible fo r the ULTRJX FODIan cl Ethernet driver design and development. Earlier, Chran was involved in the analysis of ULTRJX network performance and in the design of performance tools. He joined Digital in 1987 after receiving his M.S.C.S. from New jersey Imtitute of Te chnology. Prior to this, Chran worked as a software specialist in Ta iwan fo r the distributor of Digital's products.

Paul W. Ciarfella As a software senior engineer in the Communications Systems Engineering Group, Paul Ciarfella is currently developing DECmcc soft­ ware ro manage FDDI networks. Paul was one of the developers of the Common Node Software. Previously, he performed simulation and design verification testing of Digital's FOOl chip set ancldeveloped system software fo r the FDDI tester. Paul joined Digital in 1987 after receiving a B.S.C.S. (h igh honors) from

Northeastern University. He is a co-applicant fo r a parent related ro increasing the robustness of fDDI networks. Niogmpliies

Richard Graham Richard Graham is a consulting engineer in the Local Area Networks Business Unit. Since joining Digital in 1981, he has worked on a num­ ber of projects, including early Ethernet development, the Ethernet repeater, the Ethernet-to-Ethernet bridge (LAN Bridge 100), DEC Standard 134, and most recently, the FDDI-to-Ethernetbridge. Prior to this, Rich was a senior engineer in the Computer System Division of Harris Corporation. He holds a B.S. (1975) in electrical engineering and an M.S. (1977) in computer engineering, both from Syracuse University.

William R. Hawe Senior consulting engineer 13ill Hawe manages the LAN Architecture Group. He is involved in designing new LANs and extended LAN technologies. He and his group also design portions of DECnet and TCP/IP archi­ tectures. While in Corporate Research, Bi ll worked on the design of the Ethernet with Xerox and Intel. Since joining Digital in 1980, Bill has done extensive per­ fo rmance analysis and has established this as an integral part of Digital's net­ working architecture. He holds a B.S.E.E. and an M.S.E.E. He has published 27 papers outside Digital and has 11 patents issued or pending.

Peter C. Hayden Peter Hayden is a principal engineer in the Telecommu­ nications and Networks, Software Development Group. He joined Digital in 1986 as a member of the FDDI team and has contributed to the development of the FDDI technology and product set. He was instrumental in the development of the FDDI test riatform which allowed fo r execution of test programs in both simulation and hardware environments. Peter also Jed the team that developed the FDDI control software resident in all FDDI products. He holds B.S.E.E. and M.S.C.S. degrees from Union College in Schenectady, NY

james S. Hiscock Principal software engineer James Hiscock is currently a firmware project leader fo r a fu ture bridge/router product. Prior to this, he led the DECbridge 500 and LA N Bridge 200 firmware projects. He also worked on the UN II3US-to-Ethernetadapter and the LAN Bridge 100 device. Before coming to Digital in 1984, Jim was employed as a software engineer in the air traffic control division of Raytheon Company. He received a B.S. ( 1982) in computer systems engineering from the University of Massachusetts.

Jerry D. Hutchison Jerry Hutchison joined Digital in 1977 after receiving B.S. and M.S. degrees in physics, both from Carnegie-Mellon University. As a consult­ ing engineer in the Archictecture and Advanced Development Group, he works on aspect� of FDDI technology. Currently an FDDI architect, Jerry was involved in optics development, physical layer and MAC protocols, connection manage­ ment, and standards. Previously, Jerry designed digital and analog circuitry and worked on Ethernet and computer interconnect products. He holds one patent related to the FDDI physical layer and has fo ur patents rending.

4 Robert C. Ko chem Robert Kochem is an engineering manager in the LAN Interconnect Engineering Group. He was responsible fo r managing the DECbridge 500 development effort and is presently working on additional LAN interconnect products. In earlier work, Bob led the hardware team that devel­ oped the test bed for the FDDI chip set. Before joining Digital in 1986, he was employed by Codex and American Science and Engineering in management positions. Bob holds B.S. E. E. (1972) and M.E.E.E. (1973) degrees from Rensselaer Polytechnic Institu te.

G. Paul Ko ning As a consulting engineer in the Distributed Systems Archi­ tecture Group, Paul Koning is involved in work related to FDDI architecture, including the specification of Digital's internal architecture and participation in the ANSI FDDI standards development. In this last capacity. he is a representa­ tive to the ANSI X3T9.5 standard committee. Since joining Digital in 1978, Paul has also worked on RSTS/E and DECnet/E development projects. He has several patents pending on various aspects of LAN and WAN technology. Paul received a B.S. (1975) in physics from Lawrence University.

James E. Kuenzel As manager of the VAXcluster Systems Engineering consul­ tant team, James Kuenzel is responsible for ensuring that new products ami technological directions are well integr ated with the existing VAXcluster system. He is currently definingthe long-term strategy fo r the next generation VAXcluster interconnect. Prior to this, Jim was involved in the planning and development of Digital's FDDI product strategy and was a representative to the ANSI FDDI stan­ dards committee. jim is a 1972 graduate of Philco-Ford Te chnical Institute.

Brian T. Mayo Brian Mayo is a principal hardware engineer working in the LAN Interconnect Engineering Group. He was the project engineer fo r the DECbridge 500 FDDI-to-Ethernet bridge. Previous work includes advanced devel­ opment efforts in bridging other heterogeneous networks. Before joining Digital, Brian was a product development manager at Gould-Mod icon in the Distributed Networking Group. He has also been both a development manager and an oper­ ations manager fo r Teradyne in the Systems Test Division. Brian received his B.S.E.E. (1978) from Cornell University.

La wrence G. Palmer Larry Palmer is a principal software engineer wi th the Open Systems Networking Group. He currently leads the MicroKernel advanced development project and has been a member of the ULTIUX team since joining Digital in 1984. Larry is one of the three software developers who initiated the PM.AX software project for the DECstation 3100 product by porting the UL.TRJX operating system to the MIPS architecture. He received a B.S. (1982) in chemistry with highest honors from the Unive rsity of Oklahoma and is a member of Phi Beta Kappa. HiograjJhies

David S. Sawyer In his position as software principal engineer, David Sawyer was the project leader for software development on the DEC FDDlcontroller 700 adapter, common node software, and system software on the FDDI tester. Before joining the Communications Systems Engineering Group, he led effor ts for the design verification of the HM chip, the FDDI physical layer protocol analysis, and the FDDI chip implementation. Dave came to Digital in 1983 after receiving a B.S.E.E. (magna cum laude) fro m Northeastern University. He is a member of Phi Kappa Phi, Tau Beta Pi, and Eta Kappa Nu.

Ursula Sinkewicz Currently a principal engineer, Ursula Sinkewicz has spent six of her eight years at Digital working on the ULTRIX operating system and commtmications. She was the project leader for networking changes in lJI.TRIX symmetrical multiprocessing and worked on the 2780/3780 HJE terminal emula­ tion for the lLTRfX operating system. Ursula holds a B.S. in physics ancl mathe­ matics fr om George Washington University and an M.S. in mathematics fr om Rensselaer Polytechnic Institute. Her master's thesis was a mathematical model for pred icting plate subduction near earthquakes.

Craig Smelser Craig Smelser is a principal engineer who currently manages the ULTRJX Network Engineering (;roup. Prior to this, he worked in the Te le­ communications and Network Group where he designee! ancl developed the reusble operating system used throughout the FDDI product set. Craig also led the team that delivered the :)270 terminal option for the DECserver hardware. He earned B.S. degrees (1980) in both mathematics and computer science fr om Geneva College and an M.S. ( 1987) in software engineering fr om the Wang Institute of Graduate Studies. Craig joined Digital in 1987

Barry A. Spinney Since joining Digital in 1985, Barry Spinney has been responsible for managing the development of FDDI chips, including the \lAC. the RMC, ancl the CAM. He is currently a consulting engineer in the Distributed Systems Advanced Development c;roup working on next-generation LAN prod­ ucts. In prior work, Barry was a senior software engineer at Softech, Inc. He holds a B.S. (1979, double honors) in mathematics and computer science from the University of Waterloo and an tvi.S.C:.S. (1981) from the University of Toronto. Barry is a member of IEEE and ACM and has fo ur patents pending.

Bruce E. Sweet Br uce Sweet is a software engineering supervisor respon­ sible for the development of network manage ment software for Digital's IAN Bridge and FDDI networking products. In previous work, he was the project leader for the RBMS V2 0 prod uct and for the IAN traffic monitor V 1.0 listener software. As an individual contributor to the DECrouter 200 product, Bruce implemented the Ethernetdata link and DDCMP protocols and real-time kernel and modem control. He joinell Digital in 198:) after receiving a B.S.C:.S. fr om Northeastern UniversitY. Fred L. Templin Freel Te mplin joined Digital in 1986 and has been a member of the ULTRIX Networks and Communications team until his recent transfer. He was technical project leader fo r the ULTRIX Ethernet adapter device drivers ami fo r the ULTRIX FOOl engineering effort. Currently, Freel works as a sales support consultant fo r ULTRIX/Open Systems and networking in the Santa Clara Com­ mercial District Sales Office. He received a B.S.C.S. (1983, University Scholar) and an M.S.C.S. (1986), both from Pennsylvania State University.

Bruce W. Thompson Bruce Thompson is a consulting engineer in the Communications Systems Engineering Group. Since joining Digital in 1985, he has worked on Digital's FDDI chip set and FOOicontroller 700 adapter fo r the OECstation 5000 Model 200 workstation. Bruce also helped to define Digital's FOOl wiring concentrator architecture. Previously, he was a principal engineer at Wa ng Laboratories and Network Switching Systems. Bruce holds a ll.S. in elec­ trical and computer engineering from the University of Massachusetts. He has one patent on an aspect of the FOOl physical layer implementation.

William J. Tiffany As a principal engineer in the Network Systems Engineering Group, Bill Tiffany worked on the DECconcentrator 500 project and is currently involved in developing next-generation FDDI products. Earlier, he worked as a diagnostics/hardware engineer on the DECrepeater 200 prod­ uct. Before coming to Digital in 1986, Bill was employed by Integrity Commu­ nications, Raytheon Company, and General Dynamics Corporation. He holds a B.S. (1977) in engineering science from the University of Tex as and an M.S.E.E. from the California Institute of Te chnology.

Stephen Towning Principal Engineer Stephen Towning is a member of the Corporate Backbone Systems Group located in Ireland. He has worked in the area of diagnostics and test fo r 15 years. Steve joined Digital's FDDI program in 1986 as an independent consultant to establish test and verification strategy. He became a fu ll-time Digital employee in 1989. Steve received a B.A. (1975) with joint honors in biology and psychology from Keele University in England. He is a member of ACM and IEEE.

Henry S. Yang As a consulting engineer fo r the Distribu ted Systems Archi­ tecture Group, Henry Yang's responsibilities fo cus on the FOOl and Ethernet architectures. Since joining Digital in 1976, he has contributed to the develop­ ment of the Ethernet adapters, synchronous communication adapters, the LAN address ROM system, Ethernet chips, the FDDI technology and products, and sev­ eral Digital standards. Henry has 11 patents either issued or pending on Ethernet and FOOl. He holds a B.S.E.E. (1976, honors) from the University of To ronto and an M.S.E.E. (1988) from NortheasternUn iversity.

7 I Foreword

(today, of course, it is). However, because the stan­ dard was incomplete, our plans had to accommo­ date the implementation of specifications that would not always be known at convenient times. Further, because FDDI had not yet been implemented, we had to expect to discover errors in the existing document and to work with the standards organi­ zation to correct them. The firs� group of articles in this issue of the Digital Te ch nical journal describes how we implemented the standard and some of the techniques we applied to deal with change as the standard matured. Mark F. Kempf Of course, in Digital's business, the implemen­ Senior Consulting Engineet; tation of a technology is only a means to an end. Program Te chnical Director I AN FDDI The LAN technology must be incorporated into products that provide useful services to our cus­ tomers. One important rroduct in Digital's initial

In the early 1980s, Digital introduced local area net· FDDI offering, the DEC FDD!controller 700 work­ work (LAN) products based on industry-standard, station adarter, meets the need for more band­ 10-megabit/second Ethernet technology These prod­ width delivered to a single station. But !.AN users ucts allowed minicomputers. terminal servers, and have come to expect more: manageability, inter­ other network devices to be connected with ease op erability, ami a generally radial wiring scheme; and offered unprecedented bandwidth. As local and they have large existing !.AN infrastructures. networks grew in size ami began ro strain the capa­ Therefore, it was essential to introduce t-001 with bilities of a single LAN in the mid 19HOs, Digital a set of products that addressed all these needs introduced products based on the extended LAN as well. Digital's FDDI wiring concentrator, the concept. By connecting multiple IANs with fil­ DECconcentraror ';00, permits highly reliable, man­ tering and forwarding bridges, a much larger num­ ageable, radial wiring schemes. DECelms (extended ber of stations could be interconnected with greater LAN management software) provides network man­ aggregate bandwidth. agement capability for Digital's FDDI and preexist­ A few years after the introduction of extended ing extended TAN products. LANs, it became apparent that once again the capabil­ lt is important to remember that Digital does not ities of existing LANs were being strained, but this view FDDI as a replacement for Ethernet, but rather time in two dimensions. Not only was there need as its complement. The large number of existing for greater bandwidth in the "backbone" of the net­ Ethernet IANs can be connected to FDDI using work, but with the advent of much faster work­ another element in Digital's initial FDDI product stations and servers, there was need to bring more offering, the DECbridge ';00 FDDI/Etbernet bridge. than 10 megabits/second to a single station. The This product allows multiple Ethernetsto be inter­ current and projected performance of workstations connected using FDDI as a backbone. Ethernet sta­ indicated that any new generation of LAN suitable tions can communicate directly with stations on for future Digital products would have to offer an FDDI as well as with stations on other Etherncts. order of magnitude increase in bandwidth deliv­ The second group of articles in this issue describes ered to a single point. It was also clear that it would how we developed these FDDI products. have to adhere to a widely accepted industry stan­ During the entire development process, we relied dard, since users had come to expect the ability to on several design strategies. One emphasized the interconnect equipment from many vendors. importance of designing the entire system that After considerable analysis, Digital selected the comprises the product set, not just individual prod­ emerging ANSI FOOl (fiber distributed data inter­ ucts. Another was the decomposition of problems. face) 100-megabit/second token ring standard as Early in the design process, we identified func­ the basis for our next generation of LAN products. tions that appeared to be independent of others. The standard met our important requirements and We took into account how each component that showed promise of becoming widely accepted implemented these functions would be used in the

8 I

various products and how the products would be interconnected to form large reliable rings. This interact with each other. In our development plans, was the most important application of the heavily we assumed that once a particular independent instrumented test systems. Although analysis and component was designed, implemented, and tested, simulation can and did yield important results it could be depended on to work as expected when in this area, the complex interactions between other components were added around it. Naturally, large numbers of asynchronous stations over­ we could not assume that our initial assessment of whelm both analytical techniques and avai lable independence was infallible, and we remained computing power. alert for unexpected interactions. In the third phase, the focus was on producing Another related design strategy was partitioning products. From a logic design and software stand­ to reduce risk. As mentioned before, the FDDI stan­ point, Digital's FDDI products are largely derived dard was evolving and subject to change. There­ from the test systems used in the previous phases. fore, we physically partitioned our design to closely Of course, laboratory test capabi lities were removed mirror the separate sections of the standards docu­ and major changes in power and packaging were ment. This approach helped limit to a single compo­ made; but this approach significantly reduced both nent the impact of standards changes. the opportunity for introducing new errors and The final important design strategy can be sum­ the time to market. We were also able to use the marized as "analyze everything you can, simulate capabilities of the laboratory versions to help ver­ what you can't analyze, and prototype what you ify the correct function of the products. can't simulate." The work, expense, and recovery Numerous other activities that contributed to the time from errors increases with each step through effectiveness and timely delivery of the products this progression, and the advantages are obvious. were carried out simultaneously with engineering Of course, it is also obvious that this su mmary is design. For instance, Digital maintained a signifi­ simplistic since it is impossible to analyze, simulate, cant presence at the FDDI standards committee to and prototype exhaustively. It is necessary to step apprise the committee of various technical prob­ into the gray area of risk assessment and engineer­ lems we found in the standard, to offer solutions, ing judgment to make satisfactory progress. and to ensure that our implementation reflected FDDI LAN technology development and product the intent of the standard. In addition, the close development was done in three coordinated and working relationships fostered between various somewhat overlapping phases. In the first phase, organizations, especially between development and we concentrated on analysis and simulation of manufacturing, resu lted in products with a good the underlying FDDI algorithms specified by the balance between time to market, function, perfor­ standard . We wanted to ensure that they actually mance, and manufacturing cost. described mechanisms that would produce a reli­ The articles in this issue of the Digital Te chnical able, high-performance LA.t'l. During this phase we journal· go into much greater detail on the subjects also implemented the standard in silicon and soft­ I have touched on. I hope you find them interesting ware, and combined these components to fo rm com­ and informative. plete FDDI test stations. Since standards documents, like all written doCli­ ments, are subject to interpretation, two separate teams were given the task of implementation and verification. We used the test stations, first in a sim­ ulation environment, to execute test scripts devel­ oped directly from the standards documents to verify standards compliance. The scripts were also used to verify details specific ro our implemen­ tation and had the additional benefit of making regression test ing after incremental changes rela­ tively easy. In the second phase we concentrated on prov ing that significant numbers of FDDI stations, which had prev iously been shown to work individually, could

9 Wi lliam R. Ha we Richard Graham Peter C. Hayden

Fiber Distributed Data In teiface Overview

Aft er exploring mrious a!tenwlii'CS to sewnd-genemtion local area netu•orks (fANs), Digital selected lheji'!Jer distributed data inte;fa ce (/DDl) -� )'Stem. FDDI inzp!ements the In lerna! iona! Standards ();;r_;a11 i:ml ion (ISO) jJh)·simi lc()'er and the nzedia access control suiJ!ayer of tile data link layer Th is S)'Stem is bosed on a 100- megabit-per­ second jiber-optic ring netu·ork and uses r1 tinted-token protocol to coordinate station access to the netu·ork. D<�ilctf has det •e!oped the Ff)f)f base tecl:mo!og_); includ­ ing t•e;y !m:r;e-sm!e integration (\L\1) chips ond soji tmre. Th ese chtps, licensed to Adl'{{nced Jlicro De 1•ices mtd Jlotomla. Inc.. prr!l'ide high-qua/it)' a/ternatiues in the market and j(Jster cos! reduction Digital:\ inzp!ementation (<(fDDJ, including backbones in extended LASs. r1s ll 'el/ as /ligh-.1jJeedinterconnection of u•orkstations. serl 'ers. and cc 111m! cont[m ters. nwkes am ilaiJ!e a COIIljJfetc m ngc of S)'Sicm products

As the use of local area networks (LANs) continues It is important when developing a new LAN tech­ to grow at an exponential rate. many large networks nology to be sure the diffe rentiation from current with Ethernet backbones arc reaching their usable capabilities is sufficient to warrant the necessary capacity. In addition. the exp losi o n in the usc of investment. Moreover, a new LAN is a significant high-performance workstations is placi ng increas­ investment fo r a customer and should offe r a large ing demands on network performance as larger increase in capabilities such as speed and through­ volumes of data pass from station to station. Sev­ pur. Without this increase, the technology will eral years ago, Digital recognized this growth trend have a short I ifesp an (a few years) and technology and began to plan and develop a sccoml-gcneration such as parallel usc of existing IANs to double LAN that wo ulcl t(J llow Ethernetand provide an evolu­ capacity will be a realistic alternative to a whole­ tionary path to higher pcrt(mnancc. The selection sale rep l acement of the LAN. However, it is impor­ of FDDI as the second-generation LAN was made tant not to take such a large technological step that with great deliberation. This paper explores the exotic ami complex implementation constraints criteria fo r that choice and the history of till' FI )I) I become necessary. A LAN that does not lend itself system to the present. The theory of l'DDI opera­ to a ve ry large-scale integration (VLSI) logic solu­ tion, the development of the FDD! technology's tion will not integrate well in the computer inter­ role in Digital's networks, and the resulting prod­ connect environment and will not be cost effective ucts arc also presented and discussed. t(>r wide-scale usc. FDD!, with its tenfold increase in speed, provides Selection of FDDI significant diffe rentiation from Etbernet/H02.3 and Many of the same criteria origi nally used to select current token ring and bus technology to justify

Ethernet were again used to eva luate the app l i­ the new investment. Examination of the clocking, cation environmcnr t(lr the new LAN. The ncc(i to buffering, and state machine needs of the media consider migration from the popular I.ANs cur­ access control (MAC:) sublayer of the data link layer rently in usc presented the only new concern. also showed that the fDDl technology could be Paramount among the reasons t( >r selecting the implemented in several VLS! chip components. Fur­ FDDI technology were its tcnt(>ld increase in hand­ ther, as silicon technology improves, cost reduc­ width over Ethernet, its consistency with other tion is possible. enhancing the longevity of the IEEE H02 LANs. and the standardization effo rt FDDl LAN technology. already begun in the American National Standards Migration is another important factor in the Institute (ANSI) selection of a new Lf\N. Many devices exist with

10 I(,/. _; \u. J lj!l'ing !'}')/ Digital Te cb uical jo urnal Fiber Distributed Data In telj(tce Oi 'ert •ieu ·

embedded LAN interfaces that will never be Digital's VA..Xcluster systems.' The timed-token. media directly connected to any other IAN. Still other access control protocol itself was first publ icized devices will not benefit from the added capabil­ in 1982 by Bob Grow while he was at Burroughs.' ities of a higher speed interconnect. Examples of As a machine room interconnect between proces­ such devices include a processor or workstation too sors and storage systems, the initial ANSI standard slow to send or to receive data any faster from the requirements on the FOOl technology were quite network, an output-limited print server, or commu­ different from today's needs. In particular, as a nications servers with other, more constraining, machine room network, the number of stations was 1/0 ports. It would never prove cost effective to assumed to be relatively small compared to a LAN upgrade these devices to a new higher speed LAN and unstructured cabling was to be used. Since interface but, as a group, they would need to obtain most machines were always running, fault recovery unconstrained (no bottleneck) connectivity to the could be accomplished by having a dual ring with services of the new LAN fo r smooth migration fa ilover to the secondary ring. Thus, a fa iled station allowing protection of the investment in devices or cable could be isolated without partitioning the and LANs already in place. ring. The FDDI technology retains this property Standards are important fo r networks as a way to today. However, that basic operation capability is ensure consistent interface compatibility and inter­ insufficient in a IAN environment with structured operability fo r communications services. As the cabling requirements and a large number of sta­ IEEE 802.ld standard readily demonstrates by tions, any number of which might be unplugged or attempting to interconnect dissimilar LANs at the turned off by users. Therefore, we have expanded J'viAC sublayer, some standards are more compatible the definition of the FDDT technology to include than others. A common logical link control (LLC) such products as concentrators and adapters. fo rmat or the fo rmat within the MAC frame allows In 1982 the FOOl technology was brought to the a smooth migration between LANs by allowing a attention of the ANSI X3T9 committee, which devel­ transparent bridge to provide protocol-independent ops standards fo r l/0 interconnects and channels. translation between LANs. Ethernet, the forerunner Since FDDI was intended to be used as a machine of IEEE 802.3, does not use the IEEE 802.2 fo rmatted room interconnect, this committee was the appro­ LLC and, therefore, migration of those frames is priate arena fo r study. Over time, however, as the more challenging. need fo r a 100-megabit-per-second LAN emerged, Lastly, the media selected fo r the new LAN has the FDDI technology evolved into a local area to be consistent with current and projected future network. Some classic standards territory confl icts needs. Even fo r slower speed IANs, fiber-optic media developed between IEEE 802, the group that defines is gaining in popularity because of its superior all the LAN standards, and this Al'I SI committee. qualities in spanning greater distance, its noise While FOOl was evolving from a machine room immunity, and its declining user cost. interconnect into a general-purpose lAl'-J , the require­ The FDDI technology meets the necessary selec­ ments changed. For a machine room interconnect. tion criteria as an emerging American National some management operation to install ami initial­ Standards Institute (ANSI) standard using fiber and ize the network might reasonably be allowed. For allowing other media in place of fiber while pro­ example, the manager might set the values of vari­ viding a tenfold increase in speed. Migration of some ous parameters to control the operation and per­ devices could be affecteddi rectly by changing con­ fo rmance of the interconnect network. However. trollers, and bridging between LANs could allow in a general-purpose IAN, manager involvement is smooth migration of all existing devices. unacceptable. For simplicity, robustness, and ease of management, the industry widely accepts that L\Ns FDDI History must autoconfigure, also called "plug-and-play." Both the ANSI FDDJ standards ami the industry-wide Inevitably, FODI was required to exhibit the implementations of these standards have evolved attributes of a true local area network. Since the slow ly. A variety of factors have contributed to this FDDI technology and standards were already in course of development. 'T'he FOOl ring was originally development when this evolution of requirements invented at Sperry and Burroughs Corporation. The occurred, the ANSI committee made an attempt ring was to be used as a machine room interconnect to accommodate the fo llowing two views of the net­ between processors and storage systems, much like work: first, the network should be completely con­ the Computer Interconnect components are used in figurable with almost every parameter and pol icv

Digital Te ch nical jo urnal Vo l. j No. 2 Sp ring I')')I [ [ Fiber Distributed Data Interface

controlled by management; and second, the network ment interface to the PMD, PHY, and MAC layers.­ should be a local area network with the correspond­ To gether these components support an IEEE 802.2- ing attributes of simplicity and autoconfiguration. compatible logical link control capable of sup­ Incorporating both models into the ANSI FOOl porting client protocols such as the Digital standards made the standards complex and was one networking (DECnet) protocol, open systems inter­ fa ctor contributing to the eight-year- long time connection (OS!), local area transport (LAT), and period to completion. the transmission co ntrol protocol/internet proto­ col (TCP/11'). Stations utilizing the FODI components Th eory of Op eration can take several fo rms such as single attachment FDDI stations are composed of the basic elements stations (SASs), dual attachment stations (DASs), defined by the FOOl standards. The phys ical med ium and dual attachment concentrators (OACs). An dependent (PMD) layer specifies the fiber-optic inter­ architectural model is shown in Figure 1. face and data driver and receiver operation fo r FDDI stations.' The physical layer protocol (PHY) specifies Co nfigurations of FDDI Components the encoding and framing of data and control inJor­ A single attachment station is the simplest configu­ mation exchanged between stations.' The control ration and consists of the fundamental FDDI com­ info rmation exchanged varies with the physical ponents arranged as shown in Figure 2. There is a layer protocol type, which is either PHY-A, PHY-U, single incoming data path and a single outgo ing PHY-M, or PHY-S. The MAC sublayer specifies the data path with a MAC in between. protocols for logical ring formation and control, Dual at tachment stations, as shown in Figure 3, fo r the tr:�nsmission and recept ion of packets at a include a second physical layer and provide con­ station, and for the repetition and stripping of nections to a secondary ring fo r use in the event of packets on the ring.''· Station management (SMT) breakage on the primary ring. Under fau lt-free oper­ provides n-layer management and a local manage- ating conditions represented by the THRU STAT E

DATA LINK r------LAYER 1 I I I I I ------1 I I I I SMT I I PHYSICAL I I LAYER I I I I I I 1 ANSI FDDI ------'------

KEY:

DECNET NETWORKING PROTOCOL OSI OPEN SYSTEMS INTE RCONNECTION TCP/IP TRANSMISSION CONTROL PROTOCOUINTERNET PROTOCOL LAT LOCAL AREA TRANSPORT LLC LOGICAL LINK CONTROL MAC MEDIA ACCESS CONTROL PHY PHYSICAL PMD PHYSICAL MEDIUM DEPENDENT SMT STATION MANAG EMENT

Fi gure 1 Architectural Model

12 \lot. .i No 2 .\jJring 1991 Digital Te clmicaljourual Fiber Distributed Data lnterj(1 ce 0f!erf!iew

------, I LLC I MEDIA I ACCESS ______j CONTROL � ----1 + I I I • I I I I �I II I I I I I I I I SMT I PHY-A I I PHY-B I I I I I I I I I I I I I � � PRIMARYt SECONDARY SECONt DARY PRIMARY l ____ l IN OUT IN OUT

OUT IN THRU STATE

Figure 2 Single Attachment Station

MEDIA ACCESS area of Figure 3, the primary data path enters CONTROL through PHY-A, travels through the lVIAC, and exits through Pf-IY-R. The secondary data path enters + through PHY-B ancl ex its directly through PHY-A. If • I I a discontinuity is detected in the primary data path, II I ------1 I I either within the station or on one of the PHYs, the I PHY-A I PHY·B I I station wraps the two data paths, thereby provid­ I I ------1 ing an alternate route through the secondary data path. This situation is shown in the WRAP A STATE � PRIMARYt SECONDARY SECONDARYt PRIMARY area of Figure 3. IN OUT IN OUT

A dual attachment concentrator builds on the WRAP-A STATE dual attachment station by adding additional mas­ ter PHYs (PHY-M) in the primary data path as shown Figure 3 Dual AttaciJJnentStation States in Figure 4. Single attaclunent stations can then be included in the ring by connecting them to the additionai i'HY-Ms in the concentrator. As depicted in the upper portion of Figure 5, An FDDI LAN is formed by joining multiple sta­ the dual trunk ring is fo rmed by connecting dual tions to fo rm a logical ring topology. The logical attachment stations and concentrators to fo rm a ring can take two physical fo rms, a dual trunk ring LAN. This portion of the LAN consists of two data and a tree ring. paths in opposite directions, called the primary

MEDIA ACCESS CONTROL

I • • I • I I I I I I � I I I I ------I iI I I I I I I I I I I I PHY·A I I PHY-M I PHY·M I PHY-M I I PHY·B I I I I I I I I I I I I I ------I I I I 1 � � � t � � SECONDARY PRIMARYt t t PRIMARY SECONt DARY OUT IN OUT IN

Figure 4 Dual Attachment Concentrator

Digital Te cb 11ica/ jo ur11al Vo l. 3 No. 2 .vJring /'}'}I 1.3 Fiber Distributed Data Interface

DUAL TRUNK RING

r-:-T----r:':-1 TREE-ATTACHED CONCENTRATOR

SECON D-LEVEL TREE-ATTAC HED CONCENTRATOR KEY

A PHY TYPE A PHY TYPE B B M PHY TYPE M S PHY TYPE S DAS DUAL ATTACHMENT STATION DAC DUAL ATTACHMENT CONCENTRATOR SAS SINGLE ATTAC HMENT STATION

Figure 5 FJ)Df Dual Ring of Tr ees Top ology

and �econdary rings. Under normal operation. data station's requested token rotation time . When a flows on the primary ring from station to station. MAC recc::ives claim frames with times shorter than fn the event of a cable or a station breakage. the its own. or equal to its own hut from a station with stations adjacent to the fa ult join the primary and a n u mc:: ri ca lly larger address. it yields, stops send­ secondary rings ami then usc the secondary path ing claim.�. and repeats the claims received from to rcestabl ish a logical ring. its neighboring stat ion. Eventually the station with

A trt-c ring can be fo rmed by connecting stat ions the shortest or "winning" time will receive its own or concentrators to the l' IIY-.Vts of a concentrator claim. This station then ge nerates the ro.ken and as shown in figure ). Tn this fo rmation. the primary the ring enters the operational St:Ite. data path descends down each branch of the tree If rile claim process does not complete within passing through each station in the t ree . until it aprrox imately JOO m iII iseconds. the MACs in the fi nally reemerges into the dual trunk ring. ring pe rform a beacon process to confirm conti­ nui ty of the ring. Special beacon frames are trans­ Media Access Co ntrol Sublay er Op eration mitted continuously by all MACs until a beacon is

As me ntioned previously. the .Vtr\C sublayer provides received. at which point it stops transmitting. This the protocols fo r logical ring f( >rmation and data process continues until one :VlAC transmits and packet operations. To initial i1.e the ring. all ,\·1 .-\C:s receives its own beacon. indicating ring conti nu ity. first enter t he claim process to determ ine which 'lo transmit data packets. the MAC first wa its fo r .vt AC : wi II ge nerate the token and to cstabl ish the the token to arrive, holds it, and, then, transmits token rotation time f<>r the ring. Each station con­ the packets, rcissu i ng a new token at the end of ti nuouslv rransmits claim frames that contain the the transmitted packet stream. The time a llowance

1-l \(,/. l \·i1. 2 S!J rinu l'J'JI Dif!ilal Te cbuica/ To urnai Fib�tr Distribu ted Data ln telface Oueruiew

a station has to transmit packets after receiving very large delays on the ring while improving the a token is equal to the token rotation time estab­ efficiency on! y slightly.'' Therefore, after extensive lished by the claim process. performance modeling, Digital decided to use this

Packets received by a MACar e either repeated for default value for the requested token rotation time. reception by the next station or stripped from the The resulting network operates with low delay ring. In addition, a MAC may store a copy of a packet and high bandwidth as the default and does not for use by the station. After transmitting a frame, a need complicated network management proce­ MAC is responsible for stripping that frame from dures to achieve this level of operation. the ring after the frame makes exactly one traver­ Several unique but harmful conditions in FDDI sal of the ring. Frames left unstripped are consid­ stations must be addressed . An1ong them are the ered no-owner frames and can circu late the ring prevention of and protection against duplicate forever. This condition floods the station to which addresses, no-owner frames, and the stripping of the frame is addressed and is thus detrimental to ring frames sent by bridges or end stations that have

performance. A MAC typically strips frames by com­ multiple addresses.'" Because several of the algo­ paring the source address in the frame with the rithms fundamental to the operation of the FDDI

MAC's own address. The MACstrips any frame it has technology use the stations' addresses, the presence previously sent but repeats the frame, otherwise. of two stations with the same address causes numerous malfunctions, ranging from beacon-claim FDDI Ring Fo rmation oscillations to blocked communications between An FDDI ring is formed in several stages, beginning stations. Frames not properly stripped from the with the successfu l establishment of point-to-point ring can circulate forever, flooding the stations that links between all adjacent PHYs. These link con­ copy these frames. To protect against strip errors, nections are made by the connection management Digital's chip set has several built-in mechanisms. protocol (CMT) - This protocol defines control sig­ Digital also greatly improved the data integrity of nal exchanges to synchronize the two ends of the the ring. This improvement is particu larly impor­ link, to exchange info rmation about the PHY type tant to token ring architecture where messages traverse virtually all of the links in the network (i.e., A, B, M, S) and link testing requirements of each end to perform link quality testing, and to before arriving at their destinations. Robustness final ize the connection for normal operation. in the face of link bit errors becomes extremely Before the link establishment enters its final important. Digital designed several improvements phase, the PHY types of the two ends of the con­ to the basic FDDI algorithms, and the ANSI commit­ nection are compared, and the connection is tee adopted them to improve the undetected error allowed if the end types conform to specific con­ rates on a network -" nection rules. These rules are carefully established to ensure that rings are co nfigured correctly and Role of FDDI in Digital'sNe tworks to prevent misca bling, which could cause parti­ The FDDI technology is more than just another data tioned and unnecessarily wrapped rings. An estab­ link interface that al lows the use of transmit and lished link is co ntinually monitored for errors receive fibers between devices. Digital's decision indicated by the reception of improperly encoded to embark on the FDDI development effort was a

data and is shut down if those errors exceed a pre­ major program undertaking involving the develop­ determined threshold." ment of VLSI chips and, subsequently, FDDI software

After the PHYconnect ion is fully established. the and hardware products. A.!though the development station's MAC is inserted into the ring, and the of chips may seem to be at the heart of the FDDI claim process begins. Digital's stations al l use a program, chips are certainly not the products that default requested token rotation time of eight help customers solve problems. Chips comprise milliseconds in their claim process to ensure a only a small portion of each large printed circuit token rotation time, and hence traffic latency, board, but this port ion is an important one. similar to that experienced in other LANs like Before focusing on chip development, Digital Ethernet. FDDI provides high aggregate bandwidth carried out a large simulation effort to ensure the and, thus, delivers the low delay essent ial to many ANSI standan.lswere correct aml complete. Once the !.AN app.lications such as disk 1/0, interactive standards were verified, modeling was performed graphics, and remote procedure calls. Higher set­ to produce chips that met these standards. Al low­ tings of the requested token rotation time result in ances and trade-offs were made for unfinished

Digital Te e/ mica/ jo urnal V<>i.3 ,t>,lo. l .\j1rin,� t'J'JI 1'5 Fiber Distributed Data Interface

sections ami future standards migration. Real prod· Deployment of FDDI ucts were then planned around the chips as the usc As a baseline effort in deploying FDDI in products, FDDI of was threaded into Digital's network are hi· Digital developed the FOOl design corner shown tectures and existing products. An imrlementa· in Figure 6, consisting of chips ami FDDI control tion strategy f(Jr each product was then t FDDJ Design Comer

16 \i1! . . > \o .! \jN ·fn,� J'.!'JI DiJ.!,ilal Te ciJuical jou rnal Fi ber Distributed Data lnttt�jctce Oueruier£•

with the LANs being interconnected to fo rm can be created to best suit a customer's cabling, extended LANs by the use of bridges. The bridges network management, and availability needs.' ' are transparent to all users' protocols on the LANs. This capability fo r transparent interconnection is FDDlfor Wo rkstations one of the keys to the instant use of and easy migra­ Digital designed a simple high-performance FDDI tion to the added bandwidth and other benefits of adapter fo r TURBOchannel workstations and servers FDDI. However, providing the transparent inter­ in conjunction with the design of the OECstation connect is difficult. Frame fo rmats, bit ordering, 5000 reduced instruction set computer (RISC)-based padding fields, and even the length of the packets system.'' The aim of this product was to produce an diffe r between the Ethernet and IEEE 802.3 LANs. FDDI interface that was both inexpensive and high These differences fo rce the frames to be translated performance. With this adapter and ULTRIX oper­ as they pass through the DECbridge 500 device. ating system support, Digital attempted to address This translation, coupled with a stiff requirement many of the shortcomi ngs that have plagued most fo r data integrity fo r the MAC bridging standard IEEE adapters and operating systems attempting to 802.ld and the performance needs of dealing effi­ achieve high performance."· ciently with packet rates approaching 500,000 per The applications of the FDDJ teclmology fo r second, created a fo rmidable design task.'' workstations and servers depend on the particular choice of user applications and on the network con­ FDDJ Concentrator figuration. One application is the use of the FDDI To make the FDDI technology widely acceptable as technology fo r a work group of high-performance the next generation LAN, several of the logistical workstations and servers. Some high-performance obstacles inherent in a ring topology had to be I/O-intensive applications need more bandwidth overcome. Connecting new stations to a LANcan than a large, shared Ethernet can handle. As a result, be frequenr events. These events must not be a work group may fo rm around an FOOl network. restricted or cause a disruption if the LAN is to be One application example is called visualization, truly usable. The nature of a ring topology causes where a large volume of real-time graphics or imag­ the network to break whenever a station is removed ing information is transmitted over the network or while a station is being inserted. The dual ring from a compute server. Various applications of dig­ accommodates one such event by wrapping at the ital image transmission have fe w real-time latency - adjacent stations, but two or more such events arc requirements, but sri ll require a large banclwidth ' disastrous because they partition the network into Another application of the FOOl technology in isolated parts. A concentrator can solve this prob­ workstation environments is to achieve higher band­ lem by connect ing to a station, testing the link width fo r server/server and backbone interconnec­ between the station and the concentrator, and then tion. A shared Ethernet fo r client/server as well as spl icing the new station into the ring with mini­ server/server communications can be overloaded, mal disruption. At the same time, a concentrator particularly if the population of servers or work­ can automatically drop a station out of the ring stations is large. FODIcan be used as a backbone to should the station malfunction or lose power. provide higher bandwidth between the servers. In addition to the obstacle of connecting new sta­ Alternatively, the network may be a hybrid, where tions, it is inconvenient to map a ring topology into the FOOl network is used fo r high client/s erver star wiring, which is well known and widely used bandwidth by some systems and fo r high server/ fo r its convenience and manageability. Concentra­ server bandwidth by others. Hybrid networks often tors can turna logical ring into a hierarchical star­ have multiple FOOl networks and multiple Ethernet wired network, thus solving this mapping problem. networks. This configuration is typical of evolving Digital's OECconcentrator 500 product was built extended LANs, as new technology such as FDDI is to support simple installation and network configu­ added into the existing network infrastructure. ration, existing building cabling plants, and the cost savings and simplicity of single attachment stations. FDDJ in the Future This FOOl concentrator is low cost and flexible, As a new technology, FODI has a long fu ture with with a dual attachment and up to eight master opportunities fo r greater levels of circuit integra­ ports fo r connecting single attachment stations or tion, lower cost designs, ancl alternative media additional concentrators into the ring. With the types to match the wide range of possible applica­ concentrator, any of the described FDDI topologies tions. Significant progress has already been made

Digital Te chuica/ jourual Vu /. 3 Nn. 2 .\jJring 1991 17 Fiber Distributed Data Interface

in this direction with the introduction of the Draft Proposed American National Standard, DECcontrol ler 500 adapter, the most compact, least ANSI X3T9/90 -X3T9.5/84-49, REV 6.2 (May 1990) expensive, and one of the highest performing work­ 8. ). Hutchison, C. Baldwin, and B. Thompson, station adapters in the industry today. In addition, "Development of tlle fDDI Physical Layer," the ground-breaking technical work and standards Digital Te chnical journal, vol. 3, no. 2 (Spring committee activity regarding various types of cop­ 1991, this issue): 19-30. per med ia offe rs tremendous opportunities for cost 9. R.]ain, ''Performance Analysis of FOOl To ken Ring reduction which will enable broader utilization of Networks: Effe ct of Parameters and Guidelines the performance offered by the FOOl technology.'" fo r Setting TTRT," OEC-TR 655 (Maynard: Digita l In add ition to the original physical med ium Equipment Corporation, September 1989). dependent standard specified by ANSI fo r FOOl, an 10. H. Yang and K. Ramakrishnan, "Frame Content add itional standard has been developed fo r opera­ Independent Stripping fo r Token lli ngs," Pro­ tion on single-mode fi ber over distances up to 30 ceedings of the ACM SIGCOM '90 Sy mposium kilometers (km). This standard wi II be particularly (1990): 276-286. useful in deployi ng an FOOl network that must span one or more distances greater than 2 km such 11. R. Ja in, "Error Characteristics of Fiber Dis­ (FOOl)," as a campus backbone. Other medium types are tributed Data Interface tt:t:t: Trans­ currently under investigation to provide optimiza­ actions on Communications, vol. 38, no. 8 tion in areas such as cost and ease of installation. (August 1990). Acknowledgments 12. P Ciarfella, D. Benson, and D Sawye r, "AnOver­ view of the Common Node Software," Digital We wou ld like to thank the Te lecommunications Te chnical journal, vol. :3, no. 2 (Spring 1991 , and Networking lAN Architecture and Develop­ this issue): 42-52. ment teams. Without their hard work none of the 13. R. Kochem, ]. Hiscock, and B. Mayo , "Develop­ many technical advances made in bringing the FOOl ment of the DEC:bridge 500 Product." Digital technology to market would have been possible. Te chnicaljournal, voJ. 3, no. 2 (Spring 1991, We would also like to thank Bob Krueger fo r his rote this issue): 53-63 in successfully leading the entire program. Finally, we would like to thank Mark Kempf, Paul Koning, 14. W Tiffa ny, P Koning, and J. Kuenzel, "The and To ny Lauck fo r their timel y review of this paper. DECconcentrator 500 Product," Digital Te chnical journal, vol . 3, no. (Spring 1991, this issue) Riferences 2 64-75. 1. N. Kronenberg, H. Levy, W Strecker, and 15. U. Sinkewicz, C Chang, L. Palmer, C. Smelser, R. Merewood, "The VAXcluster Concept: An and F Te mplin, "lJLTRJX Fiber Distributed Data Overview of a Distributed System," Digital Interface Networking Subsystem Im plemen­ Technical journal, vo l. 1, no. 5 (September tation," Digital Te chnical journal, vo l. 3, no. 2 1987): 7-21. (Spring 1991, this issue) 85-93. 2. R. Grow, "A Timed To ken Protocol for Local 16. W Hawe and K. Ramakrishnan, "The Work­ AreaNetw orks," Proceedings of Electro/82 /UiL;. station on the Network: Performance Consider­ Conference, Boston, (May 25-27, 1982). MA ations fo r the Com munications Inte rface," 3. To ken Ring Physical l.c�yer, Medium Depen­ Proceedings of the Second Wo rkshop on Wb rk­ (PMD) , dent (International Standards Organi­ station Op erating Sy stems, Asilomar Confer­ zation, reference no. ISO 9314-3, 1990). ence Center, CA (September 1989). 4. To ken Ring Physical Layer Protocol, (Inter­ 17 K. Ramakrishnan and W Hawe, "Perfo rmance national Standards Organization, reference no. of an Extended Local Area Network fo r Image ISO 9314-1, 1989). Ap plications," Proceedings of the Fifth Annual 5. To ken Ring Media Access Control (MAC), (Inter­ Phoenix International Conference on Com­ national Standards Organ ization, reference no. puters and Communications, Scottsdale, AZ ISO 9314-2, 1989). (March 26-28, 1986). 6. H. Yang, B. Spi nney, and S. To wning, "FDOI Data 18. S. Ginzburg, W Mall ard, and D. Newman, "FDO! Link Development," Digital Te chnical journal, over Unshielded Tw isted Pa irs," JLEh. Proceed­ vol. 3, no. 2 (Spring 1991, this issue): 31 -41. ings, Eighteenth Co nference on Local Computer FDDI (�MT) 7 Sta tion Management Preliminary Ne tworks (October 1990)

!')<)/ 18 Vu l 3 ,Vo. 2 .\p•·in,� Digital Te cllllical jo urnal jerry D. Hutchison Christopher Baldwin Bruce W. Thompson

Development of the FDDI Physical Layer

The engineering development of the FDDI physical layer resulted in the delivery of co1nponents, sp ecifications, and protocols. The development presented new design problems related to the technology and to the op eration of token rings. The choice of the most app ropriate technologies fo r the chip set was based on technology issues, risk control, and costs. The chip set that emerged rifl er the physical layerfu nctions were partitioned uses both r:; a and CMOS technology Furthe1; three design problems of general interest arose during development: the elastici�)' buffe r and circuitry related to the distributed clocks in an FDDI LAN, the multimode fi ber-optic fink using light emitting diodes, and the media error processes as related to correctness cmd fa ult isola lion.

The fi ber distributed data interface (FDDI) is a multi­ Op eration of the Physical Layer access. packet-switching local area network (LAN) The FOOl physical layer is a collection of point-to­ that operates at 100 megabits (.Mb)per second. The point links forming a ''ring." The operation of the physical layer of fOOl-the topic of this paper­ layer is described in terms of physical links, phys­ connects many stations, each of which may trans­ ical connections, and the fu nctions of individual mit information w any other station in the network. stations. The many station types allowed by the As in other IANs, packets of user data are encoded ANSI FDOI standards are constructed with a simple according to the physical layer protocol and are physical layer functional block callecl the PHYport. transmitted as a serial data stream over a physical A physical link contains a transmitter, a receiver, media to other stations of the LAN. FDDI , however, and a segment of physical medium which conducts is unique in its use of hundreds of individual, point­ the bits of a packet from one station to a second to-point, fiber-optic connections that form a ring station. The topology of FDDI is arranged so that the network in the physical layer. The resulting LAN collection of physical links fo rms a closed path, or offers both a high data rate and a total physical ring, as shown in Figure 1. This simple topology extent of up to 100 kilometers (km). illustrates the basic concepts common to even the The development of physical layer hardware, used most complicated topologies fo r FODI. Each bit of in all FDDI products, included the physical proto­ info rmation received from one physical link is trans­ col (encoding/decoding) device, a receive clock mitted onto another physical link until the informa­ recovery device, a local clock generator, and optical tion travels around the loop and returns to where transmitters and receivers. This paper fo cuses on it started. The FDDI protocol.s provide fo r a single development of the physical layer hardware and originator of data packets; other stations repeat the describes some aspects of the design in detail. We data so that each station on the ring receives the first review the operation of the physical layer and packet of information. The collection of many point­ the functional partitioning of the implementation. to-point links fo rms the ring, which is viewed as a We then present detailed discussions of the dis­ multiaccess medium by the users. tributed clock scheme, the design of an optical link, The basic element in the topology of an FDDI LAN and the methods to control the eftects of bit errors is the physical connection. A physical connection in the physical layer. Some of the results of the devel­ contains two physical links, also shown in Figure I. opment to improve the performance, correctness, Within the station, the circuitry that implements and reliability of FDDJ described here have been physical layer functionality needed fo r one physi­ incorporated in the American National Standards cal connection is called the PHY port. The physical Institute (ANSI) FDDJ standards. connection is a fu ll duplex connection between

Digital Te cbuical jounwl Vo l ..i No. .! .�j1ri11� /'}'}/ 19 Fiber Distributed Data Interface

exactly two PHY ports. Neighbors in the ring symbol inro a code group containing 5 code bits. directly exchange the control information fo r each This encoding limits the maximum time between connection, allowing conrrol protocols in FDDI transitions on the media (allowing clock info rma­ station management (SMT) to establish the shared tion to be derived from the coded signal) and bounds states fo r a connection: in-use, starting, and dis­ the low-frequency components of the signal spec­ connected. The status "in-use" ind icates that a con­ trum. The code bits are then converted to a serial nection is part of the ring; other states ind icate it stream and transmitted as optical pulses on the is not. The control information exchanged over the fiber-optic media. physical connection is used to autoinitialize and The station is coupled to the media with a media autoconfigure the connections in the LAI"J , a method interface connector (MIC). The MIC provides the of operation currently unique to FDDI rings. concrete interface necessary fo r interoperability between equipment from multiple vendors. The

PHYSICAL FDDI Physical Layer Media Dependent (PMD) stan­ CONNECT ION dard specifies mechanical and optical properties of I I SAS the MIC.' The MIC includes both a transmit and a 1 I I lr I CON receive interface. ht,- ' Signals received from a connection are decoded I I by the PHY port fo r processing by the station. The I SAS 2 I optical input signal is translated to an electrical sig­ I I I nal. The remote bit clock is extracted from the signal I I and used to recover logic levels correspond ing to I J I SAS 3 I I I J TRANSMIT DATA RECEIVED DATA

I ------r---- I PHY PORT SAS 4 ' \ I I _j_-\ ENT ITY I I - I I \ ENCODER DECODER eHYS,CAL UNK I I I Figure I Physical Links Nmning a RingjiJr DATA the FDDI Physical Medium FRAMER LOCAL CLOCK REFERENCE L-_. ELASTICITY BUFFER There are several types of FDDI stations, and ditferent types can support different numbers of RECOVERED CLOCK DATA physical connections.' A single attachment station I I (SAS) (as seen in Figure I) can establ ish one phys­ OPTICAL C LOCK AND ical connection with a single neighbor. The dual TRANSMITTER DATA RECOVERY attachment station (not shown) has two PHY ports and may establish physical connections with two neighbors. A concentrator (CON) is a type of station that can establish connections with many neighbors, OPTICAL thereby providing attachment points for other sta­ RECEIVER tions. The CON shown in Figure 1 interconnects its PHY ports internally to configure a single ring. t _j Figure 2 shows the fu nctions of and flow of data ------+ through a PHY port which implements the FDDI I MEDIUM MEDIA INTERFACE physical layer protocol (PHY) standard -' Data pack­ � CONNECTOR ets to be transmitted over the LANar e passed as a (BULKHEAD OF STATION) stream of bytes from the data link to the physical layer. Each byte contains two PHY symbols, and each symbol represents 4 bits of user data. The FDDI cod­ Figure 2 Functional Model of the PhysicalLa yer ing scheme, called 4B/5B encoding, translates each fo r an FDDI Station

Vo l No. 2 20 . .> !Sp ring 1991 Digital Te chnical jo unwl Developrnent of the FDDI PhysicalLc �yer

the individual bits. A framer then establishes the and is relatively expensive as compared to CMOS original code group boundaries and converts the technology. We also considered ECL gate array tech­ serial code bit stream into parallel fo rm. Also, the nology, but decided against it because it was not elasticity buffe r synchronizes the received data to a mature technology, lacked requisite analog func­ the local clock reference and accounts fo r the fre­ tions fo r clock extraction, and was available from quency difference between the local and remote only a few vendors. clock references. Finally, code groups of 5 code We determined that the CDCR and CDCT were bits arc decoded into symbols, and symbols are cor­ the only fu nctions that had to be implemented in rectly paired to fo rm the data bytes which repre­ ECL technology. This determination was based on sent the received data. These data bytes are passed the need fo r the high transmission rates and fo r either to another PHYport or to the data link layer. quick conversion to and from serial and parallel A later section, Operation of the Distributed Clock data streams. The CDCR receives a 125-megabaud Scheme, expands on the elasticity buffe r design. ECL serial data stream from the FOR. Using a phase We have so far described the FDDl physical layer lock loop, CDCR extracts a receive clock to recover in terms of PHY ports and the physical connections the data bits and then converts the serial data to between them. These basic elements fo rm the a 5-bit parallel bus. The CDCT receives a 25-MHz, physical layer fo r all types of FDDI stations. 5-bit-wide parallel bus; then, by using a phase lock Different types of FDDI stations have one or many loop, it generates an internal 125-MHz transmit PHY ports, but the operation of an individual PHY clock in phase with the local 25-MHz clock. CDCT port and physical connection is independent of sta­ then converts the 5-bit-wide parallel bus to a 125- tion type and topology. In the next section, we dis­ megabaud ECL serial bit stream that is transmitted cuss the functional partitioning of the PHY port by the FOT. and the reasons behind the partitioning chosen. We selected the 5-bit width fo r the parallel bus to Subsequent sections describe the distributed clock obtain a 25-MHz bus rate. This rate is a convenient scheme, the design of the physical link, and the divisor of the I25-MHz serial rate and is within the impact of physical link errors on the LAN. operating range of the CMOS gate array technology used in the connected chips. The 5-bit bus also Fu nctional Partitioning offered the advantage of enabling us to maintain a low pin count on the devices to which the bus is In this section, we describe the partitioning of interfaced, thus further containing costs. the fu nctions of the PHY port into the fo llowing Another complication relative to the clock com­ components: ponent was how to distribute a 125-MHz clock sig­

• PHY (physical protocol chip also referred to as nal. As noted earlier, some FDDI products have many the ELM chip) PHY ports, and those PHY ports must have a com­ mon clock line fo r the transmission of the serial CDCT (clock and data conversion transmitter) • data stream. We decided to add another phase lock loop in the transmit component that would lock • CDCR (clock and data conversion receiver) onto the 25-MHz local clock, generate the 125-MHz • FOT (fiber-optic transmitter) serial clock, and convert the 5-bit parallel bus to the serial stream. With this method , the highest clock • FOR (fiber-optic receiver) rate distributed on our boards was a 25-MHz clock. Our choices fo r the appropriate partitiOning As a consequence of selecting the transmit phase and technology were founded on our decision to lock loop, we chose to specify and build separate develop a highly integrated and low-cost chip set. transmit (C:DCT) and receive (CDCR) devices in cus­ After examining several alternatives, we chose a tom EC:L technology. We were very concerned that partitioning that enabled us to use mostly CMOS the combination of two asynchronous phase lock technology (complementary metal oxide semicon­ loops on a single chip would induce cross talk. ductor), a minimal amount of custom ECL (emitter Cross talk could cause fa lse locking of the phase coupled logic), and no ECL gate array technology. lock loops to one another, resulting in lost data. Although a 125-megahertz (MHz) serial channel Therefore making a single chip was considered too requires ECL circuitry in the system, we wanted risky fo r the initial implementation. Our solution to minimize the amount of custom ECL technol­ was to specify the transmit and receive devices, ogy. ECL consumes a substantial amount of current thus elim inating the possibi lity of cross talk.

Digital Te chnical jou rnal Vo l. 3 No. 2 Sp ring 1991 21 Fiber Distributed Data Interface

The balance of the logic for the physical layer result in an unacceptable packet loss rate must be

protocols could now be designed in CMOS gate array controlled. In the sections Elasticity Buffe r and technology. The use of CMOS gate arrays was impor­ Smoother below, we describe how these problems tant in meeting schedule since it al lowed us to are addressed in the physical layer protocol. quickJy implement changes. Changes were inevitable and therefore had to be accommodated because Elasticity Buffe r the ANSI standard was not finished and stable dur­ Each PHY port of a station must accept data packets ing our design cycle. from another station with a slightly different clock All of the physical layer fu nctions such as the frequency and bit transmission rate. It is the func­ encoder, decoder, elasticity buffe r, fr amer, and tion of the elasticity buffe r within the PHY port to smoothe r were imp.lemented in CMOS gate arrays. synchronize the incoming data to the loca l clock We had simulated these functions in software; reference. The buffer is also designed to control however, we were now able to build them using synchronizer metastability, a source of undetected CMOS gate arrays and actually analyze their behavior data corruption. As a result of the elasticity buffer in rea l networks. With the hardware, we quickly operation, the size of the gap between two data FDDI verified the protocols defi ned in the standards. packets varies as the packets are repeated around Proper PHY operation is best confirmed by testing the ring. actual implementations. The elasticity buffe r is a collection of storage (FOT) The fiber-optic transmitter converts a 125- registers that are written to and read from at differ­ megabaud electrical signal to light pulses to be ent rates. (See Figure 3.) The buffe r fo rms a circular transmitted to a receiving station. The fiber-optic queue due to the movement of two independent (FOR) receiver receives the pulses from a transmit­ pointers: the input pointer selects the register to ting station and converts them to an electrical data be written to and moves at the recovered clock stream. We decided not to develop the FOT and the rate; the output pointer selects the register to read FOR components ourselves. Instead we chose to from and moves at the local clock rate. The location influence the specification of the system's func­ of the pointers is based on the gray code counters. FDD! tional requirements in the ANSI Committee The input pointer is controlled by the input state and then depend on external vendors to develop machine and the output pointer by the output the components. state machine. These state machines position the It was important to encourage the optical vendors pointers at a control led distance from one another. to standardize their components so costs wou ld Therefore pointer control prevents data from decrease, and so that more than one source of opti­ being written while it is being read from the same: cal components would be available to us. Accord­ register, even though the pointers are moving at ingly we did not combine the optical transmitter diffe rent rates. and receiver with any other physical layer functions. During normal operation, the input and output The optical link design is the subject of a later sec­ pointers approach each other and must be periodi­

tion in this paper. C:! II)' repositioned. The reposi tioning occurs dur­ ing the idle time between data packets, known as Op eration of the Distributed Clock the interpacket gap. Wh en a minimum interpacket Scheme gap time is detected by the input state machine, In the FDDI distributed clocking scheme, each sta­ a reset control signal is sent to the output state tion uses an independent, local clock reference machine. The reset signal is synchronized by the

when transmitting or repeating data packets. The output state machine to avoid metastability. \Vhen station must synchronize the receive data with its an input signal to a register is changing at the time own reference clock prior to fu rther processing. the register is clocked, its output may become inde­ Al though this distributed clock reference scheme terminant anu assume multiple values over a time

simplifies many problems, it also can give rise to data called a period of metastability.' The reset signal integrity problems and packet loss rate issues that could be changing when it is sampled by the output must be solved fo r the scheme to work effectively. state machine, so this signal is synchronized by Data must be synchronized to the local clock wa iting an interva l after each sample fo r the sample reference in a way that prevents detectnl and value to settle before the sampled value is utilized, undetected errors caused by metastabi I ity prob­ or considerc.:d valid. The reset signal is delayed lems. Further, interpacket gap shrinkage that can by this process. The circuitry guarantees that the

I')'JI 22 �1,/. 3 So. .! ,\jJ ri11g Digilal Te cb uical jour11al Deuelopment of tbe FDDJ Physical Layer

present address of the input pointer has been in register fo r more than a minimum amount of time.' the holding register on the reset condition for a This circuitry prevents the bufferfrom reading the sufficient duration and thus its stabi lity is ensured. register while its contents are changing; if the buf­ The output state machine then loads the address ferwe re read, data corruption and consequently that the input state machine stored in the holding undetected errors could result during abnormal register. The output pointer moves to that location operation. The input and output pointers are 3-bit, pl us an offset in order ro keep a minimum distance asynchronous gray code counters. The pointers are from the input pointer. compared to one another to determine whether This approach to repositioning pointers ensures they coincide, indicating that data in the buffe r has

that all data or signals are stable in their respective overflowed or underrun. registers before being sampled by the local clock. The input and output pointer counters are com­ As a consequence, we were able to specify in the pared using the local clock; that is, the input pointer design that only one control line be synchronized counter bits can change with respect to the local in the elasticity buffe r. The signal that crosses the clock, as shown in Figure 4. Gray code counters clock boundaries is the reset signal, which is gener­ limit to one the number of bits that can change in ated by the remote clock and sampled by the local the pointer counters at any sample interval. The clock. The reset signal triggers all events required comparator circuit is sampled twice using the local fo r the elasticity buffe r to correctly receive data. clock and the local clock shifted by 90 degrees. Since there is only one control line within the elas­ If one sample of the D fl ipflop notes a change, the ticity buffe r that needs synchronization, the imple­ changing bit settles down before the other sample mentation is very robust. happens; thus metastability problems are controlled. We also had to anticipate occurrences outside A logical AND of the sampled outputs signifies that normal operation. ·rherefore we designed circuitry the two pointers have had the same address for at that detects when the pointers point to the same least one quarter of the local clock interval. When

INPUT STATE MACHINE REMOTE LOCAL OUTPUT STATE MACHINE OPERATES USING CLOCK CLOCK OPERATES USING REMOTE CLOCK BOUNDARY BOUNDARY LOCAL CLOCK I I I I I I COUNTER LOAD - POINTER VALUE

RESET - CONTROL I I � FUNCTION RESET CONTROL SIGNAL t I I I I SYNCHRONIZER I I I I I I INPUT (WRITE) OVERFLOW! ERROR POINTER UNDERFLOW I INDICATION DETECT ,... t' t\ I \ I I GRAY CODE I II II I I I I f-- I I I I COUNTER I I I I GRAY CODE I I I I I I 1- COUNTER 1- I I I I - INPUT DATA BUS :I .I I I OUTPUT (READ) POINTER I I I I I I I tI I I I +- I OUTPUT DATA BUS I I I I I I I I I I I I I I \/

STORAGE REGISTERS

l·i��ure 3 Po inter Control in the t"lasticity Buffe r

Digital YeciJuical jo urnal Vu /. _; Nu. .! .)/Jrill[!. /'}'}! 23 Fiber Distributed Data Interface

UNDERRUN/ OVERFLOW

Figure 4 Oueljlow/Underf!ow Detection in the Elasticity Buffe r

the addresses are the same fo r at least one quarter The solution is to monitor the interpacket gap." of the local clock time, the error flag is raised. When If it fa lls below the 7-byte minimum size, the the error flag is raised, an overflow or underrun is smoother adds to the interpacket gap. The addition imminent. The cause is somewhere in the network. of interpacket gap to the output stream causes the For instance, the clocks are out of specification, or elasticity buffe r to use a buffe r to delay the output a downstream station is sending larger data pack­ data and then send an interpacket gap byte. The ers than are permi tted. In any case, the condition is amount of buffe ring within the elasticity buffe r is detected and prevented from increasing the risk of fi nite so that the delay wi thi n a station is not long. undetected data corruption. The smoother also reclaims storage elements by deleting bytes of interpacket gap one byte at a time Smoother from long preambles. This reclamation occurs any Another fu nction in the physical layer. cal led the time 8 byres or more of interpacket gap appear at smoother, prevents the loss of clarapackers that can a station. resu lt from shri nkage of the interpacker gap. Data The smoother is a distributed algorithm that can­ packets can be lost, or discarded, at two places. The nor be adequately proven in simulation. We buil.t a decision to discard can he mack at the physical gate array that contained the elasticity buffe r with layer as the result of an elasticity buffer overflow the added smoother function to prove in hardware or underflow Also, packets can be discarded at the that our new algorithm would operate properly. (MAC) JVIAC media access control layer. The layer is We built a 200-node ring of elasticity buffe rs that not required to copy a packet that has Jess than 6 could handle at least the test case (101 elasticity buf­ idle bytes of interpacker gap preceding the packet. fers) used in the simulation. Each elasticity buffe r The decision to implement the smoother came had a variable osci IIa tor to allow control of the dis­ after simulations of the elasticity buffe r revealed tribution of clock frequencies that we fe lt would ANSI PHY the then current draft protocol wou ld induce interpacket gap shrinkage. We also included n:sulr in an unacceptable packet loss rate. In a series special test featuresin the chip ro monitor the inter­ of nodes with a random distribution of clocks, some packet gap at every node in the rester. stations add to and others delete from the inter­ During fo ur weeks of ring operation, we observed packet gap. If nodes add or delete without regard the interpacket gap of 6.72 billion maximum size to the size of the interpacket gap, we fo und that packets ( 4500 bytes). The m inimum interpacket gap the interpacket gap could be de.leted entirely or observed was 7 bytes, which resulted in no packer MAC reduced to a minimum size. At this size, the is loss. The experiment indicated a packet loss rate of not required to copy a frame, and data packers are less than 2E-10. Since no packer loss occurred, the lost. Our simulation showed that an unacceptable actual loss rare is unknown; but this result gives us 10 percent packet loss would occur clueto 6 or less cortfidence that the loss rare predictions made by byres of interpacket gap under these conditions: analysis are correct. The 200-node hardware test bed demonstrated • One hundred one elasticity buffe rs that our algorithm worked effectively The smoother Maximum size data packer length • protocol was adopted as a mandatory parr of the

• Pseudorandom clock distribution final ANSI FDDI I'HYstandar d.' The standard allows

24 HJ I. j No. 2 .\jJring /'J'JI Digital Te chuical jo unwl Development of tbe FDDI Physiccd Layer

a variety of different designs, some having as little between single-mode and mult imode fiber opera­ as one byte of smoothing, depend ing on the num­ tion. FDDI development initially fo cused on the ber of preamble bytes required by the MAC imple­ transmission distance requirements of LANswhic h mentation in the same station. All allowed designs serve as local office networks as well as network have worst-case loss rates below l E-14 (by analysis) backbones. Accordingly the technology chosen fo r in a homogeneous ring. The worst-case packet Joss the optical link should be cost-effective and capa­ rate in a heterogeneous ring, one with multiple ble of spanning approximately 2-km distances. For types of smoother designs, has a packet loss rate these applications, the superior bandwidth and below lE-10 (by analysis). Given the addition of loss characteristics of 1300-nm LED systems prevai I the smoother to the PHY protocol, the FDDI dis­ over 850-nm LED technology; 125-megabaud trans­ tributed clocking scheme does not significantly mission over 2 km is not possible with 850-nm contribute to packet loss. LEOs. LEOs and multimode fiber prevailed over lasers operating with single-mode fiber because at the FDDI opticalLi nk Design design time the former was more rei iable and had a As noted in the earlier section, Operation of the better chance of achieving the cost goals required Physical Layer, the physical connection is the basic by short-distance office imerconnection spans. The FOOl element in the LAN topology. The physical selection of appropriate technology was especially connections between FODI stations are fu ll duplex, difficult because the technology was rapidly evolv­ fiber-optic links that deliver a serial code bit ing. The Wo rking Group made the basic technology stream from one station to another with a bit error choices in the 1984-1986 time frame; the chosen tech­ rate (BER) less than 2.5E-JO. Each station has a sep­ nology represented the best compromise between arate transmit and receive link, and both links are available technology and reasonable anticipated cabled together to the same destination. The opti­ improvements. The FDDI committee later addressed cal link requirements are defined and measured at the long-distance requirements (greater than 2 km) MIC. FOOl the Any set of conforming stations con­ of a campus LAN with a single-mode fiber and laser nected together with a compliant cable plant in transmitter PMO (SMF-PMD). That development effort a legal topology are guaranteed to provide the is not addressed in this paper. required transmission service. Conformance to the optical requirements can be measured indepen­ Op tical Link Overview c.lently of both the interconnecting media and the The optical link is composed of three basic elements: attached station. Measurements can be taken from a transmitter, a cable plant, and a receiver. The either end of a physical connection. transmitter is provided with a serial 125-megabaud The technology choices we confronted and code bit stream and creates an amplitude modu­ the design methods we used in the development lated 1300-nm optical version of the bit stream. of the optical link are summarized in the fo llow­ The code bit stream has previously been encoded ing sections. These methods can be applied to any with a 4-bit into '5-bit (4B/5B) non return to zero transmission system design problem with similar invert (NRZI) coding scheme that ensures that the requirements. The Physical Layer Medium Depen­ serial sequence has sufficient transitions to allow dent (PMO) Wo rking Group of the FDOI committee recovery of the transmit station's timing clock at adopted these methods, and Digital played a lead­ the distal end of the link. The cable plant uses a ing role in the design of the PMD Standard. The glass, graded-index, multimode optical wave guide Wo rking Group developed the design in a manner to ferry the signal to the receiver; the cable has an that combined theoretical analysis with empirical arbitrary number of junctions (e.g., connectors). modifications in an iterative process to arrive at The cable plant is described by its optical loss and the specifications for the system. The fu ll detail bandwidth. The receiver in turn converts the opti­ of the models has been documented previously in cal signal back into a logic-leve l code bit stream. the literature.-" When a station is not sending data, the transmitter is provided with special code bit sequences which Te chnology Choices ensure that there is always an optical signal on the The FDDI distance and bit rate requirements clearly med ium between packet transmissions. Thus, when­ mandate the use of a fiber-opt ic transmission sys­ ever the link is a part of a ring, the optical system tem. However, the choices are not equally obvious stays in its equilibrium operating conditions, and between laser- or LED-based transmitters, between the clock recovery circuit is always synchronized 8'50-nanometer (nm) and 1300-nm operation, and with the incoming stream.

Digital Te chllical jo unwl Vo l. .i N(). 2 Sp ring J9'JI 25 Fiber Distributed Data Interface

Design Jl-'!ethods central wavelength from the zero dispersion wave­ The design of any digital transmission system must length of the fiber. The wide spectral width of

provide su fficient end-ro-end bandwidth and si g­ 1:)00-nm LEDs is su ffi cient to cause systems based nal power. Further, the design must demonstrate on their use to be distance limited by chromatic

bounded jitter characteristics in order to provide dispersion, even tho ugh the s ystem is operating at the requin:d data transfer at the desired HER. The 1300 nm, which is the nominal zero dispersion

bandwidth allocation, jitter budget. and loss bud­ window of fi ber. A model was deve loped and veri­ FDDI get fo r the optical system are described next. fied fo r the chromatic bandwidth: the equation fo r the model is 4 in Figure '). Hand�l'idth Affocatioll and ,Hodels Nyquist com­ Equations 1 through 4 are the complete model mu n i cati ons theory requires a system bandwidth fo r the bandwidth of the FDDI optical system. The of at least one half the baud rate to prevent error inputs to the model are the transmitter spectral rate degradation due to intersymbol interference. center wavelength, spectral width, transmitter rise Practical systems require a somewhat greater band­ and fa ll t i mes , the i ber length, the fibe r modal width. We determined the LED-fiber bandwidth fo r f bandw idth , the fiber's zero dispersion wavelength I'DDI by measuring the sensi tivity of commercial (Ao), and the zero dispersion slope (.)�,). These 12) -megabaud optical receivers as a fu ncti on of parameters completely defi ne the constituents of i ncreas i ng input rise time (d ecreasing ba ndwidth) and by observing when the chann el bandwidth started cause a p nalt y in the m asu red receiver - J to e e nw�r.� (1) BEl{ perfo rm ance. The 0.'5decibel (dB) optica l power penalty poi was fo und at 9') :v1 11z. That p i n t is nt o Where: the bandwidth req uirement t< H· the LED and fi ber = total optical system bandwidth (MHz) combination in a worst-case max imum length link: B Wsrs = elec rical bandwidth of LED (M Hz) lower bamhvidth causes i n creasingly hi gher pena l­ R Wele r

= fiber modal bandwidth (MHz) ties in UEH performance and must be prevented . B W11wrt

The bandwidth of an LEO and multimode fiber = ch ma ti c bandwidth of ED- be (MHz) fJ Wcln ro L fi r optical system is modeled with three components which add in a root mean sq uare (fuvlS) fa shion as 470 s how n in equation in Figun: . The design prob­ MHz (2) I ) nwele T,.f lem confronted is as fo llows: how are the three dit� fcrent bandwidth componen ts rationally allocated --NIB W MHz (3) /J Wmod { to meet the 9) -:VlHz LED-fi ber requireme nt. and what is the maximum distance that can be achieved and 37'5 still meet this req ui rement' Al though t he electrical MHz and modal bandwidth lim itations are well known

(equations 2 and 3 in hgure ')). the chromatic band­ widrh lim i tation caused by the I.ED and fiber com­ (4) hi nation was not wel l understood. :x,.i Chromatic bandwidth limitation is caused by ·(( :X.. .-7)- () ) the interaction of the LED spectral width with the ( 5 <\--7) wavelengtl1 dispersion of the glass fiber. Thirteen hundred-nm I.EDs arc not mo nochroma tic: their Where em ission spectrum is typica lly 170 -nm wide at the = LED rise or fall time (ns) ha lf optical power poi nt. The p rop gat ion elocit y T,1 a v = modal bandwidth d istance product of the MIJ W of ght in glass is a fu nction of the wavelength of li fiber (M Hz · km) the light: light of diffe rent wavelengths experi­ I = the fi ber length (km) ences diffe rential delay or dispersion. Accord ingly � :X.. = O.H) · Full Width Half Max im u m (run) LED 2 a signal of appreciable optical spectral width expe­ S0 = the fiber d ispersi on slope (ns/nm · ktn) riences dispe rs i on that causes an increase in the :X.. c = the LED central wavelength (nm) signal transition times and limits the bandwidth. :\0 = the fiber zero dispersion wavelength (nm) The amount of dispersion experienced by a pulse is a hmction of the length of the fiber. of the optical spectral width. and of the separation of the pulse Figure 5 IDDI Op tical Link Handtl'idth Model

I ;,r .i ,Vo. ..! .\jiring I')'J/ Digital Te ch nical jo unw/ Det •elopment oj the FDDI Physical Layer

the bandwidth of a multi mode fiber-optic transmis­ DCD is static and is caused by switching thresh­ sion system. In an iterative sequence of calculations old variation and mismatched rise and fa ll times in with the model, we evaluated a trade-off of fiber driver circuits. DDJ is caused by bandwidth 1 imi­ length, fiber modal bandwidth, LED chromatic attri­ tations in transmission components and is also a butes, and LED rise and fall times to arrive at a 2-km function of the transmitted cocle bit stream. We maximum fiber length with transmitter chromatic developed a worst-case test pattern that evinces high­ and temporal requirements that could reasonably frequency DDJ components caused by local run be met by vendors. The transmitter requirements length variations in the transmitted bit stream and were described by a series of curves that balanced low-frequency DDJ components caused by varia­ transmitter rise and fa ll times and chromatic tions in the average power of the unbalanced 4B/5B attributes. These requirements guarantee the 9'5-MHz code bit stream. RJ is caused primarily by thermal LED-fiber bandwidth requirement for a 2-km fiber. noise corrupting the signal in receivers and is appar­ Thus transmitters are allowed slow rise times if they ent at low optical powers. RJ adds in a root-mean­ have narrow spectral widths or central wavelengths square fashion with other RJ components; DDJ and that match the minimum dispersion wavelength of DCD add linearly to 1\J . the fiber. Transmitters with wider spectral widths The FDDI jitter budget tracks these three com­ and central wavelengths displaced from the zero ponents of jitter through the optical link. The bud­ dispersion wavelength have fast rise time require­ get ensures a sufficient allocation fo r the clock ments. The curves in ANSI FDDI PMD Figure 9 show recove ry implementation to place the clock cor­ the final allowed transmitter spectral and temporal rectly in the jitter-free window to retime the data. trade-offs 5 They were generated with a slight modifi­ The specific values of jitter allotted to each link cation to the basic model described above that used element were determined largely by empirical explicit fast Fourier transform (FFT) descriptions of methods. The sum of all jitter allocations must not the LED electrical bandwidth component. The trans­ exceed the code bit width [8 nanoseconds (ns)]. mitter requirements depend on the fiber meeting Table I summarizes the jitter budget, showing the modal bandwidth and chromatic dispersion specifi­ totals for each jitter component as it adds through cations. We empirically established the 500 MHz · km the link. minimum modal bandwidth distance product Only the jitter components visible at the I'MD requirement and the aUowed range of dispersion MIC (PMD out and PMD in) are enforceable parts of parameters shown in the ANSI PMD Figure 14.' the standard. Note the sum of the jitter comro­ nents at PHY in (the exit of the receiver function) is Jitter Budget In most high-speed serial digital com­ 5.87 ns, leaving a 2.13-ns jitter-free window remain­ munications systems, the clock used to recover the ing in the 8-ns bit cel l. This window is allocated to received data must be extracted from the bit stream. the static alignment error and R.J of the clock recov­ The recovered clock is used to sample the data, and ery implementation. Digital developed specialized the sampling transition is nominally in the middle test equipment to generate and receive the DDJ test of the bit interval. If the sampling clock location pattern and to signal received bit errors; the error overlaps with the signal transition between bits, rate at the worst-case optical conditions (minimum errors occur. Jitter is time dither of the bit stream power, maximum jitter) was measured as a function signal transitions: the measured value is a function of clock sampling position to measure the jitter­ of the probability of its occurrence. Because jitter free window at the receiver exit. The 2.13-ns jitter­ is the predominant source of communications sys­ free window is the measured receiver component tem error, it is measured at a probabil ity equal to the requirement. BER requirement. A jitter budget tracks the accumulation of jitter in the bit stream edge position and allocates it to diffe r­ Ta ble 1 FDDI Jitter Budget Example (Nanoseconds Peak to Peak) ent components. The budget ensures there is a jitter­ free opening. or window. fo r the placement of the Measurement Point DCD DDJ RJ sampling clock . .fitter consists of three basic types: PHY out 0.4 0.0 0.32

• Duty cycle distortion-DCD PMD out 1.0 0.6 0.76 PMD in 1.0 1.2 0.76 • Data dependent jitter-DDJ PHY in 1.4 2.2 2.27 • Random jitter-RJ

Digital Te chnical jo urnal lf,f. _:; No. 2 5jJriiiJ'. /'J'JI 27 Fiber Distributed Data Interface

Op tical Loss Budget The optical loss budget fo r Fl)l)l • Isolation of error source ro a component of the is the diffe rence between the minimum op tical power network when the error rare is too high. Error launched into the fiber and the minimum optical rates may exceed acceptable levels as the resu lt of power required at the receiver. Decreased optic::ll a misconfigured network or a fault. Isolation of power in a receiver causes a reduction in the signa l­ the problem is the first step in a repa ir process. to-noise ratio and is evinced on the serial data stre:�m as an increase in its 1{1 . The optic:-tl power require­ A discussion of protocol correct ness and fault ments a re defi ned in terms of the performance mea­ isolation must consider more sources of errors sured with 62.5-micron-core mulrimode fiber. The than the normal bit error process discussed in the fiber core size is sp ecified bec:lllse the launch power previous section. l

tion due to inadequate receiver signal-to-noise ratio. budget limits. The important fa ults and miscon.figu­ rarions reduce the channel bandwidth or increase Physical Link Error Process the optical loss beyond the design limits. The error An important part of the physical layer development rare may exceed the design limit but rhe physics of was the analysis of the med ia bit error processes. In the error process remains the same. the previous section, we presented the design of the We analyzed the impact of this error process given optical link to control the bit error rate. This section FDDI the encoding/decoding and error-detecting considers the effect of the error process and the protocols. An example of an error event is shown isolation of certain types of faults that cause errors. in Figure 6 to illustrate the effect ofmedia noise on To eva luate the error process, we had to know the FDDI encod ing schemes. The code bits on the source s d the of the erro rs and tu y their effect on med ia are encoded as NRZI, where a signal t ra nsi­ the protocols for the FDDJ. The error process must tion represents a code bit I and a lack of transition be considered in light of rwo metrics: (for a bit time) represents a code bit 0. With this • Correctness of the protocol. Error events may lead encocli ng. a single noise event results in two code

to undetected corruption of user cl ara. Detected bit errors \v here the resulting pair of bits are the errors reduce performance. but an undetected complement of the original bits. In Figure 6, the

error may have nearly unbou nded bad effects fo r pair of code bits 0. 1 are changed to l ,0. There are four the user. Therefore the undetected error rate possible pairs of code bits -(lO. 10, 01, 11-thar

must be ve ry low. change to 11, OJ. 10, 00, respectively, by an error

28 l•b/. J No. 2 Sp rin.� /':J'JI Digital Te e/mica/ jo urnal Deueluprnentojthe FDDI PhysicalLaver

event. The FOOl PHY uses a block code in which 5 Although this method works reasonably well fo r code bits represent a symbol, and a symbol contains a bus topology, it is more difficult to use with 4 data bits. In the example, the single error event FDD! topologies. The quantity of physical links may changes 2 code bits, which results in a decoded greatly outnumber the MAC:s in the topology. The symbol with 4 incorrect data bits. The number of errors from more than a single physical link may data bits affected by an error event is multiplied by be counted by one MAC, thus masking which links the decoding process. exceed the error rate. For example, in a wrapped dual ring of single-i\ilAC, dual attachment stations, data errors occurring in only half the physical DATA DATA TRANSMITTED RECEIVED I inks in the network would be counted by a single event counter. A similar situation occurs in an FOOl SYMBOLS F F DATA BITS: 0000 1111 tree topology. The MAC error counters are not asso­

CODE BITS 1 1 1 1 0 1 1 1 0 1 ciated with a particular physical link. Fault isolation must be based on facilities present TRANSITIONS: � _I1_lL fo r each physical link. For this purpose, we devel­ oped a protocol cal led link error monitor (LEM).

NOISE EVENT LEM takes advantage of the requirement in the PHY \ standard that a set of code bit groups representing violation symbols and certain sequences of control F(�ure 6 Example of Single No ise Event u symbols not be transmitted (repeated) onto a phys­ ical link. Our study of the error process indicated Error <.letection is provided by redundancy in the that roughly 30 percent of the error events could data packet. Errors are detected by the MAC proto­ be detected by the physical layer decoder. '" This col based on a frame check sequence (FCS). The accuracy is acceptable as BER variations of many probability of an undetected error is related to the orders of magnitude are often the most important. number of error events in the packet and to the LEN! counts the decode violations that are received specific symbols created. Our analysis, based on a only at one point in the LAN immediately after the draft of the FODl iVlAC: protocol, ind icated that error event occurs. Errors not counted by LEM are undetected data corruption could occur with high those in which the created symbol may be repeated probability.'' In the important case, a new frame was by a PHY port, such as when a data symbol is changed created when a noise event changed a data symbol to another data symbol. An instance of LEM proto­ into an ending delimiter and created a smaller col may observe each PHY port and detect events frame. This truncation process resulted in an unde­ associated with a particular physical link. tected packet error rate of 3E-14 fo r large rings The accuracy of a LEM BER estimate is compara­ ('500 stations).'" Our design requirements include ble to other methods and has the advantage of pro­ the much more strict limit of I E-21 on this rate. viding better fault isolation. The accuracy of a LEM For this reason, an enhancement to strengthen the estimate is affected by the statistics given above fo r ending delimiter was proposed and accepted by the error process and the length of packet trans­ ANSI X3T95 fo r the MAC protocol." In accord with mitted on the ring. Generally we only assign signif­ this enhancement, a frame is valid only if its ending icance to the order of magnitude of the estimate, delimiter is fo llowed by a symbol that cannot be i.e., the exponent of the BER written in scientific created by the noise event that created the ending notation. This type of accuracy problem is shared delimiter. Thereby, undetected corruption was by J.lER estimates based on MAC FCS error counters greatly reduced in the final, standard MAC proto­ as well. For instance, the FCS-based estimate of BER col." This enhancement results in a undetected also depends on packet length and additionally on error rate of SE-24 fo r the protocols, allowing sig­ ring utilization. The FCS error counters count nificant margin fo r actual implementations.'" errors in va lid packets only, so estimates of error To isolate a faulty physical link, we need to know rate are strongly affected by ring utilization. The which of many links exceeds the design-specified LEM estimate includes error events in tokens, error rate. Each error event must be detected and stripped frames, and those that occur during the counted at one point in the topology. Using a tradi­ idle period between packets. tional method, we would isolate faults based on the The !.EM protocol counts errors and provides a information provided by the MAC FCS error counters. BER estimate fo r each link in the FOOl LA N. Network

Vo l. .) No Digilal Te e/mica/ jo urnal . .2 .)jJrinK 1991 29 Fiber Distributed Data Interface

management applications may collect this data and References identify margi nal links within the IAN. In addition, 1. Hawe, R. Graham, and Hayden, "Fiber LEM provides a mechanism ro automatically elimi­ W P Distributed Data Interface Overview," Digital nate fa ults from the network. This fault-recovery Te chnical .Journal, vol. 3, no. 2 (Spring 1991 , procedure preserves the integrity ofthe ring when this issue): lH. physical links would otherwise prevent ring opera­ 10- tion. The LEM protocol was proposed and included 2. FDDI Pf-I Y, ANSI Xj. / 48-1988, To ken Ring in the draft FDDI Station Management (SMT) pro­ Phys ical Layer Protocol (New Yo rk: American posed standard '' National Standards Institute, 1988). The analysis of the physical layer error process 3. FDDI PMD, ANSI X3 . 1 66-J990, Physical Layer resulted in two important changes that reduce the Medium Dependent (New Yo rk: American impact of errors. A change proposed ami adopted 1990). in the FDDI MAC protocol greatly reduced the rate National Standards Institute, of undetected corruption. The isolation of compo­ 4. T. Chaney and C. Molnar, "Anomalous Behaviour nents contributing to a high error rate is fa cilitated of Synchronizer and Arbiter Circuits," IEEE Tr ans­ I.EM, FDDI by now a part of the draft Station actions on Computers (April 1973): 421-422. Management standard. These developments have improved the correctness and maintainability of 5. B. Thompson and ./. Iannarone, Method and the FDDl IAN Apparatus fo r Detecting Impending Overflow and/or Underrun of Elasticity Buffe r, U.S. patent Summary no. 4,945,')48. Our development work on the FDDI physical layer 6. C. Kaufman, M. Kempf, and .J. Hutchison, Method provided physical layer components, specifications and Apparatus fo r Nodes in Networks to Avo id and new protocols. This paper has described the Shrinkage of an lnre rframe Gap, U.S. patent no. operation of the FDDI physical layer and the func­ 4,878,2 19. tional partitioning of the chip set. The functional partitioning resulted in greater integration and 7. .J. Hutchison and D. Knudson, "Developing Stan­ lower cost for the chip set. Much of the work on the dards fo r a Fiber Optic LAN-FDDI," SPIJ:: Pro­ physica l layer centered on the need to control the ceedings, vol. 715 ( 1986). error characteristics of bo th the constituent links 8. "LED and interplay of many asynchronous !.inks as a sys­ D. Hanson and ). Hutchison, Source and tem. Three important design problems were solved Fiber Specification Issues for the FDDI Net­ during the development effort. First, the elasticity work," JJ:J:/:' Computer Society Proceedings buffe r and smoother protocols, which we re devel­ COMPCON ( 1987) oped fo r the distributed clocking scheme, resolve 9. FDDI Draft of proposed standard fo r MAC, Accred­ data integrity and data loss problems. Second, the ited Standards Committee (ASC) X3T9.5/83-16, design of the fiber-optic link for FDDI required To ken Ring Media Access Control, Rev 10 (1986). methods to allocate system bandwidth and power margins. The bandwidth, jitter, and loss budgets pro­ 10. R. Jain, "Error Characteristics of Fiber Dis­ vided a means to allocate channel margin between tributed Data Interface," IEEE Tra nsactions on 8 (1990): 1244-12)2. individual components and can be applied to the Co mmunications, vol. 38, no. design of many transmission systems. Finally, the 11. R. Jain, Error Analysis," FDDI X3T9.'i analysis of the physical link error process resulted "FDDI Working Group on SMT, Committee Document in increased correctness through a reduction of SMT-80 ( 1987). the undetected error rate and enhanced fa ult isola­ tion provided by the link error monitor, I.EJY! . 12. fDDI MA C, ANSI X3. 136-J987, To ken Ring Media Access Control (New Yo rk: American National Acknowledgments Standards Institute, 1987). The authors have reponed on the results and work of many individuals. Special acknowledgment is 13. H. Levenson, "Link Error Monitor (LEM)," FDDI extended to Raj )ain, Don Knudson, Charl ie Kaufman, X3T91 Wo rking Group on SMT, Com mittee Docu­ and Herman Levenson fo r their original contribu­ ment X3T95/88-251 (October 1989). See also tions to the development and fo r their help in writ­ "J.EM Error Monitor," Document X3T9.5/8 8-198 ing this paper. (August 1989).

I Vo l. .) No. 2 .\fJI'iiiP, I')') Digital Te e/m ica/ jounwl Henry S. Ya ng Barry A Sp inneJ' Stephen To wning

FDDI Data Link Development

Th e fi ber distributed data interfa ce (FDDI) data link is based on the ANSI X3 T9 .5 FDDI standards with Digital's enhancements to provide greater pe1jormance, relia­ bility, and robustness. Th e FDDI proj ect team encountered significant challenges, including the evolving ANSI X3 T9.5 FDD! standards and the development of the technology to irnplement the data link, coupled with time-to-market pressure. App ropriate considera tions and design trade-offs were made to design complexity, performance, risk, cost, and schedule, when deciding fu nctional partitioning and semiconductor technology Ex tensive simulations and a nouel test approach were used to verify the algorithms, the fu nctional models comprising the chips, and the physicalchi ps themselves.

The prol iferation and importance of distributed development methodology used in simulation, ver­ system appl ications place special requirements on ification, and testing. networks in terms of topological flex ibility, per­ fo rmance, reliability, scalabil ity, robustness, and FDDI Data Link Overview efficiency. The advent of fiber optics, large- scale The FDDI data link provides an upward multi­ integrated circuits, and related technologies makes plexing, datagram service to support multiple data it possible to provide a relatively low-cost, high­ link users concurrent ly within the same computer speed local area network (LAN) with large physi­ system. The FDDI data link incorporates the cal extent and connectivity. One IAN standard is ISO 8802-2 logical link control (LLC) and FDDI stan­ the fi ber distributed data interface (FDDI), a 100- dards. Also, the FDDI data link provides a mapped megabit-per-second token ring that uses an optical Ethernet service, defined by the Internet R.FC 1103 fiber med ium. standard, to map an Ethernet frame onto an ISO The scope of FDDI spans the data link layer and 8802-2 LLC frame fo r transport over the FDDI LAN.' the physical layer. The FDDI data link provides its The FDDI data link consists of an LLC sublaye r, a users with communication services on a multi­ media access control (iVlAC) sublayer, and their man­ access !.AN fo r transmitting and receiving frames agement. The LLC sublayer provides LLC services, with best-effort delivery service (also called the the mapped Ethernet service, and multiplexi ng/ datagram service). The deve lopment of the FDDI data demultiplexing services fo r multiple users. The key link encountered several significant cl!al lenges, functions provided by the MAC sublayer include including the instability of the standard, unproven the FDDI token ring protocol, frame transmission technology and protocols, and an order of magni­ and reception, initialization and error recovery fo r tude increase in speed from the International the token ring, address recogn ition and filtering Standards Organization (ISO) 8802-3 carrier sense on receive, and frame error detection. Figure 1 is multiple access with collision detection (CSMA/CD) an example of the FDDI architecture model, show­ IAN. The design of the FDDI data link involved ing a dual attachment station (DAS) or dual attach­ performance considerations such as throughput, ment concentrator (DAC) with a single data link latency, data integrity, and reliability." entity. A DAS or DAC may have zero, one, or two In this paper we discuss the development of link entities; two or more physical (PHY) port enti­ Digital's FDDI data link and present some of the key ties; and control of their management. A link entity

algorithms developed by Digital. We then describe is an instance of data link that contains an LLC and a the design and development of the FDDI data link MAC entity. A data link user accesses the data link technology and its implementation in the FDDI, services through the port entity. As shown in focusing on the FDDI data link's very large- scale Figure 1, multiple data link users may use the same integration (VLSI) chip set. Finally, we describe the link entity.

Digital Te cb nical jo urnal Vo l. 3 No. 2 !c!jJ rin}; /'J'J/ 31 Fiber Distributed Data Interface

SMT USER 1 USER USER M MANAGEMENT 6 I I APPLICATION �--��----� I·. ·I I I I I I I ------� ------T ------r ------� ---- 'f 'f 'f 'f PORT A PORT S PORT T SMT PORT N

' ' ,------. ------DATA ' LINK STATION 1 LAYER t t • 'f LINK 1 B I MAC I ------r------STATION PHYSICAL ATTACHMENT MANAGEMENT

PHY PHY PORT O PORT R

PHYSICAL LAYER EJ ... EJ I PMD I I PMD I " -� � DASIDAC

II II � II � II KEY: DATA LINK SERVICE INTERFACE DUAL ATTACHMENT STATION ____,... DAS DAC DUAL ATTACHMENT CONCENTRATOR DATA LINK MANAGEMENT INTERFACE - LLC LOGICAL LINK CONTROL MAC MEDIA ACCESS CONTROL = � FIBER-OPTIC INTERFACE = = PHY PHYSICAL ENTITY ------� INTERNAL INTERFACE PMD PHYSICAL MEDIUM DEPENDENT ENTITY

Figure 1 An Example of the FDDJ Architecture Model

The FDDI data link uses the FDDI MAC protocols amount of time a MAC may transmit." The speci­ to provide fair access to a multiaccess channel, fied token rotation time is called the target token which is built of individual point-to-point physica l rotation time (TfRT). The TTRT is negotiated using links.' A token ring consists of an ordered, cyclic a distributed algorithm called the claim token set of MAC protocol entities called MACs. It oper­ algorithm, which is invoked each time the ring is ates by passing a token sequentially from MAC to initialized. The FDDI MAC protocol also includes MAC around the ring. Only the MAC with the token fa ult detection and recovery fu nctions to aid in the may transmit frames onto the ring, and only one restoration of ring operation in the presence of token can be present on the ring at any instant. At transient fau lts. the end of transmitting its frames, the MAC trans­ As shown in Figure 1, each instance of a station mits the token omo the ring. Frames circumnavi­ (e.g., DAS or DAC) contains a set of station man­ gate the entire ring and are subsequently removed agement functions. Some of the key station man­ (stripped) by the originating MAC after one and agemem functions included are initialization, only one rotation. The FDDI MACprotocol is differ­ observation and control of link and PHYport enti­ em from the ISO 8802-5 token ring protocol.; ties, control of the insertion and removal of the

The FDDI MACprotocol uses a timed token proto­ station from the ring, topology control, fault detec­ col, whereby MACs on the token ring cooperatively tion and recovery, and a set of frame-based proto­ attempt to maintain a specified token rotation time cols. The set of frame-based protocols for station by using the observed network load to regulate the managemem includes duplicate address detection,

32 H')f. 3 No. 2 Sp rinf( 1991 Digital Te cht�icaljournal FDDI Data Link Development

neighbor notification protocol for ring map gener­ Therefore, the usc of the source address march ation, and loopback protocol The scope of the sta­ algorithm by a bridge imposes significant costs tion management fr ame-based protocols is limited and implementation complexity. The design of the to a single ring. These funct ions are implemented frame stripping algorithm is made more difficu lt in the Common Node Software (CNS) and the FDDI by the fact that there can be up to 560 frames out­ chip set.- standing (i.e., transmitted but ye t to be stripped) and the fact that there is less than one microsecond FD DI Data Link Algorithms to decide whether to strip or to re peat the fr ame . Digital's realization of the FDDI data link includes Digital developed an algorithm called the fr ame major enhancements and value-added features. The content independent stripping (FCIS) algorithm, development of the FDDI data link encountered which is implemented in the MAC chip.''11 The algo­ several architecture and implementation issues. The rithm is based on stripping the same number of key issues included design for high performance, fr ames that the MAC transmitted on the ring inde­ error characteristics and data integrity, removal of pendent of the content of the frame. After the MAC frames by bridges, cleaning the ring of unwanted captures the token for frame transmission, a local frames and long fragments, and stability and relia­ count is incremented each time a frame is trans­ bi l ity of the ring.' Digital provided the impetus to mitted. After transmitting all its frames, the MAC resolve these issues as well as the solutions to be transmits a void frame, which is a minimum size incorporated into the ANSI FDDI standards. Some of frame (i.e., 17 bytes), before transmitting the token." u these solutions are described below. On receive, if the count is greater than zero, the received frame is stripped; and for each error-free Fra me Co ntent Independent Stripping fr ame received and stripped, the count is decre­ (F CIS) Algo rithm mented . After transmitting the token, the MAC does The underlying logical topology of all token ri ngs not strip frames when the count is equal to zero, is a closed loop structure, which inherently has the except for frames with a source address matching pro perty of cont inuously circulating frames trans­ the MAC's address. When receiving an error-free void mitted on the ring. Because of this property, all fr ame with the source address matching the MAC's token rings have algorithms for removing frames address, the co unt is unconditionally reset to zero. transmitted by MACs on the ring. The frame removal As a result, the algorithm uses a count that is kept algorithm is called a frame stripping algorithm. locally to track the number of outstanding fr ames When stripping a frame, the fr agment size (remnant for stripping, and it uses a transmitted void fr ame as of the fr ame) must be less than 17 bytes. Frames a backup mechanism to indicate the end of stripping. that are not properly stripped can remain and tra­ The operation of the FCIS algorithm is seen in verse the ring repeatedly, wasting bandwidth and the space-time diagram of Figure 2, which shows resources within systems on the ring and causing three stations on the ring whi le station B is the only FCIS severe congestion in the systems due to delivery of station participating in the algorithm. Time duplicate fra mes. increases fr om top to bottom, and the position of The FDDI MAC protocol uses a frame stripping downstream stations on the ring goes from left to algorithm in which all MACs on the ring continu­ right in this diagram. Stat ion A first receives the ally strip received frames whose source address token T and transmits frames A I, A2, and the token. XI, Yl, matches their own MAC address. Limitations of this Station 13 then transmits fr ames and Zl. X, algorithm arise when implementing bridges or which have source addresses Y, and Z that are systems that need to transmit frames with source not the same as station B's MAC address. As shown addresses which are differem from each MAC's own in Figure 2, each time station B transmits a frame. address." For example, a bridge may forward frames it increments its count; and each time station B with no modifications, and, therehJre, the fo rwarded strips an error-free fra me, it decrements the count. frames contain source addresses which are differ­ After transmitting fr ame Z I, station B transmits its ent from the bridge address. Also, a bridge may sup­ voicl fra me, VB, and then the token. \Xih en station B port tens of thousands of stations in the extended receives its void fra me, the count is reset to zero. !.AN.'' lf a bridge were to use the source address which ca uses station B to stop fra me stripping. match algorithm, the bridge would have to complete Subsequently. an entirely new epoch of transm is­ the address match operation within one micro­ sion and frame stripping can begin with the next second from the beginning of the fra me reception. token capture.

Digital Te clm ical jou rnal /'J'JI Vol. :i No. 1 .\jJri11g 35 Fiber Distributed Data Interface

circu late around the ring, along with the token, continuously at the speed of the ring. NOFs may be received by one or more systems on the ring re peatedly at an extremely high rate, which can lead to severe congestion and waste of system resources. For example, a single NOF on an idle ring can create a frame arrival rate of 2,700 frames per second (for maximum size frame) to about 290,000 frames per second (for m in imum size frame). In the best case, the NOFs arc removed when they arrive at a MAC that is transmitting (i.e., holding the token). Digital developed an algorithm called the ring purging algorithm to remove NOFs. The ring purg­ ing algorithm ensures that NOFs do not traverse the ring more than twice. The implementation of ring purging consists of two related, but diffe rent, algo­ rithms. The first one is the purger election algo­ rithm, which is a distributed election algo rithm to select a designated MAC to be the purge r for the ring. The second algorithm is the purging algo rithm, which is executed by the designated MAC to clean the ring of NOFs. We describe the purging algorithm in this section. The purging algorithm adopted fo r Digital's FDD I data link purges the ring transparently each time a token is received by the purger. When the pu rge r receives a token, it begins a purge cycle by trans­ - TOKEN mitting two special frames. called void frames. If FRAME STRIPPING the purger has frames to transmit, it completes the STATION B IS USING THE FCIS ALGORITHM transmission of its frames before starting the purge NOTE THIS FIGURE IS NOT TO SCALE cycle. Once the purge cycle has started, the purger unconditionally removes all frames or fragments Ngure .2 haiJ/e Content Independent Sl ripjJ ing received. The purge cycle is terminated when the Algorithm £.\WiijJ/e purge r receives one of its error-free void frames, a token, or a ring initialization frame. In order to increase the probability of correctly terminating The combination of the count ancl the vo id frame the purge cycle, the purger transmits two void provides robust frame stripping that is independent frames: but it terminates its purge cycle based on of the content of the transmitted frame. The FClS receiving only one error-free vo icl . algorithm also has the desired property of operat­ Figure :) shows the operations of the ring purge r ing transparently with other :YIAC:s which are not in removing an NOF, cl uring an idle ring and dur­ implementing the algorithm. Implementing the FCIS ing a buS\' ring. Each time station S:) (the purger) algorithm in the ;\lAC chip greatly retluccs the cost receives the token, it may transmit its frames, fo l­ and complexity of products, which otherwise may lowed by rwo void frames and then the token. It need additional hardware components to perform purges the ring until it receives one of its error-free similar functions. void frames. As shown in the example, the NOF was p urgetl hy station S:)on its second traversal around Ring Ptuging Algorithm the ring. Also. the example shows that the purging One of the well-known problems on a token ring is of the ring is transparent (i.e., there is no disruption the circulation of frames or long fragments that are to the ring). not stripped by the transmitter. The frame or long The impact of ring purging on ring performance fragment not stripped b\· the transmitte r is called a is negligible, because the ring purger only initiates no-owner frame (NOF). On an idle ri ng, 1"\J OFs can the purge cycle when it has the right to use a token.

I''' .> '''· l .\jJrin.� /')')! Digital Te cb uical jo urnal fDDI Datu Unk Det ·e!opment

SPACE- allows the purger election algo rithm to be more optimistic (i.e., when in doubt during election, one w--=------can start purging) during the transition period when the distributed election algorithm is stabilizing. The purge r election algorithm is implemented in the Common Node Software (CNS), and the purg­ ing algorithm is implemented in the :.1AC chip.

FDDI Data Link Chips The fDDI 1\1AC sublayer hmctions arc implemented by the three FDDI data link chips: the ring mem­ ory controller (RMC), media access control (iVIAC), and content addressable memory (CAM). The RJVIC interfaces between the frame buffermemory on the system side and the 1'<1AC chip on the network side. �- - .. It consists of a direct memory access (DMA)engine ' / /' : . MAC · LV '// / F11 ' · // //"';•=:: designed to supply the chip with frames to send - · ..1. • I . , / - - • - - - /// - - - - - // I - - - - - /1 - - :;: --- - - ...... and to store frames received from the ;vlAC chip. - - . ------I - --= ::1 The interface on the system side provides gathered reads on transmit and scattered writes on receive. Although tile &VI C's interface to the MAC chip was custom designed for FDOI operation, the RM C can, in principle, be used fo r other data links that run at 100 megabits per second or less. The MAC and CAM chips implement the FDDl MAC: protocol func­ tions. The functions implemented by the MAC chip include the token access protocols, frame del in­ eation, frame parsing. address recognition, frame check sequence generation and verification, frame insertion, frame repetition, frame removal, token ge neration, and error detection and recovery algo­ rithms (e.g., the beacon and claim algorithms). The CAM chip provides the destination address filtering, which determines if a received frame is to be received or discarded, and the setting of the A-indicator, which is part of the frame status field.

NOF NO·OWNER FRAME RMC Chip NOTE THIS FIGURE IS NOT TO SCALE. The RMC chip is a high-performance coprocessor intended fo r full-duplex data transfer between the Figure 3 Ring Pu rging t: :xarnple buffer memory and the MAC chip. It uses a pair of circular buffe r queues fo r transmit and receive to In the worst case, the ring purger's effe ct on usable manage DMA data transfers to and from the buffer bandwidth is Jess than 0.22 percent. For implemen­ memory. Two independent on-chir. first in, first tations compliant to the ANSI FDDI MAC standard, out (FIFO) buffers, fo r receive and transmit, are pro­ these void frames have no effect since the standard vided to decouple the buffe r memory from the real­ prohibits copying void frames. time nature of the �lAC interface. A fragment ancl The ring purging algorithm removes NOFs, includ­ frame fi lter is provided to reduce unnecessary mem­ ing long fragments. without disrupting the operation ory accesses caused by the reception of fragments of the ring, and it removes NOFs within two traver­ or frames not addressed to this station. As shown sals of the frame. In acklition, it has an important in Figure 4, the RM C chip has three interfaces: the property that permits more than one purger to processor interface, the MAC chip interface, and the opc:rate in the same ring at any time. This property buffe r memory interface. The processor interface

Di,�ilttf Te e/m ica/ jo urnal HJ/. 3 No. .! Sp rinf, 1991 35 Fiber Distributed Data Interface

TO/FROM BUFFER MEMORY

! 32 BITS DATA PLUS PARITY � ------�------1 t I BUFFER MEMORY INTERFACE AND ARBITER I I I I RECEIVE DMA TRANSMIT DMA I STATE MACHINE STATE MACHINE I I I I TRANSMIT FIFO RECEIVE FIFO I BUFFER BUFFER I (128 BYTES IN (256 BYTES IN I 32 BY 40 BITS) 64 BY 41 BITS) I I I PROCESSOR t INTERFACE I I MAC RECEIVE MAC TRANSMIT INTERFACE INTERFACE • I I I I I I I I • 8 1 8 BITS I �--�:�•••••• ••••••••••_! 1 L------�-�------�-----

1 t t MAC INTERFACE PROCESSOR INTERFACE KEY:

- INTERNAL DATA PATH

..... EXTERNAL DATA PATH INTERNAL CONTROL - MAC MEDIA ACCESS CONTROL

DMA DIRECT MEMORY ACCESS

FIFO FIRST IN. FIRST OUT

Figure 4 Ring Memo1T Controlfer Cbip Block Diagram

allows the i n i t ia li zation, control. and observation It is also a fu lly synchronous design ] some self- t imed of the RMC The MAC ch i p interface consists of high­ logic is used in the FIFO random access memory speed transmit and receive data paths for tra nsfe r (RAM) devices] using a 12.5-megahertz (MHZ) pri­ of data and control i nformat ion between the MAC m:�ry clockand 25-MHz clock fo r sampling incom­ chip and rhe Ilvi Cs f'IFO buffers. The buffe r memory ing signals. The FJFO RAM was implemented us i ng i nterface p rovides a hu rst mode DMA fo r read/ a fu ll custom methodology, and the remainder write from/to the buffe r memory on a s ingle bus was i mp leme n ted using an automated standard arbit ration cycl.e, where the burst size can he up eel I met hodology. flYI C to fo ur or eight longwo rds. The RMC: can provide a The i n terfaces to buffer memory us i ng a data maximum data transfer rate of about 44 mega­ and address mu l t i plexed bus, which is a 32-bir-wide byres per second. bus plu s the fo ur pari ty signals and additional con­

The RM C is imp lemen ted using a I .') -mi cron­ trol signals. A bus transaction consists of an address drawn, rwo-meral-lay<.:r custom complementary cycle dr iven by the RM C: followed by a burst of data metal oxide semiconductor (CMOS) technology. lt cycles, either driven by the RMCon receive or driven uses rough l y 87,000 transistors, a large number of by the buffer memory on transmit. One of the which are useu fo r the two FIFO buffe rs, where the unique featu res of this chip is that it is able to use a receive FIFO buffe r is 2'56 byres and the transmit 52-bit-wide buffe r memory composed of low-cost l'IFO buffe r is 128 byt<.:s.The RMC uses 102 signal 1100-nanosccond (ns) access time ] dynamic random­ pi ns and is avai lable in a 132-pin cnquad package. access memory (DRAM)ch ips, whereas many of the

VrJ/. J No. 2 S{J ring 1')')1 Digital Te chnical jo urnal FDDJ Data Link Det •elopment

other FOD! memory controllers avai I able on the buffe rs (and hence multiple descriptors), but a speci­ market require a 64-bit buffe r memory or require fic buffer/descriptor can only be used by one frame. very fast DRA!'vl or static random access memory Each receive buffe r is required to be 512 bytes (SRAM) chips. To achieve these reduced memory long. The RM C rewrites each descriptor in the and chip requirements, the RMC's buffe r memory receive ring to indicate the number of bytes actu­ accesses were done in bursts of between one and ally used; and if the buffe r is the last one for the eight Jongwords. Then, by making use of the D�Yl 's frame, then the RMC writes the receive status and the fa st page mode, in which subsequent, sequential frame byte count into the descriptor. Transmit buf­ reads/writes are faster than the first one, the needed fe rs are also 512 bytes long and the RMC reads the buffermemory bandwidth is attainable. size (in bytes) from the first descriptor fo r the frame. The R.MC directly accesses a transmit ring and a receive ring, each consisting of a circular queue MACChi p of descriptors. Each descriptor supplies the buffe r The MAC chip implements the FDD! MAC protocols, memory address of a transmit buffe r or a receive and it interfaces between the RMC or equivalent buffer. An OWN bit mechanism is used in each chip and the FDDI physical layer chip.'' As shown in descriptor to determine if the descriptor and its Figure 5, the MAC has fo ur interfaces: the processor buffe r is owned by the .RNIC or not. It supports gath­ interface, the RMC interface, the physical layer ered read and scattered write in which frames to chip interface, and the CAM interface. The proces­ be received or transmitted can use one or multiple sor interface allows the initialization, controL ancl

PROCESSOR RMC INTERFACE INTERFACE r------r-�------�--• I • ························• ••...... ••...... 1 r I t I I RMC RECEIVE RMC TRANSMIT I INTERFACE INTERFACE I

I ADDRESS I TIMERS CAM COMPARE - · ...... ------· INTERFACE r I I t I RECEIVE TRANSMIT PROCESSOR STATE STATE I INTERFACE I MACHINE MACHINE I r

TRANSMIT CRC RECEIVE CRC SEND CHECK! CHECKER FRAME GENERATE f-

I 8 BITS t

PHY RECEIVE PHY TRANSMIT INTERFACE INTERFACE

� l······------••••••••••••••••••••••••• J ------� - � ------� I t PHYSICAL LAYER CHIP INTERFACE KEY

- INTERNAL DATA PATH RMC RING MEMORY CONTROLLER CRC CYCLIC REDUNDANCY CHECK •• EXTERNAL DATA PATH • - CAM CONTENT ADDRESSABLE MEMORY - INTERNAL CONTROL PHY PHYSICAL LAYER

Figure 5 Media Access Control Cbip Block Diagram

Digital Te ch nical journal Vo l. j No. 2 .Sj; rinp, 1991 37 Fiber Distributed Data Interface

observation of the MAC. The RM C interface is cus­ as part of the frame. The packet request header is tom designed fo r FDDJ operation ancl allows the used to instruct the iV lAC chip on how and when to MAC to interface to either the RMC or to equivalent transmit the frame. For example, it instructs the chips implemenring the RJv!C interf<�ce. The physi­ MAC chip on whether to append the FCS to the cal laye r chip interf<�ce allows the ,vlAC to receive frame or not, or on the type of token to use when and transmit on the FDDI ring. transmitting the frame. The MAC chip is implemented using a L5-micron­ drawn, channelless, two-metal-layer CMOS gate-array CAM Ch ip technology and uses roughly 49,000 transistors The CAM chip provides a 64-entry content address­ (12,000 used gates). The MAC chip uses 86 signal able memory where each entry is a 48-bit address. pins and is available in either a 120-pin pin grid Typically, the entries in the CAiVl consist of multi­ array (PGA) package or a 120-pin plastic quad flat cast addresses usecl by upper layer protocols and pack (PQFP) package. This fu lly synchronous design the data link management protocols. The CAiVl is uses primarily a single 12.5-MHz clock (80-ns cycle used to parse the destination address field of each time): this operation is applied to the microproces­ frame received to decide whether the destination sor bus as welL An additional double-speed clock is address received matches one of the entries in the used in some of the peripheral interface logic to CAM. The result of the address match is then pro­ sample incoming signals and prevent hold time vided in real time as input to the MAC chip, which problems associated with clock skew between dit� decides whether to receive or to discard the frame. fe rent chips. In the worst case, the J'vlAC protocol requires that the The internalstr ucture of the �1AC chip has a hill­ destination-address-match decision be completed duplex architecture. No logic is shared between the in less than lO bytes of time (i.e., 800 ns). starting receive ancl transmit port ions of the chip. Hence from the end of the destination address field. this chip can receive and transmit simultaneously The CAM chip is implemented using a 1.'5-micron­ for an indefinite period. This capabi lity complies drawn, two-metal-layer custom CMOS technology C with the ANSI FDDI ,vlA standard that implemen­ and uses roughly 44,000 transistors - 34,000 of tations be able to receive, parse. and validate cer­ which are used fo r the core array of 64 words of tain frames (e.g., claim frames and beacon frames) 48 bits (plus a valid bit). The CAivl chip uses 37 signal

even while transmitting. Two separate frame check pins and is available in a 44-pin cerquad package. It sequence (FCS) checke r/generators are required also is a fu lly synchronous design using the same fo r transmit and receive fu nctions. The MAC chip 12.5-MHz primary clock (plus the 25-il·lHz clock fo r calculates the FCS, which is specified as a specific sampling incom ing signals). 32-bir cyclic redundancy check, eight bits at a time. As shown in Figure 6, the CAi\1 consists of three A one-bit implementation is much smaller, but interfaces: the processor interface, the MAC chip requires a 100-MHz clock. Even with such a clock, it interface, and the physical layer chip interface. The is nor easy to implement one bit at that speed. A byte­ physical layer chip interface consists of an 8-bit wide implementation requires considerably more bus which transfers data bytes from the physical exclusive OR (XOH) gates, but exploits more of the layer chip to the MAC chip. Every 80 ns, one byte is inherent parallel ism of the algorithm and hence loaded into the appropriate position in the 6 -byte can be implemented using slower clock speed. internal compare register. Once the last byte of a One of the important fe atures provided by the 48-bir address has been clocked infO the C:Ai\1 chip's .YlAC: chip is the support of a 3-byrc packet request compare register, a march/no match indication is header fo r transmission. The use of the packet given to the MAC chip within 120 ns by the MAC request header construct allows simple, pipelined chip interface. processing of the transm it descriptor along with The other interface to the CAivl is the processor the rransmit data at high speed. This construct interface. which is used by the processor to load a I lows a software device driver to build a trans­ ancl change the CA.'vl entries. The processor does nor mit descriptor to prl'cede and identifythe frame, directly read or write the CA.\1 array, but instead which is then passed through the system bus and reads and writes (16 bits at a time) to three 16-bit DMA data movers fo r delivery to the MAC chip. data registers and one 16 -bit command register. By Every frame transmitted by the MAC chip must first clearing the appropriate hit in the command regis­ contain the packet request header, which is used ter. the processor requests a read, write, or compare as thc transmit descriptor and is nor transmitted from the CAM array. The arbiter ensures that the

i No Hi/ .. . .! !:>/Hill� I'J'JJ Digital Te c!Jnicaljo urnal FDDI Data Link Development

Every chip was partitioned into logical subblocks and a DECSIM (Digital's simulation tools and lan­ guage) block-behavioral model was developed fo r each subblock. The externalin terfaces and internal structure of the model accurately represented some unit or subunit of a chip. When the model inter­ faced to some nonexistent model (e.g. unwritten behavioral model or external interface to another chip), a transactor was provided. The transactors provided accurate models of interfaces (timing, control, and data signals), but lacked the internal detail of a behavioral model. The use of the trans­ actors was particularly important when modeling the buffermemory interface, since the design of the interface is implementation dependent (designed I FROM PHYSICAL according to product needs by the development LAYER CHIP group using the FDDI chip set). KEY Each block-behavioral model was tested in isola­ - INTERNAL DATA PATH tion; then all were combined to produce a behav­ EXTERNAL DATA PATH •••• ioral model of the target chip. When the chip model - INTERNAL CONTROL was successfully tested, each behavioral subblock CAM CONTENT ADDRESSABLE MEMORY was replaced by a corresponding structura l model MAC MEDIA ACCESS CONTROL representing gate/transistor logic. The new chip model was then retested unti I the structural model Figure 6 Content Addressable Mem01y Chip behaved identically to its behavioral counterpart. Block Diagrarn As the model of each ch ip was completed, the trans­ actors driving its external interfaces were replaced CMvt array is idle (i.e., not used by the compare with the model fo r the next adjacent chip. The operation) before al lowing the processor's request resulting combination was then tested together to complete. In order to ensure atomicity fo r read, using the remaining transactors and test vectors. write, and compare, this arbitration is required This process tested interoperability between chips because the actual CA.J.\1 array can only rerform one and was repeated until all chips in the chip set had transferor compare at a time. The arbitration mech­ been tested together as a system. anism guarantees that the compare register access After chip layout was completed, the structural is never delayed, since the MAC chip requires the models were enhanced to reflect the more accurate match indication to be valid during a specific clock timing data. The test vectors were again applied cycle. The CAlviis also designed to al low entries to using a checker model, which consisted of one be added and removed at any time (even while the block-behavioral model ami one structural model FDDI ring is runn ing). of the same chip. The test vectors were applied to each model simultaneously, and external and inter­ FDDI Simulation and Ve rification nal signals of both models were compared fo r con­ Digita I 's FDDI technology development method sistency. Any discrepancy between the monitored was a top-down approach, starting with high-level signals was thoroughly investigated and corrected system models of FDDI behavior and progressing to as necessary. more detailed behavioral and structural models as For test and debug, we planned ro develop a dedi­ confidence in fu nctionality increased. Several simu­ catec.lhardware tester to test the physical FDDI chips. lation models and analytical models were developed Unfortunately bugs fo und by using such a tester occur to study and model the FDDI at the architecture and too late in the process-the chips are alreac.ly built. system levels. Using these models, studies were In order to meet our time-to-market goa l. we needed done on the error characteristics and robustness to maximize activity in the simulation environment. of FDDI. stability of the ring to[)Ology. performance Rather than waiting fo r the hardware. we decided and operational behaviors. and correctness of the to develop and apply as many of the tester-based protocols.' tests as possible in the simulation environment.

Digital Tixb nical jo urnal \·'IJI. j No .! .\j>rillt: I':J'JI 39 Fiber Distributed Data Interface

Simulation Te st Bed 2. It \V<"t S far easier to debug the tests in the simula­ The ve rsion of OECSLYI we were using provided a C tion environment rather than on the physical language interface capability. Hy writing rlw tesrs hardware. Duri ng simulation we cou ld observe in C language and making use of a common but ami control chip behavior. Te st results were easy small enviro nment-specific inte rface library. it to determine with clear pass or fa il ind ications, was possible to simulate the system behavior using and no human interpretation of large strings of O's the chip models. Hy rewriting the interface library, J's and was required. the same rests were run unchanged on the test er I 5. Since aJ rest code was developed in C, no specia 1- and were used in regression resting of the chips i:t.ed rest language was required , and standard during the fa brication process. The rests. inte rface support tools were readily available. Libraries of I ibrary, and chip models became known as the sim­ reusabk (debugged) test -support functions were ulation rest bed. This was the first attempt to use available, provid ing such functions as create I this DFCSLM capabi i ty as a cornerstone of a devel­ frame, compare frame, write control/status reg­ opment strategy. ister, and read control/status register. Al. l the chip models were combined to construct 4. Easy transit ion back ancl fo rth between the phys­ an entit�· resembling a single attachment station ical and simulation environments was imporram. (SAS) which was then tested as a fu ll system, rhar Any bugs fo und in the physical environment had is, a single operational FDDI node (in loopback). be reproduced in the simulation environment Extensive use of mixed mode simulation (mixing to in order to rest bug fixes. transactors and behavioral and structural models) aided rest bed performance because the level of JDD! Te ster model detail could he varied. depending upon the In parallel with chip design, another development area bei ng rested . Time was saved by substituting group was assigned to rest the chip set in a proto­ higher level models in areas peripheral to those type fDDI system. Their approach was to design an under test. FDDI tester that used the FDDL chip set. We also The simulation cluster was a cluster of fo ur VAX wanrec..J make this tester configurable as various HH40 systems. each system hav ing fo ur processors. to FDDI entities, e.g., an SAS, a wiring concentrator, or Some idea of the extent of the d'fo rt expended can an Ethernet-to-FDDI bridge. so that we could use be conveyed by the fo llowing statistics: the FDDI tester to build an FDDI ring to investigate

• The total number of Cl'll hours used h>r the: the behavior of the ri ng. des ign and ve rification dl<>rt was 30.240 (196560 The main value of the tester was its capability to II-7HO C!'l t VAX hours equivalent). perform long-term. steady-state testing, using bil­ lions of frames. It was also req uired to test complex • For the single-node t<.:st b<.:d. there were H27 topologies usi ng multiple testers (as FDDI stations) MAC and .1 H4 tests. llsing rh(_' HH40 cluster. RMC in large rings. These activities are prohibited in the the test suites required _) )6 (2. 1H!t ll-7HO VAX: simulation environment because of t.he excessive hours) and 192 hours ( l.24H II-7HO Cl'l CI'L VAX amount of compute rime req uired. The tester had to hours), respectively. fo r completion. be capable of driving the data at full FDDL baml­ width. introducing co ntrolled error conditions on The i ndi\·id ua l tests varicd in com piLx it\· from those req uiri ng a fe w CPl . min utes to tbose req uir­ rl1c: fi ber. ami accurately monitoring act ivity on the ing days to run. For example. one MAC test which ri ng at hill FDDI bandwidth. The tester itself was con­ loops back ten ')12-hyte packets within the single­ structed so that it could be controlled via an exter­ nal Ethernet link and multiple resters could be node test hed requ ired 3C>

I. The tests assisted the chip designers to discmTr riming expectations of the tester environment. bugs in the chip designs at the correct stage of The next stage of testing involved combining development -in design rather than after si I icon. testers into multinodc FDDl configurations and

Digital Te ch uical jounwl 11>1. _; Su . .! .\jHiJig /'}')/ FVVI /Jato UHk DeuelopmeHt

exchanging data at fu ll speed over extended periods. References This stage was also successfu l; no additional bugs l. R . .Jain, "Characteristics of Fibc.:r Distributed were discovered, and the data link performance Data Interface (FDDJ)," IU:L ha nsactions on and ring stability exceeded the expectations. The Communications, vol. .)8. no. 8 (August 1990). steady-state testing was then augmented by intro­ 2. ducing pathological conditions into the ring such R . .Ja in, "Performance Analysis of FDDI To ken as stations with duplicate addresses ancl noise on Ring Networks: Entity Effect of Parameters the fiber. The prom iscuous capture modes and and Guidelines fo r Setting "JTHT:· Proceedings A01 SIGCOMJJ frame/event rime stamping proved invaluable in of tbe '!.)() (S<.:ptcmber 1990). analyzing subsequent behavior in these cases as the .). Standard fo r the TrcniSJIIi.,·sion of JP Data­ effects were often complex . Station management grams Ouer FDDI Netu•orl?s, Internet Engi­ implementation was highly stressed anclperf ormed nc.:ering Ta sk Force (IE1T) IU'C 110:) Oune 1989). without error. 4. M. J Johnson, "Fairness of Channel Access fo r Finally, the same testers were configured as Non-Time-Critical Traffic Using the.: FDDl Token bridges, ancl prototype Ethernet-to-FDDI bridging Ring Protocol," NASA Am<.:s Research Center, firmware was introduced. At this stage several Research Institute fo r Advance Computer minor deficiencies with the control algorithms Science. RIACS TR 86.9 (March 1986) required for bridging were detected. These deficien­ cies were cluemainly to insufficient analysis of FDDI '). To ken Ring Access Method al/{1 Pb )'sical Lc�J·er bridging requirements by the test team; therefore. SjJ eCijications, ANSI/IEEE Standard 802.5-1989 the test cases were correspondingly inadequate. (New Yo rk: The Institute.: of Electrical and All problems were repeatable on the simulation Electronic Enginee rs, lnc., 1989). test bed, ancl bugfixes with new tests were devel­ 6. R. M. Grow, "A Timed '[(>ken Protocol for Local oped. No deficiency was severe enough to pre­ Area Networks," Electro/H2, To ken Access Pro­ vent testing the prototype FDDl bridges ancl the tocols ( 17/3) (May 1982). development of more efficient algorithms fo r the 7. PCiar fella,D. Benson. and D. Sawyer, "An Over­ FDDI bridge products. vinv of the Common Node.: Software:· Digital Te chnical .Journal, vol. .). no. 2 (Spring 1991, Conclusion this issue): 42-52 The development of the FDDI data I ink and the chip 8. ,HAC Bridge .speufiwtions. ANSI/IEEE Standard set represents a major accomplishment ancltech ni­ 802.1 d (unapproved draft), 1'802.1 <1/DS (New cal breakthrough in the high-speed LANarea. Signifi­ Yo rk: The Institute of Electrical ancl Electronic cant contributions were made by Digital in the area Engineers, Inc., 1989). of FDJ)I MAC algorithms and protocols to improve the performance and robustness of the FDDI !.AN . 9. B. Stewart, W Hawe, and A. Kirby, "Local Area The FDDJ data link chips described in this paper Network Connection:· Te lecommunications are used in all members of Digital's FDDI product Magazine (April 1984). line, including bridges, wiring concentrators, and 10. H. Yang and K. K. Ramakrishnan. "Frame adapters. These products have benefited tremen­ Content Independent Stripping for 1oken . dously from the verification and test method Rings, . Proceedings o/ t/le ACII SJC,"COJLIJ '90 adopted. Digital has built on its knowledge and (September 1990). experience in systems, nc.:tworks, computer-aided 11. K. K. Ramakrishnan and H. Ya ng, "Frame design/simulation, and s<.:miconductors to provide Content Independent Stripping fo r To ken FDDJ design, development, and methodology. Again, Rings," Proceedings of the 151/J Co nference on Digital has shown industry leadership by produc­ I.ocal Computer Net ll 'o rks (October 1990). ing the FDDI chip �et and products. 12. Fiber Distributed Data lnteJfoce (IDDI) Media Acknowledgments Access Control. Ame rican National Standard. ANSI X:HW-1987 (June l9HH) The authors acknowl<.:dge the technical contri­ butions of Raj Jain, Brian Myrick. Chuck Lee. and 1:). .J. Hutchison. C. Baldwin. and I.l. Thompson. . K.K. Ramakrishnan. The authors would like to thank . Dc.:velopment of the.: l'DDI l'hysica I Layer:· .Jc.:rryHu tchison, Bill Cronin. and Raj .Jain fo r their Digital Te chnical.foumol. ,·ol. :). no. 2 (Spring rcvic.:w and several useful comments on this paper. 1991. this issue): 19-:l,O.

Digital Te ch nical jo urnal Vo l . .i So. l SjH ill,� I')') I i I Pa ul W. Cim:fella David Benson David S. Sawyer

An Overview of the Co mmon No de Software

To address mz aggressit•e fiber distributed data interfa ce (FDD!) program schedule and reduce the complexity of tbe concurrent det•elopment of multiple FDDI prod­ ucts. Digital det•eloped a common implementation of the FDDI station manage­ ment standard. Th is implementation, called the Co 111mon Node Sojtware. manages the phys ical and logical connections to the 100- megabit-per-secondfi ber-optic ring fo r Digital's FDf)fpmduct set. Including the Co11wwn Node Sojiware in each prod­ uct yields mnsistent behm•ior at the JDD! data fink and jJh)'sical fayers. Direct reuse of the softll'are reduces the del•etop111e11t and testing ejforts by prouiding a prot •en imptemen tat ion

The fi ber distributed data interface (FDDI) data link team. and interopcrahility testing among FDDI ven­ prov ides cl ients with a conncction lcss data trans­ clors. Finall y, the benefits of com mon code rea lized mission and reception service. An essential element by the project team are discussed. of this service is r<:!iable connection to the physical network. allowing clients to transfer intormation Background of the CNS Development across the network. Effo rt The Common Node Software (CNS) implc:mcnts When development eftorts for the DECbridge 500 and the parr of the FDDl station tilat controls ancl moni­ DEC:conccntrator 500 firmware began, the American tors connections within the FDDI network. To pro­ National Standards Institute (Ai\SI) FDDI MAC:, phys­ vide this service. CNS implcments the protocols ical layer (PHY). and physica l med ium dependent defi ned by the FDDI station management (S,\rts could result in diffe rent interpretat ions of In this paper, we begin with a discussion of the SMT protocols. of how Digital's FDDI chip set l'\ cnts lead ing to the development of C\S. Next, we works. and of the fu nctions data link software present a de tailed fun ctional descript ion of the should and should not pcrt{)l'Jn . Producing multi­ I'DDI station management and chip management plc. independent SMT implementati ons could lead services and specifics of the core and external to incompatible products that exhibit inconsistent libraries of the common code. We then describe behavior and are unable to communicate. the C'\IS dn·ciopmcnt effort. A su mmary of the test­ Du ring the c\'olution of the S:'vlT draft. the physi­ ing process fo llows. including details of the des ign cal connection management (I'CM) protocol pseudo­ verification test (I>VT) monitor. developed by the code defined by the draft changed often: some

I u/. .\t1. 3 _! .\jJri11g I'J'JI Digital Te e/mica( jo urnal An Ot •ert •ieu · of the Com111on Node Softtmre

The CNS team worked closely with Digital's rep­ COMPLIANCE TO THE ANSI FDDI resentatives in the ANSI X:H9S FDDI S,\lT working STATION MANAGEMENT group to develop the fu nctional requirements of tltc STANDARD software. As the SMT specification evolvecl, the htnc­ tional requirements were completed ami the devel­ opment of the software began. The end result is a set of reusable software I ibraries which prm ide SUPPORT FOR SUPPORT FOR INDIVIDUAL FDDI - - OTHER ANSI the functions necessary to implement the S.\IT pro­ PRODUCTS FDDI STANDARDS tocols ancl to manage the FDDI chip set on each FDDI product. This reusable code, called the Common Nocle Software, is shared by the DECbridge ')00 and COMMON NODE DECconcentrator ')00 products and also b\ more SOFTWARE recently introduced FDDI products. such as the DECcontroJier 700 adapter. COMPLIANCE TO THE DIGITAL SOFTWARE As the CNS design matu red, the alh·antages and NETWORK f---- c____ REUSABILITY benefits of a common code implementation ARCHITECTURE AND LICENSING SPECIFICATION became more apparent to the FDDI program team. Originally, the design constrained CNS to operate under the same operating system used in the DECconcentrator 'SOOand DECbridge 500 products. FDDI CHIP SET CNS was extended to support microprocessor and MANAGEMENT SERVICES operating system independence to accommodate fu ture FOOl products, thus facilitating the port a­ bility of CNS to other environments. CNS Figure 1 Functional Requirements Fulfilled This extended support allows a nriet\ of by the Common Node Software implementations, including host-based network control lers and firmware-resident drivers. Conse­ quently, Digital is benefiting by distributing the changes caused previous versions to be incom­ software externally through third-party licenses to patible. PCM is responsible for the management of promote rapid introduction of quality FDDI prod­ fu ll-duplex physical connections between two ucts to the expanding marketplace. FDDI PHY entities. The PCM pseudocode defines a synchronized bit-signaling communication pro­ Fu nctional Description cess between two connecting nodes to exchange connection information. If two nodes attempting This section presents details of the station manage­ to connect to one another execute incompatible ment standard services and the management of .'llT­ versions of the PCM pseudocode, these nodes will viees provided by the FDDI chip set. A hmction:ll not connect. block diagram of the Common Node Softvv :tiT is To avoid this scenario, and possibly others, a shown in Figure 2. decision was made to produce a common. reusable implementation of the SNIT protocols and FDD! Station ;l1anagement chip management. The initial goal of the C:NSpr o­ The station management standard Lkfi ne� the .s er­ ject team was to constrain the domain of possible vices used by every station in an FDDI ring to mon i­ SMT- related problems which could appear during tor and control both the station itself and the state the development of the FDDl product set. of the ring. The fu nctionality defined bv the stan­ Another important goal of the CNS project was dard includes connection management (C.\IT). ring compliance with both the SMT standard and the management (RMT) , and SM'T' frame-based sen ices. Digital Network Architecture (DNA) FDDI data link fu nctional specification. Compliance with Connection ,1/anagc/1/ent C.\lT is primaril1 respon­ the SMT standard could increase the probability sible fo r the maintenance of phYsical connections that Digital's FDD! products would intemperate to the FDDI ri ng. This involves initializing ami with those of other vendors. Compliance with the establishing connections between ph\ sica! l:t\ cr DNA architecture would guarantee interoperabil­ port (PHY port) entities and configuring the intl-r­ ity among Digital's current and future products. nal data path of a station. C\lT is cli\'illecl into the

Vo l. .! .Vo. l J'.J'.JI Digital Te ch nical jo urnal .)jning Fiber Distributed Data Interface

COMMON NODE SOFTWARE

DUPLICATE PACKET RING PURGER ADDRESS MEMORY ELECTION DETECTION

STATION MANAGEMENT RING MEMORY GATEWAY CONTROLLER

STATION TO SYSTEM - --- EXTERNAL MANAGEMENT MANAGEMENT : MANAGEMENT FRAME-BASED I I INTERFACE SERVICES I FAULT I MEDIA ACCESS I MANAGEMENT CONTROL I I RING AND I MEDIA ACCESS I I CONTROL I MANAGEMENT I CONFIGURATION : 1 __ I 1 SWITCH : I I: L---..----..J

i 6�����iloN ��---,-.;;J-L:--t====:::;-, : MANAGEMENT [ - I I PHYSICAL I I LAYER TO PHYSICAL I I -- MEDIUM I I COMPONENTS _:::::::: DEPENDENT L------J �-----..J LAYER

Figure .2 Functiona/ Biock Diagram of the Co mmon No de Sojtware

tl>llowing three areas: entity coordination manage­ orr is also responsible fo r topology control on ment, which coordinates the trace process and new connections. To pology control, implemented manages the optional optical bypass fu nction; con­ by CNS as part of the I'CNI pseudocode processing, figuration management, which manages the con­ enforc<:s a set of connection rules defined to pre­ figuration of the PHY and MAC entities within a vent the fo rmation of illegal ring topologies. The station; and I'C:\-1 . which manages the plwsical con­ ropology rules utilized by CNS are more strict than nection between a station s I'HY and the PHY of rhe the default rules suggested by the SMT stJnrm. attempt to locate a fa ulty MAC or data path. The sta­ The link error monitor checks the quality of the tion causing the break should fail its test and nor connection after ir fo rms by computing and moni­ rejoin the ring. Another value-added function in toring the link error rare ro determine if this rate CNS is the enhancement of the trace algorithms to is acceptable. If the rare fa lls outside the accept­ ensure proper termination of a trace even in the able range, the connection wi ll be terminated. A presence of simultaneous network reconfiguration. value-added fe ature of the link error monitor is Connection management is implemented in both error detect i on exceeding that requ ired by the hardware and software. CNS provides the Gv!T ser­ SMT standard.' vices that must be implemented in software but

\b/. _) .\'o. 1 .\jHing I')')I Digital Te c!J nical jo urnal An Oi'eruiew of the Co mmon No de Sojhmre

Ta ble 1 Connection Management Functionality

ANSI SMT Value Name Standard Added Description

Connection management Yes Manages physical connections and station configuration. Contains entity coordination management, configuration management, physical connection management, and link error monitor.

Yes Enhances topology management.

Entity coordinat ion Yes Coordinates the trace process and manages the optional management optical bypass function. Yes Enhances trace function to be insensitive to network reconfigurat ions.

Configuration management Yes Manages the configuration of the PHY and MAC entities within the station.

Yes Function is integrated within hardware.

Physical connection Yes Manages the physical connect ion between connected management PHY entities.

Yes PCM is implemented in hardware.

Link error monitor Yes Monitors active physical connection quality.

Yes Additional errors are recognized above those required by the standard.

are not performed by the ELM chip. For example, frames due to a fau lt condition on its data path. the PCM state machine, implemented in the EI.M The logical ring does not fo rm because the MACis chip, specifies the timing, state, and physical bit­ not able to receive its own beacons. Detecting the signaling used in the connection process. The PCM stuck beacon cond ition, RMT sends special MAC pseudocode, on the other hand, is implemented frames, called directed beacons. onto the ring to in CNS. This pseudocode is used to communicate notify other stations of the cond ition. After send­ connection information between neighboring PHY ing these directed beacons fo r approx imately 370 ports. Coordinating the PCM state machine and m i l lisecon ds, RMT tel ls CMT to initiate the trace pseudocode provides fu ll PCM functionality as fu nction. When the trace fu nction successfully defined in the SMT standard . completes, the data path fa ult is detectecJ and the Another example of CMT services provided by logical ring is fo rmed. CNS is the link error monitor. The EL'vl chip provides If two or more stations have identical MAC fa cilities to detect and signal the occurrence of bit addresses, the MAC protocols are adversely affected. errors on the link. CNS computes the link error rate If the ring becomes operational, these stations based upon the data from the ELM and determines with duplicate addresses strip each other's frames whether to sustain or to break marginal connections. from the ring, causing ring instabi lity and increased packet loss. If the duplicate stations are perform­ Ring Mmw�ement R:vlT, also implemented in CNS, ing ring initialization, however, successful ring monitors the state of the logical ! ink by using logi­ initialization may be prevented and the ring will cal link status info rmation received from the MAC not become operationaL sublayer. The info rmation is then passed on to net­ The RM T state machine can detect only the dupli­ work management, such as Digital's extended LAN cate condition of its own MAC address. When this management software (DECelms), or to the U.C sub­ condition is detected, RMT has the option of layer, fo r example, to report that the link is avail­ removing the MAC from the ring, changing its able for transmission service. address to a unique address, or fo rcing its MAC to This .\1AC information is also used by RMT to lose the claim process and allow the ring to detect and to resolve fa ults such as duplicate become operational. With the first option, a special addresses and stuck beacon cond itions. A stuck control frame, cal led a jam beacon, is sent directly beacon condition occurs on the ring when a sta­ to the dupl icate address to inform all stations with tion's ,'vlAC continuously transmits MAC beacon the dupl icatc address of the conc.l ition. LLC service

Digital Te cbuical journal 11!1 . .i No. J .\jJring 19'JI 4'5 Fiber Distributed Data Interface

associated with the MAC is disabled whi lc thc dupl i­ the use ofvendor-defi necl frames. This service pro­ catc cond ition is present. vides the capability for exercising new SMT frame­ R.\ITdupl icate address detection is complemcntcd based services. bY tile neighbor notification protocol. This proto­ 'fhes<: services are also useful fo r certain func­ col allows a station to learn both its downstrcam tions embedded within Digital's stations. These ancl upst ream neighbors' addresses. I3ytr ansmitting <:mbcdded fu nctions utilize the SMT frame-based a pniodic neighborhood information request frame s<:rvices and are summarized in Ta ble 2. CNS (\"IF)to its downstream neighbor. the protocol is makes use of the extended frame service by able to detect from th<: received NJF r<:sponscs that implementing the ring purger eJection protocol , its .\ lAC add ress is duplicat<:d on the ring. l.l.C: ser­ which supports the ring purger fu nction. The ring ,·icc associated with the MAC is disabled until the purger is one of several fu nctions defined by condition is removed. Digital's FDDI architecture that add value to the

SMT standard .2

Station .lhmap,ement fm 111e-fJused Se n•ices Station The ring purger election protocol is an algorithm management frame-based services include both impkmemed within CNS; the purging fu nction mandatory ancJ optional fu nctionality. The manda­ is implemented in the MAC chip. 'This distributed ton ft J nctiona litY includes the NIF. status informa­ algorithm clects one station on the FDDI ring to be tion frames. echo frames. request deni<:d frames. rhe ring purger. Candidate ring purgers, using the extended service frames. status report ing frames, SMT cxtemled frame service, send ring purger elec­ ami the as yet undefined resource allocation tion fr ames to a multicast address known only to frames. Par::�metermana gement frames arc defi ned stations participating in the algorithm in order to as optional fu nctional itv. communicate with each other. The station that

The s<:t of S.v!T fra me-based services supports either wins the claim process or has the highest l1igher-lcvel network management fu nctions which address becomes the ring purger. Once elected, a are not pan of s,vJ T. The information provided by ring purger enables the purging in the MAC chip and the sen·iccs helps network management to dcter­ sends out periodic "hello" messages to the ring. If m i ne the topology and state of the ring and to con­ these messages are not received after a period of trol the network. The status r<:port fra me servic<: time. the election process is repeated. Other error announces statlls information to network manage­ conrrol ancl recovery procedures are built into the ment. The optional parameter management frame algorithm to increase robustness. serYie<: faci litates remote manage ment of FDDI sta­ Another va[u<:-added feature of CNS is the SMT tions. The status informat ion frame service provides gat<:way protocol This protocol helps build ring station configu ration and operation parameters. maps on nerwork management consoles. A ring The echo frame service provides station-to-station. map is a database that can be used to build a graph­ loopback testing using S:viT fr ames. Request Llenicd ical representation of the network topology. The frames arc sent by a station in response to receiv­ map not on ly presents a visual image of the network. ing service requests that ar<: not understood or arc hut also displays characteristics about each node not implemented. Extended service frames allow on the ring such as its type and number of ports.

Ta ble 2 Embedded Functions Su pported by SMT Frame-based Services

ANSI SMT Value Name Standard Added Description

Duplicate add ress Yes Periodic NIF requests inform neighbor of station's test existence, get neighbor's address, and test for duplicate of this stat ion.

Yes Sending another NIF request to our own address improves coverage over ANSI required test.

Ring purger No Yes Digital 's ring purger rids the ring of no-owner frames election and frag ments. Ring purger election is controlled by a distributed algorithm.

SMT gateway No Yes Management may use any stat ion as an agent to query other stations for SMT information.

\iii .l .Vo. l .\jJrin.� I')') I Digital Te e/m ica/ jo urnal An 01•ert•ieu • (�(the Co 111num No de Sojiu•m·e

The S:VIT gateway within CNS uses status int(>rrna­ effecrivclv disabled the product unti I the storm tion and neighbor inhmnation request frames to ended. The: flo,.v control method was devised to solicit info rmation about other stations on the throttle events in such a situation. Once the occur­ network. Responses received include information rence rate of the event slows down, the throt­ about a station's configuration, its network address, tling stops. current counter values, and FDDI timer values. The gateway col lects these responses and returns them. Implementation Sp ecifics using the management protocol within the product, When the C;\IS development began. some impor­ to the host management station building the map. t<�nt questions arose. Many of the questions were By providing a well-defined and protocol-inde­ about FDDI itself. Most of the questions dealt with pendent interface to its clients, the SMT gateway issues concerning the structure of the reusable can be used with any network management proto­ software, fo r example, col . The first product to utilize the SMT gateway is • \Vhardef ines common code·, DECelms software.'

• \'V'h ar fu nctionality wou ld be common across all FDDJ Chip Management products' CNS manages the services provided by the fDDI • How do we handle common functionality that chip set, including the initialization of all config­ must be implemented on d iffcrent hardware plat­ urabl<.: chip parameters such as timer values and fo rms with different interfaces·; interrupt masks. Special chip modes, such as vari­ ous frame reception modes supplied by the .'v1AC, • What requirements. such as consistent interfaces. need to be placed upon external hardware and multiple loopb<�ck modes in the EL\-1 , and parity detection in the RMC, are also controlled by CNS. software' Software control of these modes is supplied by a set of inte rface fu nctions within CNS. Many of the Co mmon Code CMT Si\oiT- relatcd protocols, such as and the ring CNS is code written entirely in the C language, purger algorithm, are implemented in both the FDDI which can be executed on different hardware plat­ CNS. chips and CNS controls the operation of these fo rms without modifying the source code. This algorithms through its management of the chip set. common code includes a library of core functions. CNS also provides consistent FDDI chip fault man­ which arc completely portable. and a set of exter­ agement fo r all products using CNS. Compile time nal functions to provide services that cannot be and system initialization options allow rhe prod­ implemented in the same way on all products. The uct designers to specify. for each fa ult, whether it CNS core and external library interfaces with the should he considered fa tal or nonfatal to the system. remaining system firm•vare are shown in Figure :).

Also. fo r each event, a threshold is set indicat ing The CNS project team developed a coding stan­ the maximum number of times the event is to occur dard to facilitate a high degree of portabil ity. The over a predefined period before any action is taken. standard provides, for example, a portable set of When the threshold is reached and a fau lt is classi­ type definitions to ensure that a long integer is fied as fatal, CNS removes the station from the ring always :) 2-bits wide and a short integer is always and notifies rhe firmware kernel. The kernel is then 16-bits wide. Other type defi nitions specify the size responsible for fu rther action, such

In addition, the algo rithm can disable a recurring guage allows the assignment of variable names to event fo r up to one second. Th is provides flow con­ individual or strings of bits. Control/status register trol of fa ult events and allows other processing to con­ bit manipulation is easy to perform �Ising the com­ tinue. During lab testing of the DECconcentrator bination of structure and bit-field definitions. But 500 prototype, broadcast packet storms caused the ordering of bit assignments can change from receiver overrun conditions in the R.M C me mory compiler to compiler. Wh ile one compiler assigns control ler. An interrupt event is associated with bit 0 to a bit-field declaration, another may assign the overrun, so the event occurred continuously. bit 31 . Correctly mapping bit defi nitions in software using up all available processor time. This situation to their corresponding definitions in a hardware

2 I')') I Digital Te cbuical jo urnal Vo l. .l Nu. .\jJri11g 47 Fiber Distributed Data Interface

,------1 PRODUCT SYSTEM FIRMWARE OTHER FUNCTIONALITY: BRIDGING, MANAGEMENT. ETC.

SYSTEM SERVICES (SCHEDULER, BUFFER SYSTEM TIMERS. MESSAGE MANAGEMENT INITIALIZATION PASSING. ETC.)

WELL-DEFINED INTERFACES

EXTERNAL LIBRARY

,------, ,------, I I I FUNCTIONS I I I I I CORE I COMMON TO I I PRODUCT-SPECIFIC I LIBRARY I I FUNCTIONS I DIGITAL'S I : : I PRODUCT SET I I I I I I I 1------J l ______j

COMMON NODE SOFTWARE

------�

Figure3 Co mmon No de Software interfa ces

register cannot be guaranteed. Thus, the use of library also provides interrupt processing func­ unions and bit-field definitions was eliminated. tions fo r the MAC and ELM chips. The external library provides the interrupt service routines (ISR), Th e Core Library and these core functions are called from within ISH. The software is partitioned into two libraries con­ the Built within these interrupt processing taining core and external fu nctions. The core library functions are faci lities to manage chip fa ults, such consists of a set of completely reusable fu nctions. as parity errors or overrun conditions signaled by No sou rce code changes are necessary to execute the FDDI chips. this software on different products and hardware platforms. The Ex ternalLi brary The core library provides the major fu nctions of The external library is based on a well-defi ned set CNS, but relies on the externalli brary to provide of fu nctions such as allocating a transmit buffe r, operating system and hardware support. The core starting a software timer, or enabl ing the FOOl data provides a set of interface functions that manage path scrub function used to clear frame fragments the physical and logical connections to the FDDI from the ring. This library provides the direct data link. interface to the product's operating system, buffe r

The core implements many of the SMT protocols management services, and hardware configuration. such as any Gv1T not performed by hardware, dupli· Digital's implementation of the external library

Glte address detection, the SMT frame protocols, and is further d ividcd into two sublibraries. The first sub­ Digital's ring purger election protocol. The core library consists of fu nctions specific, but common,

I'J'JI 4H Vol 3 No. 1 .\jJring Digital Te cb 11icaljo urnal A 11 Overview of the Common Node Software

to Digital's FOOl product set and is completely por­ support fo r various configurations and fo r the exe­ table throughout the set. The other subI ibrary of cution of configuration-based protocols. For exam­ fu nctions is specific to each fDDl product and is not ple, in the DECbridge 500 product, the CNS data portable among them. These fu nctions deal mainly structures consist of one Station, one Li nk, and one with special hardware access and configuration Phy connected to reflect the single attachment sta­ management, such as inserting the MAC onto the tion (SAS) architecture. The DECeoncentrator 500 physical data path internal to the DECconcentrator firmware has one Station, one Link, and 12 Phy data 500 product. The interfaces to these functions are structures connected to fo rm the configuration of the same fo r all products, but diffe rent hardware the station. A six-port concentrator with a manage­ designs or the limited fu nctionality in a product ment board is represented by one Station, one Link, may require distinct implementations. For exam­ and six Phy data structures. An eight-port concen­ ple, the OECconcentrator 500 device has an inter­ trator without a management board does not con­ nal data path that interconnects the MAC and PHY tain a r\1.AC, and, thus, is configured as one Station entities within the product. Logic surrounding the and eight Phy data structures. MAC chip can be used to insert and remove the MAC from the data path or to bypass the MAC. The exter­ Th e CNS Development Effo rt nal library for the DECconcentrator 500 firmware The actual design and development cycle for CNS contains two fu nctions to insert and bypass the covered a six-month period. During this time the r\1.AC. In contrast, the DECbridge 500 product's MAC: SMT draft standard went through many revisions; is always on the data path and does not need to the concept of licensing the CNS code was intro­ bypass or insert the MAC; therefore, the firmware duced; and Digital's FOOl chip set underwent a does not contain bypass or insertion functions. second-pass design cycle. To facilitate consistent memory access, low-level The licensing effort required rethinking and packet memory fu nctions enforce network byte making adjustments to the partitioning and struc­ order in SMT frames. Network byte order defines ture of the design to accommodate several layers the order in which bytes are transmitted and of interfaces and support fu nctions in CNS. These received. These fu nctions perform 16- and 32-bit requirements resulted in an additional fo ur to six read and write operations when building and pars­ weeks of effort, but yielded the benefits of a code that ing frames. Other product-specific functions pro­ could be used outside Digital and a more generic vide access to designated packet memory locations design to accommodate fu ture product designs. fo r generating special MAC-level control frames Of all Digital's FOOl chips undergoing second­ cal.led directed beacons, which are used by ring pass design, the ELM chip required the greatest num­ management in special fa ult situations. ber of changes. The majority of these changes were due to the major redefinition of the physical con­ Data Structures nection management portion of the Ai\JSI SMT draft.' Supporting both libraries is a set of three data struc­ To minimize risk to the chip development effort, tures, Phy, Link, and Station, that store state, configu­ simulation of the connection management portion ration, and counter attributes information pertaining of the CNS code was performed on Digital's logic to CNS. The Phy data structure contains info rmation simulation system (OECSlM) rest bed . This testing about a single PHY port. The link data structure utilized one of the same test beds constructed reflects the state of the logica l link associated with by the ELM chip designers configured with two the MAC, while the Station data structure main­ ELMs connected together-' This test bed utilized the tains inform ation about the ge neral state of CNS. behavioral chip models in order to speed execu­ Al l information about the state of CNS and SMT is tion. Additional routines we re written to emulate contained within these data structures. Other firm­ resources normally provided by the operating sys­ ware agents residing within the products, such as tem (e.g., timer services). Roughly one week was extended LAN manage ment software responders, spent in simulation, two days of which fo cused on need only to look in a central location for manage­ developing and debugging the environment. Testing ment information pertaining to SMT. Global variables included the initialization of a good connection as defined by CNS fo r its sole use are kept to a minimum. we ll as connections that resulted in topology rejects, The structures are linked to reflect the actual link confidence test fa ilure, and link error monitor configuration and clara path of the MAC and PHY fa ilure. No bugs were fo und in the ELM design, entities within the station. This linking provides easy however several coding bugs were discovered.

49 Digilal Te clmical]ounral 11!1.. i Nfl . .! Spring 1991 Fiber Distributed Data Interface

Later, when the first DECconu:ntr:ltor 500 hard­ To faci litate testing, the CNS team developed a ware w:1s available in the l:1b, :1 nd the operating tool rckrred to as the design verification test (DVr) system services were debugged and available, CNS monitor. This tool provides a detailed view into PHY code required only one-half day to debug the operation of CNS as well as automated tests for before becoming operational. Thus, prior simula­ CNS interface functions. The tool also has the capa­ tion of the code was clearly beneficial. bility to insert fa ults into the ring and exercise The first product that utilized CNS was the many of the SMT protocols. DECconcentrator 500 firmware. The integration, The DVT monitor has two components: a con­ debug, and design ve rification process spanned an nection to a universal asynchronous receiver/ eight-week period. For subsequent products. the transmitter (UART), which provides serial commu­ duration of this phase was greatly reduced: two nication between the product and a display termi­ weeks fo r the DECbridge 500 firmware and one week nal; and monitor software, which is layered on top fo r the DECcontroJJer 700 adapter. This reduction of CNS. Thus, the monitoring and managing of CNS in time was, of course, due to the reuse of the core is achieved by out-of- band access via the UART. This and external libraries. Only a small subset of CNS access is necessary to perform testing of ring fault was unique fo r each product; hence, debug time conditions. The monitor software is run at a lower was minimal. Also, as experience was gained in ver­ priority than all other system components to leave ification testing, and the related tools improved, the system timing or operation unaffe cted. This the test process became more efficient. software provides both nonintrusive or "peek" and invasive or "poke" management capabi lities. Pass­ Te sting the Common Node Software word protection on a login screen prevents unau­ One fo rtuitous advantage to the structure of CNS is thorized users from disrupting the network. To that the core functionality only needs to be tested usc the tool, one must log on to an FDDI product exhaustively on a single platform. The partitioning running the test software. of core and externalfu nctionality and the external As a nonintrusive tool, the DVT monitor provides requirements to which each product environment passive monitoring of the network status and related must adhere yield this advantage. Thus, the only events. The tool provides rea l-time monitoring of testing necessary on an individual product is the all physical (or port) and logical (or MAC) connec­ initialization of C:NS and the external interface tions in the product. Status windows continuously between CNS and the system firmware. These prod­ display the state of all physical and logical connec­ uct dependencies include S1'

'50 Vo l. ; No. 2 .\j1ri11g 1991 Digital Te e/m ica/ journal An Ouerview of the Co mmon No de Software

FDD! ring and made it simple to characterize the well as to correct operation under both normal and responses of the products to a wide range of net­ aberrant network conditions. work traffic conditions. Connection management testing covers the physical connection management and configura­ FDDI Interop erability Te sting tion management processes. PCM testing covers lnteroperability testing among the FDDI vendors is the bit-signaling and connection initialization important fo r many reasons. Comprehensive inter­ algorithms, the link confidence test and link error operability testing among the vendors both rein­ monitor, and verification of the connection matrix fo rces the correct interpretations of the standard defined in the SMT draft. CFM testing verifies the and performs the more immed iate goal of verifying correct operation of the reconfiguration scrub and the correctness of the implementations. The proper MAC insertion funct ions. operation of all stations on the network both in Ring management testing covers dupl icate normal operation and in response to fault condi­ address detection, including stuck beacon detec­ tions is necessary if the commercial marketplace is tion and recovery, directed and jam beacon initia­ to fully accept FDDI. tion and reception, and the trace fu nction. Other The nature of a ring topology requires the active miscellaneous testing monitors the abusive use of participation of each station up to the MAC sublayer. restricted tokens and extended service frames. Each FDDI station must establish and maintain Frame-based testing covers all required SMT frame physical connections and repeat frames without protocols. These protocols are tested extensively error. If a single station does not repeat frames, fo r compliance to the SMT draft. Parameters within ring connectivity is lost. The X3T9.5 standards com­ the frames are examined fo r consistency and cor­ mittee anticipated fa ults that prevent normal ring rectness. For example, all timer values presented in operation and devised algorithms to deterministi­ SMT frames are verified to be in two's complement cally resolve such conditions. The addition of fault fo rm, and all canonical addresses are correctly con­ recovery schemes devised fo r the FDDI system, verted to FDDI most significant bit order. while necessary to guarantee proper operation of the network, made the standard more complex. Results of /n teroperability Te sting Correct implementation of these complex protocols The interoperability testing uncovered problems and distributed algorithms is essential to ensure in many vendors' implementations, including that one vendor's implementation will operate cor­ Digital's.; Nl any of these problems resulted from rectly on the same ring with other vendors' imple­ inconsistent interpretations of the SMT draft; others mentations-especially in the presence of fault were due to incomplete implementations that did conditions. not support some functions defined in the SMT Every vendor must make certain that its imple­ draft; and still other problems could be attributed mentation adheres to the fu nctionality as defined to changes in the SMT draft overlooked by some by the SMT standard. Digital encouraged coopera­ implementations. As a result of the testing, many of tive testing among the vendors and participated these problems have been fixed, and the number of in testing with many vendors at Digital's FDDI interoperability problems within FDDI networks development center in Littleton, Massachusetts, at has been reduced. customer sites, and at other vendors· locations. Conclusion In teroperability Te st Methodology The Common Node Software provides the FDDI A test plan was developed originally fo r design ver­ project with a very flexible and stable implementa­ ification and, later, fo r interoperability testing tion of a major portion of the FDDI data link. The between Digital's FDDI product set and products initial investment in time spent on development from other vendors. The test plan concentrates was longer than that expected fo r independent soft­ mainly on the interoperabil ity of the FDDI data ware development efforts but is justified by the link, defined by the FDDI PHY, MAC, and proposed long-term benefits of common code. Independent Pi'viD and SMT standards. development efforts fo r the DECconcenrrator 500 The plan covers connection management, ring and DECbridge 500 products, fo r example, proba­ management, and SMT frame-based services and func­ bly would have taken Jess time. Each design would tions defined in the SMT draft standard. The intent be based upon the hardware design and system of the plan is to verify plug-and-play capability as requirements of each product. Independent designs

Digital Te ch uicaljourual Vf J/. .i NfJ. 2 Sf; ring 1991 51 Fiber Distributed Data Interface

would reduce the amount of software developed of the common code is a product of their ideas. fo r each product because the designers wou ld not Appreci at io n is also extended to Bi ll Cronin, who have to address portabil ity issues, such as generic was instrumental in expanding the scope ami

interfaces to the operati ng system or hardware. effectiveness of the interoperabi li ty testing. Finally, But these two products were the first of a new the authors would I ike to acknowledge Henry Ya ng network architecture, and the station management fo r his careful and constructive review of the paper. draft was not complete during product develop­ ment. Thus, independent implementations would References have resulted in a higher bug rate, with each prod­ FDDJ Station Manage ent, Draft Proposed uct exhibiting its own set of behavior at the SMT 1. m American National tand a rd, X3T9.5/94-89, level. future updates and revisions to the SMT stan­ S REV 6.2 (May 18, 1990). dard wo uld have resulted in i nde penden t revisions of each product"s firmware and. possibly, a new set 2. H. Ya ng, B. Spinney, and S. To wning, "FDDI Data of problems. With CNS. only one source needs to Link Development," Digital Te chnical jo urnal, 1991 , be changed and tested. vol. 5, no. 2 (Spring this issue): 31 -41 The fDOJ standard promotes multivendor inter­ 3. ). Hutchison, C. Baldwin, and B. Th ompso n, operahi l i ty, so that independent products can com­ " Developmen t of the FOOl Physical Layer," municate effectively in a heterogeneous FDDI Digital Te chnical jo urnal, vol. 3. no. 2 (Spring network. The development of CNS s ignificantl y 1991 , this issue) 19-30 increased i nteroperabi liry between Digital's prod­ ucts and those of other vendors. 4. B. Sweet, "DECelms-Managi ng Digi tal's FDDI The advantages of reusable software that were and Ethernet Extended Local Area Networks,"

realized by the proj ect team can be summarized as Digital Te chnicaljournal, vol. 3, no. 2 (Spri ng fo llows: 1991, this issue):76-84.

• The major design of the software needs to be '5 D Benson. P. C iarfella , and P Hayden. "FOOl­ done only once. l brary and a well-defi ned A core i Meet i ng the ln teroperabi lity Challenge," Pro­ external interface are provided. When a new ceedings of the IT:EE Fifteenth Conference or1 product developed. only the spec f c imple­ Networks (October 1990). is i i Local Computer menration of the external interface needs to be supplied.

• Test and verification time is sign ificantly reduced.

Only the interfaces to CNS and the system depen­ dences must be rigorously tested with the tl<:-sign of each new product.

• The bug rate is red uced significantly. Each new rroduct uses the pretested and proven core l. ihrary.

• The software requires little maintenance. Si nce the core library is stable. den:lorment only needs to be performed on the externalli brary.

The CNS development project was the design team's introduction to reusable software. We have probably not done everyt h i ng in the best possible way, bur the success of the project and the time and effort saved have convinced us of the benefits of reusability.

Acknowledgments

The authors wo uld like to thank Peter 1-Ja�·ti<:n and Herman Levenson fo r thei r technical contributions

to the development of CNS. M uch of the structure

I'J')/ Digital Te ch uical journal ltd. .> , Yo. 1 .\jJri11g Robert C. Ko chem james S. Hiscock Brian T. Mayo

Development of the DECbridge 500Pr oduct

The DECbridge 500 product connects Ethernet/802.3 local area networks (fANs) to fi ber distributed data inteJface (FDDI) fANs and is, therefo re, a fu ndamental element of an extended LAN.Developers of this product encountered mm�Ji technical htn·dles. The higher data rate and token ring topology inherent in the FDDJ technology impose several demands on any bridging product connected to an FDDJ LAN. The differences in fo rmats and size of fra mes on the two types of fANs introduce fu rther requirements. The development team met these requirements and delivered a high­ performance product that provides seamless integration of both LAN ty pes.

Bridges are essential to the creation of extended Throughout this paper, the expression Ethernet/ local area networks (IANs) because they provide 802.3 is used to identify such I.ANs and to distin­ transparentfo rwarding of traffic between adjacent guish them from 100-Mb-per-second FDDI I. ANs. LAl'\ls.' Traffic may be fo rwarded to or from individ­ The terms Ethernet,80 2.3. and fi)[)J are used when ual destinations, to groups of destinations (multi­ discussing the specific MAC frame fo rmats. cast), or to all destinations (broadcast). Bridges only fo rward traffic destined fo r other I.ANs; local traffic Sy stem Description is confined to its home LAl'\1 . Two typical extended LAN applications involving One important fu nction of bridges is the ability, DECbridge 500 devices are shown in Figure J. The under network management control, to block traf­ backbone application employs an FDDl LAN to pro­ fic of selected protocol types or traffic from specific vide a high-bandwidth interconnect of multiple sources. Restricting unnecessary traffic, especially Ethernet/802.:) I.ANs. The DECbridge '500 deviceis multicast or broadcast, significantly improves the the point of connection between the Ethernet/ utilization of IAN bandwidth. 802.3 LAN and the FDDl backbone IAN. In the work In this paper we first discuss the role of the group application, FDDI l.ANs provide localized DECbridge 500 product in an FDDI and Ethernet/ connectivity of users, such as DEC:station 5000 802.3 extended LAN and outline the design of the workstations, that have high throughput require­ bridge. We then describe the operation of the bridge ments. File servers and other common resources by tracing the flow of LAN traffic through it. This may also be part of the local FDDI !AN. Here, the description gives insight into many of the complex role of the DECbridge 500 product is to provide a tasks that a bridge must perform to connect two path from the local work group to other parts of dissimilar fANs. Key points of the development the extended IAN via Ethernet/H02.3 lANs. methodology are also presented. In either application, the bridge must perform the fo llowing fu nctions: DECbridge 500 Design Considerations

The DECbridge 500 device serves as the point of • Forward traffic between nodes residing on two connection between a new family of LAN prod­ different I.ANs ucts based on the fiber distributed data interface • Prevent (i.e., filter or not fo rward) traffic between (FDDI) technology and a large installed base of nodes on the same side of the bridge from get­ Ethernet/802.3 L.ANs. The DECbridge 500 product ting to the !.AN on the other side of the bridg<: must meet the requirements of both LANs to pro·

vide a smooth migration path fo r Digital's cus­ • Be responsive to host-based network manag<:· tomers. Note that Ethernet and 802.3 have media ment, prov ided by, fo r example, Digital's extend<:d access contro.l (MAC) frame fo rmats that may be LAN management software (DECelms) and Digital's used on the same tO-megabit (Mb}per-second IAN. management control center (DEC:mcc) product

Digital Te clm ical journal Vo l. 3 No. 2 ,\jJrin,� 19')1 Fiber Distributed Data Interface

BACKBONE APPLICATION FDDI RING TO OTHER EQUIPMENT

DECBRIDGE 500

DECBRIDGE 500

DECBRIDGE 500

WORK GROUP APPLICATION

TO OTHER WORK GROUPS

DECSTATION DECSTATION DECSTATION DECSTATION DECSTATION DECSTATION 5000 5000 5000 5000 5000 5000

WORK GROUP A WORK GROUP B

Figure 1 Extended LANAp plications of the DECbridge 500 Product

• Be a proper participant as an end station on other two processors' subsystems. The AP executes both LANs to which it is connected much of its own program directly from the non­ volatile memory, although some high-performance Interact with other bridges in the topology of • operations are executed from static RAM. the extended LAN to prevent redundant paths The DECbridge 500 device may have an entire or loops' new operating program downloaded over the network and stored in the nonvolatile memory. Hardware Description This allows rapid updates of functionality without Figure 2 shows a block diagram of the DECbridge the need to perform a hardware upgrade on-site. 500 hardware design. The applications processor Program updates are received via either of the (AP), a subsystem based on a 68020 microprocessor, attached LANs and stored in an area of RAM referred performs initialization ami maintenance of the to as the "landing pad." The AP then transfers the bridge hardware as well as some steps involved in new program into the nonvolatile memory and processing frames. The AP also acts as the manage­ initiates a firmware reset. ment entity fo r the bridge. The FDDI and Ethernet/802.3 chip sets and some The operating programs fo r the AP as we ll as two analog interface circuitry provide connection to other processors, the queue manager and the trans­ the two LANs. The bridge represents a single lation processor, are stored in a nonvolatile elec­ attachment station (SAS) on the FDDl ring. On trically erasable programmable read-only memory the Ethernet/802.3 side of the bridge, switch­ (EEPROM). At initialization. the AP distributes the selectable ThinWire ancl attachment unit interface programs to random access memory (RAM) in the (AU!) connections are provided.

54 Vo l . .i .Vo. 2 S{Jri11g 1991 Di/!,ilal Te chnical jo urnal Deuelopment of the DECbridge 500 Product

APPLICATION PROCESSOR CONTROUDATA BUS r------1 I I 1 I I I I I I 1 ADDRESS APPLICATION : PROCESSOR TABLE : I I I I I I I QUEUE I -- I MANAGER I I I I I ------1 �------�------� I I I I I I I I TRANSLATION FDDI PACKET - '- ETHERNET/802.3 ETHERNET/802.3 FDDI CHIP SET PROCESSOR AND � MEMORY DIRECT MEMORY f---. PACKET MEMORY f-+- CHIP SET ACCESS I TO IFDDI I TO NETWORK ETHERNET/802.3 NETWORK

KEY:

DATA FLOW CONTROLS

Figure 2 Block Diagram of the DECbridge 500 Hardzl 'are Design

Each chip set checks every incoming frame fo r wide by 14 inches deep and may be rack-mounted

integrity. Also, some rudimentary identity, or or installed on a tabletop. It ope rates over the range address, tests are applied. Frames that meet the of 100 to 240 voltage AC (VAC) at 50 or 60 hertz (Hz). integrity and identity requirements are then placed An exploded view of the bridge is shown in in a packet memory. The bridge maintains a table of Figure 4. The electronics is implemented by the fo l­ learned MAC addresses. The table contains data fo r lowing four logic modules: each address that is used to decide if a frame should • AP, the applications processor be fo rwarded or filtered. The queue manager is a QM, the queue manager sub yste including the sub-system dedicated to checking each frame • s m learned address table received in FDDl packet memory against the infor­

mation contained in the learned address table. • Fl, the FDDI chip set and the FDDI packet memory

Based on this info rmation, the que ue manager • NI, the Ethernet/802.5 chip set and packet mem­ decides whether to filter the frame , fo rward the ory and the translation processor frame to the Ethernet/802.3, or deliver the frame to the bridge entity fo r action. The FDDI and Ethernet/802.3 l.AJ'\Jsem ploy dif­ ferent data link protocols. The translation proces­ sor, a second 68020 subsystem, examines frames to be fo rwarded from one side of the bridge to the other. Each frame is reformatted to the appropriate outbound prowcol and moved from the incoming packet memory to the outbound packet memory.

The two chip sets examine their respective packet memories fo r outbound frames ancl transmit them onto their LANs.

Physical Description The DECbridge '500product is shown in Figure :). The hardware is approximately 7 inches high by 17 inches Figure 3 DECbridge 500 Product

Digital Te ciJ nical jou rnal Vi;/. 3 No. 2 .\jJring J')')! '5'5 Fiber Distributed Data Interface

BRIDGE ENCLOSURE'

BACKPLANE I 54- 1 9377-01

70-27394-01

OM-MODULE 54-19371-01

-NOT A FIELD-REPLACEABLE UNIT

Figure 4 DECbr idge 500 Assembly

External signal connectors are located on the several challenges beyond those encountered in front euge of the two network interface cards, FI and previous bridges that connnect similar Ethernet/ Nl. Each module has light-emitting diodes (LEOs) 802.3 LANs. The principal sources of these new fo r various status functions and also diagnostics. In challenges were: add ition, the AP has a bank of switches fo r setting • Higher data rates on the FDm LAN. Ethernet/ certain bridge operating functions. 802 3 operates at lOMb per second and has a ·rhe power and packaging were designed to sim­ minimum MAC frame size of64 bytes. The max­ plify the swap out of field-replaceable units (FRUs). imum Jrame arriva l rate is 14,880 frames per The fo ur logic module FRUs can be replaced through second (fps). FDDI operates at a rate of 100Mb the front of the box, without opening it. I3ytakin g per second and has a minimum MACframe size out only two screws, the outer shell of the case can of 17 bytes. The maximum frame arrival rate is be removed. This gives access to the thn:e other FRUs, 446.429 fps, a rate 30 times greater than that of namely, the power supply, the passive backplane, Ethernet/H02 3. and a fan assembl y. The five-sided design of the outer shell results in a product that is mechanically • Different frame fo rmats. Ethernet, 802.3, and FDDI strong and provides shield ing from electromag­ have diffe rent MAC frame formats. Traffic enter­ netic and radio frequency interference (EMl/RFJ). ing an FDDI LA N from an Ethernet/802.3 IAN must be properly translated to an FDDI frame Op eration fo rmat. This translation must be performed in As mentioned previously, the DEC:bridge 500 device such a way that passage through a second bridge fo rwards traffic between two diffe rent LAN types. back to a different Ethernet/802.3 LAN results in Consequently, the product development team faced a frame that recovers its original frame format.

Vo l. J No.2 .\jJring 1'-J

• Different frame sizes. Ethernet and 802.3 diffe r Receiving FDDI Fra mes from in both maximum and minimum FDDI The FDDI chip set in the bridge pl aces all received frame sizes. frames shorter than the Ethernet FDDI frames in the FDDI packet memory on the receive and 802.3 minimum must be padded. FDDI queue. Frames in the FDDI packet memory can be frames longer than the Ethernet and 802.3 maxi­ accessed by subsystems in the bridge by using a vir­ mum cannot be fo rwarded, with the exception tual address method. A page table memory is used of special protocol types, which must be broken to assign a physical 512-byte bu ffe r to each of 16K into multiple, smaller frames. virtual buffers. Queues contain sequential sets of virtual buffers. Data frames are ··moved" from one Objectives queue to another by moving the virtual address The bridge can only fo rwa rd frames from the FDDI buffe r pointers from one queue to another. LAN to the Ethernet/802.3 LANat the maximum rate Frames received from the FDDI LAN may be as accepted by that LAN , i.e., 14,880 fps. Ye t the arrival long as 4500 bytes. Frames longer than 512 bytes are rate of frames from the FDDI LAN may be in excess chained, that is, stored in multiple buffe rs. Each of 440,000 fp s. The incom ing frames consist of an buffe r has an associated descri[HOr longword con­ unknown mixture of frames that need to be for­ taining status info rmation about the frame such as warded, frames directed to the bridge itself, and error conditions, frame length, and flags indicating frames to be discarded . the start and end of multibuffe r frames. The abi I ity To comply with Digital's bridge architecture spec­ of the bridge ro chain small buffe rs to handle frames ification and the IEEE standard 802.l d fo r bridges, of various sizes increases the efficiency of the packer the bridge must examine all incoming frames.' memory by minimizing the amount of unused buf­ It must identify, set aside, and process frames of fer space. (Statistically, IAN traffic has a higher con­ each protocol type directed to itself, in the order tent of shorter frames.) Thus, more buffe rs are made received . To meet product performance require­ available to handle bursts of traffic. Additional ments, the bridge must be able to fo rward frames information about buffe r status is contained in the ar the fullEther net/802.3 rate. A best effort must be page table memory. This info rmation is generated made to buffe r frames received in bursts exceeding and used by the queue manager and the translation that rate. Frames should not be erroneously dis­ processor subsystems. carded. Results of this compliance visible to the network user are: Queue Ma nager Process

• Transparency. Nodes across the extended LAl'./ oper­ The queue manager subsystem operates on all ate as if they were connected to the same LAN. frames in the receive queue to determine if they should be discarded, forwarded to the Ethernet/ • Stability. The paths in the LAN remain constant 802.3 or received and processed by the bridge yet can reconfigure around equipment changes LAN, management entity. Discarded frames are returned with a minimum loss of connectivity. Frames to the receive queue; the remaining frames are are not duplicated ; nor are they received out of order. placed on the fo rward or bridge queues. The queue manager constantly makes updates to the table • Manageability. Network management can always of learned addresses based on source addresses observe and control the components of the observed on the FDDI LAN. extended LAN . The queue manager's operational decisions are The operation of the DECbridge 500 device is based on the fo llowing data: best described by examining the progress made • The frame descriptor containing assorted status through the bridge by frames received from the information such as transmission errors and FDDI LAN. Tracing this flow of traffic also gives frame length insight into many of the challenges faced by the product's development ream. The subsystems that • The frame co ntrol field specifying the type of process these frames and the flow of frames through frame logical queues in these subsystems are shown in

Figu re 5. The operation of the subsystems as the • The type and quantity of frames previously frames progress through them is described sequen­ received (used to prevent a flood of any one tially in the fo llowing sections. type of frame from blocking out other types)

Digital Te chnical journal VtJ!.. i No. 2 Sp ring 1991 57 Fiber Distributed Data Interfa ce

QUEUE MANAGER SUBSYSTEM TRANSLATION PROCESSOR

TABLE ADDRESS TRANSLATION APPLICATION PROCESSOR LOOKUP TABLE RAM ENGINE ENGINE I I I I I I QUEUE MANAGER GAB I GAB ENGINE I r- I I I

FORWARD FORWARD QUEUE r-----1 QUEUE � RECEIVE � QUEUE -I BRIDGE MANAGEMENT TRANSMIT I � QUEUE f----.1QUEUE � QUEUE �I � I I FREE SPANNING QUEUE -1 TREE QUEUE � y 1- PACKET MEMORY ETHERNET/802.3 PACKET MEMORY FOOl

FOOl ETH ERNET I CHIPSET I CHIP SET I i � � FROM FDDI LAN TO ETHERNET/802.3 LAN

KEY

GAB GATE ARRAY BURSTER

500 Figure 5 Dl::Cbridge Sub�ys tems Flow Diagram

• A learned database containing addresses indi­ The table address RAJ\'1 and the TLU are key com­ MAC cating on which side of the bridge each is ponents of the queue manager. The RAJ\'1 contains a located and special filtering status int()rmation table of up to !6K 4H-bit addresses. Each address assigned ro each address by network management also has status bits that determine what action the bridge should take when a frame's source or desti­

The gate array bu rster (GAR) allows the queue nation address matches a particular address. The manager to access the FDDI packet memory. This TLU is an ASIC with a port that is a slave to the a pplication-specific integrated circuit (ASIC) is a queue manager engine. The queue manager engine specialized direct memory access (DMA) device. It inputs an address to the TLU which scans the is capable of moving selected fields or large sec­ lv\..\1 fo r that address. If the address is found, the tions of frames into or out of FDDI packet memory TLU presents that status to the queue manager pro­ 'fhe objects may be moved either into internal cessor. Otherwise, the TLll gives the queue man­ holding registers fo r examination by the queue ager processor a programmable status indicating manager engine or directly to destinations such as whether to fo rward or to discard the frame. A sec­ registers in the table lookup engine ClUJ). Note ond TLll port allows the Tl.ll and the table address GAB that the used in the queue manager subsystem RAM to serve as slaves to the AP. Thus, destination is the same device used in the translation proces­ address fi ltering for traffic received from the sor, which is discussed later in this paper. These Ethernet/H02 .:)LAN and table maintenance can be two subsystems have many simi lar requirements. performed. but each also has unique requirements. Using one To keep up with the packet arrival rate, the c;AB design fo r both subsystems reduced the over­ queue manager subsystem makes extensive use all development effort. of pipclining. The queue manager engine operates

.\j>rin,� I'J'JI Digital Te ch nical jo urnal 58 llr>/. 3 No. .! Deuelopment of the DECbridge 500 Product

concurrently on six packets. The TLU unit per­ track of the number of fo nvarded, FDDJ SNIT, bridge fo rms three searches concurrently: one each fo r management, spanning tree, and error frames. the source and destination addresses on FOOl The queue manager also has counters that sum­ packets and one source or destination search on marize its activity. These counters are periodically Ethernet/802.3 packets. dumped to the AP and are used to calculate LAN uti­ lization statistics required by network management. Discarding and Keeping Fra mes The decision to discard a frame is based principally on the frame's Translation address or its contents. The fo llowing are typical of Bridges operate at and below the data link level in frames that are discarded: the seven-layer International Standards Organiza­ tion (ISO)/Open System Interconnection (OS!) ref­ • Frames destined for nodes that the bridge rec­ ognizes as not on the Ethernet/802.3 side of the erence model shown in Figure 6. The data link LAN. Also, network management may specify layer is divided into a lower MAC sublayer and an addresses to be discarded regardless of location upper logical link control (LLC) sublayer. The LJ.C in the topology. protocol is specified in ANSI/IEEE standard 802.2.'

• Frames of either a reserved or undefi ned frame control type. APPLICATION

• Frames that are either too long or too short. PRESENTATION

When a frame is discarded, its buffe rs are returned SESSION

to the end of the receive queue by reassigning them TRANSPORT

in the page table. NETWORK Frames that are kept are placed on either the fo r­ DATA LINK LOGICAL LINK CONTROL ward or bridge queues. Frames ultimately destined { ------LAYER MEDIA ACCESS CONTROL fo r the Ethernet/802.3 LAN are placed on the fo r­ ward queue. Frames placed on the bridge queue, to PHYSICAL be processed interna.l ly by the bridge, are of the fo llowing types: Fig m·e 6 150/051 Seven-layer Reference Model • FDDI station management (SMT) frames

Digital's extended LAN management software • When fo rwarding frames from one LAN to (DECelms) frames or maintenance operation another, the DECbridge 500 device converts the protocol (MOP) frames outgoing frame to the MAC frame fo rmat of that

• Spanning tree frames, containing messages used LAN. This process is called translation. Also, when to determine the network topology and turn incli­ a frame is generated by the DECbridge 500 product vidual bridge ports on or off to eliminate path on either LAN, the data link frame fo rmat of that redundancy IAN is employed. By performing translation, the DECbridge 500 Frames containing errors • product complies with the IEEE 802.ld require­ Frames placed on the bridge queue not fo rwarded ments fo r transparent bridging. This enables end to the Ethernet/802.3 LAN.However , after receiving nodes to communicate across the extended LA N as and processing these frames, the bridge may gener­ if the nodes are directly connected to the same ate one or more frames on either or both LANs. For LAt'\J.An alternative to translation, called encapsu­ example, received SMT frames are never forwarded, lation, is possible, but it does not comply with the but a given SMT frame may cause the bridge to IEEE 802.ld requirements. Further, using encap­ transmit additional SMT frames on the FDDI LAN . sulation puts restrictions on the configuration of the network. Counters Each frame type is guaranteed a mini­ mum amount of processing time by the bridge. If at The Process Ethernet, 802.3, and FDDI have dif­ any time the bridge holds too many of any one frame fe rent MAC frame fo rmats. When Ethernet or 802.3 type, it discards further frames of that type. The frames are bridged to an FDDI LAN, they are refor­ queue manager uses allocation counters to keep matted to the FDDT MAC frame fo rmat. The original

Digital Te clmicaljournal vb l. 3 No. 2 Sp ring 1991 59 Fiber Distributed Data Interface

MAC type (Ethernet or 802.3) is indicated by setting bytes. Frames received from the FDDI ring that are info rmation in the LLC header. If the frame passes longer than 1518 bytes after translation are dis­ through a second FDDI-to-Ethernet/802.3 transla­ carded with the exception of frames discussed in tion at another bridge, the LLC information is used the text that follows. to determine ifthe bridge should translate the frame The internet protocol (IP) is a widely used, into Ethernet or 802.3 MAC protocol. IEEE standard network-layer protocol. Nodes on FDDI rings may 802.1 defines a mechanism fo r translating Ethernet generate IP frames longer than the Ethernet/802.3 frames into an IEEE 802.2 format (as is used on FDDI maximum size. The DECbridge 500 product per­ LANs). Figure 7 illustrates how Ethernet frames and forms one fu nction beyond the process of trans­ two types of 802.3 I.LC frames are translated into parent bridging. The bridge breaks up large IP three diffe rent types of FDDI data link frames. packets into smaller ones. This function is sup­ Maximum and minimum frame sizes of the LAJ"fs ported by IP and is called fragmentation. Without also impose requirements on the translation pro­ fragmentation, the queue manager would discard cess. Ethernet and 802.3 MAC protocols require a these long IP frames, preventing communication minimum data field length of 46 bytes. The FDDI between nodes on separate FDDI rings that are MAC protocol supports zero-length data fields. linked by Ethernet/802.3 LANs. When a bridge forwards frames that originated at Since the translation process alters frames, the nodes on an FDDI LAN to an Ethernet/802.3 LAN, original cyclic redundancy check (CRC) field is no the translation process must add padding (null longer valid. In the DECbridge 500 device, the trans­ bytes) to any short data fields to bring them up to lation process concurrently verifies the received the 46-byte minimum size. CRC, translates the frame, and generates a new FDDI has a maximum frame size of 4500 bytes. CRC. This concurrent processing results in a high The Ethernet/802.3 maximum frame size is 1518 degree of data integrity.

ETHERNET/802 3 FRAME FORMAT FOOl FRAME FORMAT

F R 1 c7 I e c H

ETHERNET V20 FDD' SNmSAP ~ DATA SNAP SNAP CTRL OUI TYPE DATA L------11..---·r 1 1 1 1 1 1 r = AA = AA = 03 = 00,00,00

802.2 LLC DSAP/SSAP FDDI LLC I DSAP I SSAP I CTRL I DATA ---·1DSAP I SSAP I CTRL I DATA AA AA 03 00,00,00 AA AA 03 00,00,00 ?' ?' ?' ?' ?' 7' 7' ?'

802.2 LLC SNAP/SAP FDDI SNA P SAP i-;;_;;_....;;...;,...;...... ;/ ;_;_T"""_-.,- ---==--...,....----. P D SNAP SNAP C RL DATA SNAP SNAP CTRL DATA I I I T I OUI p;o I 1+----•1 I I I OUI i I I = AA = AA 03 00,00,00 = AA AA = 03 00.00,00 = ?' = 7'

KEY:

DA, DA' DESTINATION ADDRESS ('=BIT REVERSED FORMAT) SA, SA' SOURCE ADDRESS ('=BIT REVERSED FORMAT) TYPE/LEN TYPE FIELD FOR ETHERNET, LENGTH FIELDS FOR 802.3 FC FRAME CONTROL ED END DELIMITER CRC CYCLIC REDUNDANCY CHECK OUI ORGANIZATIONAL (VENDOR) UNIQUE IDENTITY CODE DATA DATA FIELD FOR DSAP. SSAP. SNAP. SAP. CTRL. PID. AA. AND TYPE. REFER TO 802.2 STANDARD 3

Figure 7 Tra nslation of Ethernet/802.3 Fra mes to FDDJ Frames

60 Vo l. 3 No. 2 5pring 1991 Digital Te chnical journal Development of the DEC bridge 500 Product

Address Bit-ordering The bits of the destination a useful mechanism to prevent all frames of one or and source addresses are transmined in the reverse more protocol types from propagating across the order on FDDI from that on Ethernet/802.3 data extended JAN. The AP also uses the table lookup links. Digital's FDDI chip set performs a bit-reversal engine to check frames against a list of source operation on receive and transmit fo r only the MAC addresses loaded in the address table RAM for filter/ frames' destination and source address fields. Since fo rward requirements. Source address fi ltering may these MAC fields are stored inside the bridge in IEEE be used to contain traffic from nodes with an 802.1, canonical bit-ordering, only one ve rsion of unusually high transmit rate. In addition, this filter­ each address needs to be kept in the fo rward ing data­ ing may be used as a fo rm of security to deny access base. Also, when generating management messages, to nodes that are masquerading, that is, transmit­ this method of frame storage allows the bridge to ting by using another node's address. move an address from the source or destination field The bridge checks frames against protocol and of a received frame into the data field of the manage­ source address lists after the frames have been fi l­ ment message without modification. When calcu­ tered by destination address in the queue manager. lating the CRC on incoming packets, or generating The rate of frames here is lower, nor exceeding the a new CRC on fo rwarded packets, the translation Ethernet/802.3 LAN rate. Performing such checking process must take into account the bit-ordering. on all incoming traffic from the FDDI LAN would require significant additional computational work Implementation The translation processor con­ by the queue manager subsystem. sists of principally a GAB and a translation engine The frames in the fo rward queue that pass the (based on a 68020 subsystem). The GABand transla­ protocol and source address filters are placed on the tion engine interactively copy frames from FDD! transmit queue of the Ethernet/802.3 chip set. The frame memory to Ethernet/802.3 frame memory. AP must merge these frames into the transmit queue Concurrently, the translation engine makes changes with management traffic that the AP has generated to the frame fo rmat, and the GAB calculates both fo r the Ethernet/802.3 LAN. the CRC of the incoming frame, using old bit-order­ ing, and generates the CRC of the translated frame, Nl- to-FDDI Fo rwarding using both new bit-ordering and new frame format. The bridge processes tratJic received from the FDD! Frames from the fo rward queue in frame Ethernet/802.3 JAN in much the same way as FDDI memory are, thus, translated and moved to the fo r­ LAN traffic processing was described in the preced­ ward queue in Ethernet frame memory. Frames ing materiaL It is essential! y a mirror-image process, from the bridge queue are separated into manage­ but a few significant differences exist. ment and spanning tree queues in Ethernet/802.3 The lower arrival rate of frames from the frame memory. The translation processor returns Ethernet/802.3 LA.t'-1 does nor require a dedicated buffers from the fo rward and bridge queues to the frame-processing engine such as the queue manager. FOOl free queue in frame memory. The queue man­ Thus, destination address filtering is performed ager returns buffe rs from the free queue to the by the AP, which shares the Tl.lJ engine and table receive queue, making them available to store address RAM with the queue manager. newly received frames. Also, aJ location counters are not used on Ethernet/802.3 traffic. The AI' directs all incoming N/- side Processing traffic into clif ferent queues at fu II rate. Unusually Frames placed in output queues by the translation high bursts of a particular fr ame type could over­ processor are processed next by the AP. Frames in flow a given queue. the spanning tree and management queues are des­ Another difference is a requirement fo r stations tined fo r the bridge as a manageable entity on the placing traffic on the FDDI ring. On token ring net­ extended LAN. The AP proe<::sscs these frames and works, the transmitting station is responsible for may generate response traffic on either the FDDI or removing its own frames from the ring. A typica l the Ethernet/802.3 LAN. station knows which frames to strip by recogniz­ Frames in the fo rward queue are subjected to ing its own address as the source address. When a additional match or nonmatch fi ltering by the AP. bridge transmits a frame, the source address is that These frames are checked against a list of protocol of the originating node. In the DECbridge 500 prod­ types loaded by network management (e.g., TCP/JP uct, the stripping fu nction is handled in the FDDI and AppleTalk protocols). Protocol fi ltering is often chip set. A bridge strip algorithm is implemented

Digital Te cbuical jo unwl H>l. .i No. 2 !>j; rin,� [<)')/ 61 Fiber Distributed Data Interface

that generates frames marking the end of a block of firmware associated with the bridge entity and transmitted frames and also makes use of counters with Ethernet-side processing was modified from fo r sent and stripped frames. the LAN Bridge 200 product. The queue manager ancl translation processor required all new coc.le. Development Me thodology It was important to get the DECbridge 500 product Mechanical and Po wer Designs Previous products to market in as short a time as possible. The solidi­ typically consisted of a single module mounted in a box. The DECbridge 500 device required develop­ fi cation of the ANSI FDDI specifications, coupled with the appearance of products from diffe rent ing a multimodule system with a backplane. The vendors, created a fi nite window of opportunity. initial goals were to instaJI two, or at most three, At the same time, the requirements to be mer were logic modules. lo minimize the risk to the module significant. The next three sections present brief development schedule, a fo ur-board approach was descriptions of some elements of the development adopted, which closely follows the block diagram methodology that were employed to meet the shown in Figure 2. The box, the backplane, and the design requirements while optimizing the devel­ power supply are all new designs. opment schedule. In tegration of FDDI Products and Ch ip Set Utilization of Ex isting Te chnology Development 500 The OECbridge developers combined technology A strategy was adopted to maximize the common­ from existing products with their own new tech­ ality of effort in the development of rhe DECbridge nology in the following design areas: system level, 500 and the DECconcenrrator 500 products, and in electrical, firmware, and mechanical and power. the evaluation of the FDDI chip set. When a prod­ uct set was defined, plans were in place to develop Sy stem-leuel Design The AP and the Ethernet/ a hardware test bed fo r the FDOJ chip set. The test 802.3 500 packet memory of the DECbridge product bed design was expanded midstream so that sepa­ correspond approx imately to the processor and rate modules could be added, turning it into a memory of irs most recent predecessor, the LAN breadboard for either a DECconcentrator or a 200. Bridge The high FDDI packet rare required DECbridge device. The two DECbridge modules con­ the use of a separate processor to filter incoming tained the queue manager, the translation proces­ FODI traffic. Also, another ded icated processor was sor, and the Ethernet/1::!02.3 chip set and packet necessary to perform the translation fu nction. memory. The test bed provided the FDDI interface, (Ethernet-to-Ethernet bridges do not require a trans­ an FDDI packet memory, and an application proces­ lation function.) The resulting increase in the rate sor, as well as a power/packaging platform. \V hile at which a processor accesses frame data required evaluation of the breadboards was still taking place, the development of a separate packet memory fo r activities were accelerated to develop the products. the FDDI LAN . Te chnical Risk Analysis Electrical Design The Ethernet interface and packet memory designs again were borrowed from Diffe rent approaches were adopted for various the IAN Bridge 200 product, but several extensions parts of the bridge design based on technical risk. were needed. The AP design is very similar to the Completely new technology, e.g., the queue man­ design of the processor in the LAN Bridge 200, but ager and the translation processor, were simulated, it has severa l new fe atures, namely, a down-line, breadboarded, and tested. Areas that were under­ loadable progr:�m memory, a bus system fo r commu­ stood but still new, e.g., packet memory designs, nicating over the backplane with other modules, were evaluated largely by gate-level simulation. and a distributed interrupt system. The queue man­ High- confidence areas, such as designs taken from ager, the translation processor, and the FDDI chip previous products, were evaluated in the proto­ set with packet memory are new designs. The addi­ type products. 500 tional circuitry resulted in a multimodule system The DECbridge product employs three pro­ with a backplane. cessors. Thus, a Jot of the bridge functionality was in firmware, and changes cou ld be made with rela­ Firm ware Design The OECbridge 500 product tively little impact on the schedule. Also, in several uses the same operating system as other Telecom­ instances, deficiencies found in the system-level

municu ions and Network products. Much of the design could be corrected in the firmware.

62 Vol. ) Nu. l S!Jrinf! t'J<)I DiJ!.ilal Te cb nical 1o urnal Det 'elopment of the DEC bridge 500 Product

Us e of Pa rallel Activities Acknowledgments Several parts of the usual development process A large number of people contributed to the suc­ were overlapped to minimize time. A combined cess of the DECbridge 500 pro jeer. lr is not possible fu nctional and design specification was generated to name all of these important contributors in the instead of going through two serial stages to space provided. Therefore, the authors wish to produce separate specifications. Tn the hardware acknowledge that the DECbridge 500 product could design, module layout started once a confidence not have been developed without sign ificantcontri­ factor was achieved through simu.lation. Design butions from members of the Te lecommunications reviews were held concurrently with module lay­ and Networks (T&N) Communications Systems out, and performance simulation continued through­ Engineering Group responsible fo r the diagnostics, out the process. There was a close interaction of firmware, hardware, product assurance, and soft­ the printed circuit board layout groupand the elec­ ware and from the fo llowing T&N organizations: trical designers. Architecture, Customer Services Systems Engineer­ In product qualification, a pipeli ned system of ing, Marketing, Manufacturing, Product Manage­ reliability qualification testing (RQT), process qual­ ment, and Publications. ification test ing (PQT), and internal/external field test was set up to accommodate a phased-release of Riferences firmware. RQT and PQT started with a hmctional, I. F Backes, "Transparent Bridges fo r Interconnec­ subset release of the firmware. Hardware confidence tion of IEEE LANs," lt:t."t: Network, vol. 2, no. 1 grew. After electrical design verification testing, Qanuary 1988). firmware with the minimal functionality for field test was tested briefly in RQT and PQT and then 2. Local Area Network MAC (Media Access Con­ shipped to the field. New firmware releases were trol) Bridges, IEEE Standard 802.l(d) (New York: developed with increased functionality as well as The Institute of Electrica l and Electronic Engi­ corrections to recognized problems. A process was neers, Inc, 1990). developed whereby new releases were tested fo r a .3. Logical Link Control, A.l"\JSI/IEEE Standard few days each in RQT and at internal field test sites 802.2-1985, ISO/DIS 8802/2 (New Yo rk: The Insti· and then released to external field test sites. The tute of Electrical and Electronics Engi neers, down-line-upgrade ability was instru mental in Inc., 1985). allowing us to use this process.

Conclusions Diffe rences in frame fo rmat, frame length, and trans­ mission speed place requirements on an Ethernet/ 802.3-to-FDDI bridge that are not encountered in bridges between I ike data links. The DEC bridge 500 product met these requirements by dedicating one processor subsystem to the translation process and another to the process of filtering and sorting incoming FOOl frames. By adhering to the require­ ments of the IEEE standard for transparent bridging, the DECbridge 500 device allows the problem-free interconnection of FDDI LANs to the large existing base of Ethernet/802.3 LANs. The development team concluded that by perform­ ing risk analysis and having backup plans in place, several parts of the standard design process could be compressed or overlapped. Fundamental to the design was the ability to make remote, nonvolatile upgrades to the product's operating firmware.

No. 2 Digital Te clmicaljournal Vu l. 3 Sp rin8 I')')J 63 Wi lliamJ. Tiffany G. Paul Koning I James E. Kuenzel

Th e DECconcentrator 500 Product

Digital'sdeci sion to implement the fi ber distributed data inte1jace (FDDI) physical topology with a dual ring of trees, as opp osed to a dual ring onlJ\ resulted in the development of the DECconcentrator 500 product. Th e dual ring of trees topology provides high availability, manageability, and support fo r building wiring stan­ dards. The fu nction of the concentrator demanded that the product be reliable, provide fo r remote management and control, and allow a low cost per connection. The use of common FDDI hardware and software components developed by Digital helped the product team to meet these goals.

Concentrators in the ANSI FDDI FDDI concentrators are more than wiring hubs, X319.5Standar d unlike certain other LAN technologies. 'fhey also In the initial stages of its development, the American perform two fu nctions that are key to network National Standards Institute (ANSI) standard fo r the integrity. When a station's connection to a concen­ fiber distributed data interface (FDDI) technology trator is activated, a multistep initialization proce­ was intended fo r a computer room system inter­ dure called physical connection management (PCM) connect, similar to Digital's Computer Interconnect takes place, using physical layer (PHY)sig naling. In (CI) bus. All stations were to be dual attachment this procedure, the station and the concentrator stations (DASs) and the interconnections between exchange some topology information, ancl a link the stations were to be wired directly, without confidence test is performed to verify that the data patch panels or similar structured wiring schemes. integrity on the link is acceptable. Once the PCM Using this dual ring topology is feasible fo r a small initialization is complete, the connection becomes number of stations in a single, tightly control led part of the ring. room. However, soon the emphasis of FDDI shifted Also, concentrators continuously perform link primarily to local area networks (IANs). A IAN con­ error monitoring (LEM). Each active link is moni­ sists of many nodes, spread over a large area and tored for data errors, and a link found to have with potentially many individuals able to connect excessive data errors is disabled. In this way con­ and disconnect stations. centrators ensure that the ring error rate, and there­ To accommodate LAI'I topology requirements, fo re the packet loss rate, remains acceptably low. ANSI chose to add the concept of concentrators Given the many choices fo r concentrator inter­ midway in the development of the FOOl standard.' connection allowed by the ANSI standard, it is In the simplestcase , a concentrator is a device that possible to construct highly complex topologies, attaches to a dual ring (via A and B ports) and pro­ including many that have "bad " properties. When a vides additional ports (M ports) to which stations station is physically plugged in and that connec­ can be connected by means of radia.l ly wired cables. tion is operating properly, the station should be able These additional stations can be single attachment to communicate with all other stations in its own stations (SASs) with a single port (S port) rather than network. This property is often stated as "Physical the pair ofports required by a DAS . This simple topol­ connectivity equals logical connectivity," or, in ogy was soon generalized, allowing concentrators other worcls, "Heing plugged in implies being able to be nested to any depth in a dual ring of trees. to communicate." In bad topologies, this property Concentrators may be singl y attached (by using an does not hold. Such topologies are ve ry confusing S port plus M ports rather than A and B ports plus to network managers and are therefore undesirable. M ports), and OASs may be connected to concentra­ The ANSI standard specifies some topology rules tors. Figure 1 illustrates the basic FOOl topologies. to reduce this problem. However, the decision to

64 Vo l. 3 No . .Z .\jJring 1991 Digital Te c!J nical jou rnal TheDEC concentrator500 Produ ct

DUAL RING TOPOLOGY

TREE TOPOLOGY

KEY:

A PHY-A PORT B PHY-B PORT M PHY-M PORT PHY-S PORT s � DUAL ATTAC HMENT STATION � CONCENTRATOR � SINGLE ATTACHMENT STATION DUAL RING OF TREES TOPOLOGY 0

Figure 1 Basic FDDI Top ologies

accept or reject offered connections is based only the B to M connection becomes active, and the A on local knowledge (i.e., information held locally to M connection remains in a "standby" state. The in each station or concentrator), and it is not possi­ standby connection is not part of the ring, but it

ble to detect all the bad topologies. FOOl therefore can quickly change to the act ive state if the B to depends to some extent on the competence of the M connection breaks. In this way, connectivity is network manager to avoid bad topologies. maintained when fa ilures occur. One special case allowed by the topology rules We next present the reasons Digital chose the is called dual homing, as shown in Figure 2. With a dual ring of trees topology over the dual ring topol­ dual homing configuration, the A and B ports of a ogy and examine the re sultant need fo r a concen­

concentrator or a DAS are connected to M ports, trator. A detailed discussion of the development of usually of two different concentrators. In this case, the DECconcentrator 500 product fo llows.

Figure 2 Dual Homing Tree

65 Digital Te chnical jo urnal Vo l. 3 No. 2 Sp ring 1991 Fiber Distributed Data Interface

Digital's Choice of the Dual Ring of Initial ly, cost was a major concern in the decision Trees Top ology to implement a tree-type architecture. Some users A dual ring topology consisting of dual attachment saw the addition of the concentrator as an added stations may be appropriate in very small networks cost burden. However, the cost increase, which can that do not use structured and permanent cable be amortized over the entire network, is greatly plants. But a dual ring of trees topology, using single outweighed by the added advantages. With a con­ attachment stations as the end-user devices and con­ centrator, stations can be separated into two cate­ centrators as hubs, provides the optimal solution gories, end-user devices and backbone devices. This for a highly flexible, reliable, available, maintain­ is especially important in large networks where able, and robust FDDI LAN.As described in the pre­ the fu nctions of network administration and users ceding section, the concentrator is the cornerstone of the LAN are totally disparate. The concentrator of an FDDI network. Its significance in building becomes a tool of the network managers that sim­ large, robust, and most importantly manageable plifies their role. As a demarcation point between FDDI networks has been recognized by many of our end-user devices and the backbone, the concentra­ key customers. Digital chose the tree/dual ring of tor protects the backbone from inadvertent disrup­ trees architecture implemented with concentra­ tion caused by the end user. tors over a purely dual ring approach because this As shown in Figure 3, the actual number of com­ architecture offers: ponents that are required to connect eight FDDI stations, whether into the dual ring or in the tree, • Support fo r industry standard radial wiring is approximately the same; only the distribution of practices these components is different. • Manageability In both topologies we have eight physical con­

• Configuration flexibility nections. A physical connection, whether in the tree or in the dual ring, is a point-to-point, full-duplex • A definable demarcation point between the end path between adjacent physical layers. The initial user and the backbone reasoning for a dual, counter rotating ring was to Scalability • create a bidirectional data path between adjacent Although the specific behavior of an FDDI con­ stations in which the secondary path's main pur­ centrator is relatively new, the concept is not. Most pose is to assist in startup, initialization, and recon­ system vendors and users of large systems have figuration of the primary ring. adopted the use of manageable hubs (multiport Either topology requires roughly the same num­ repeaters) fo r Ethernet networks and med ia access ber of components, i.e., PHY entities, optical trans­ units (a type of passive concentrator composed of ceivers, cables, etc. With the tree, the PHYs are bypass relays) in token ring networks. rearranged so that conceptually we have taken a

DUAL RING TOPOLOGY

KEY:

DAS DUAL ATTACHMENT STATION SAS SINGLE ATTACHMENT STATION P PHYSICAL LAYER PORT TREE TOPOLOGY

Figure 3 Co mponents of Tw o FDDI PhysicalTop ologies

66 Vo l. 3 No. 2 Sp ring 1991 Digital Te ch nical jou rnal TbeDEC concentrator 500 Product

PHY entity from each DAS, in conjunction with a also be an ordeal, since access to fault information data path switch, to create the concentrator. This would be limited to the stations remaining on the approach does incur the additional cost of the portion of the ring to which the management sta­ power and packaging. However, from the perspec­ tion was directly attached. tive of the network administrator, this smaU incre­ An FDDI concentrator provides fault tolerance in mental cost is offset by the increased ability to a different way, as illustrated in Figure 5. When a manage and control the network, made possible by station connected to a concentrator is removed or the concentrator. powered off, the fa ilure is bypassed through the A large network of many stations has a high concentrator data path switch at the PHYlevel. Any probability of disruptions. Although a DAS and a one or all of the stations can be effectively bypassed concentrator are fundamentally different, they have through the concentrator without affecting the in common the role of controlling the network connectivity of the other stations or the global topology through PHY-Ievel signaling. In the case topology of the FDDI network. of a disntption, whether an operator-initiated func­ tion (i.e., a station powered down, installed, or Structured Wi ring removed) or a failure of the station or cable, the To fully appreciate the benefits of the concentra­ token path is modified according to the station tor, let us consider an FDDI network implemented management/connection management (SMT/C MT) in an office building environment in conjunction algorithm to maintain a continuous logical ring. with structured wiring. The main diffe rence between the two approaches A typical building environment includes wiring to solving the disruption problem is in the way a between offices and equipment rooms on the same station is bypassed. With the dual ring approach floor and in between floors. The wiring is permanent shown in Figure 4, when a disruption occurs, the and involves a relatively large number of end-user stations adjacent to the disruption bypass the devices as well as backbone devices over moderate offending station(s) and reconfigure the ring by distances. Moreover, frequent adds/moves/changes wrapping the secondary and primary rings to fo rm occur in this environment, and the ability to move a new single continuous ring. This provides a degree from one location to another without manual inter­ of fault tolerance but is limited to only a single vention or network disruption is desirable. A clear disruption. demarcation between end-user devices and back­ In the case of multiple failures or disruptions, bone is required to maintain the integrity of the all dual attachment stations adjacent to the faults backbone and to minimize disruption to, or manip­ reconfigure, thereby creating multiple disjoint rings. ulation of, the backbone cabling. Even though the majority of the stations in the net­ The Telecommunication Industries Association work might be operational, they would operate (TIA), together with the Electronics Industries over several disjoint networks. The potential loss Association (EIA), is defining a commercial building of the service access point would effectively leave wiring standard; draft EIA/TIA 568 has the frame­ the network nonoperational from the client/server work as designated in Figure 6.' According to this perspective. Management of such a situation would standard, end-user devices in offices should be

RING 1 RING 2

KEY

DAS DUAL ATTACHMENT STATION 00 DISRUPTION

Figure 4 Disruptions in a Dual Ring Top ology

Digital Te chnical journal Vo l. 3 No. 2 Sp ring 1991 67 Fiber Distributed Data Interface

KEY:

SAS SINGLE ATTACHMENT STATION 00 DISRUPTION

Figure 5 Disruptions in a Tree Top ology

wired from a telecommunications closet (TC). All the concentrators located in telecommunications closets in each building then connect to the inter­ closets, which are maintained and controlled by the mediate cross connect (JC) or to the building hub network administrator. When concentrators are for that building. In turn, all building hubs connect used, the most cost-effective user stations are SASs. to a single main cross connect (MC) or to the campus Connection to concentrators keeps the end-user hub in the campus. devices separate from the backbone so that a dis­ ruption in an end-user device, such as disconnect­ Single Atta chment Stations ing a station, does not affect the operation of the The tree topology, as illustrated in Figure 7, facili­ network. The concentrators in the various closets tates structured (or radial) wiring as prescribed connect to root concentrators in the building hubs. by the draft ElA/TlA 568 standard. The end-user If other backbone devices are present in the build­ devices, implemented as SASs or DASs, connect to ing hubs or communication closets, such as bridges,

HORIZONTAL DISTRIBUTION

KEY:

MC MAIN CROSS CONNECT - (CAMPUS HUB) IC INTERMEDIATE CROSS CONNECT · (BUILDING HUB) TC TELECOMMUNICATIONS CLOSET TC 2-1 TELECOMMUNICATIONS CLOSET (BUILDING 2, FLOOR 1)

Figure 6 Draft EIA/TIA Building Wiring Standard

68 Vo l. 3 No 2 Sp ring 1991 Digital Te chnical]ourtwl The DECconcentrator 500 Product

they can also be connected to concentrators in the building hubs or the campus hub, as appropriate. The tree topology offers the fault tolerance as well as configuration flexibility required in a structured wiring system. Also, it allows for adds/moves/ changes without disrupting or manipulating the backbone cabling.

DualAt tachment Stations End-user devices directly attached to the dual ring, however, are not easy to isolate from the backbone LAN. Both the end-user devices and the backbone devices are part of the same physical loop, as shown in Figure 8. To a network administrator, management and control of the backbone becomes an ever increasing ordeal because each end-user station is now considered part of the backbone. Even though rules fo r the end-user behavior can be Figure 8 Implementation of a Dual Ring established, they cannot easily be enforced. Top ology in Structured Wiring The availability of the backbone is increased by the use of concentrators, since these are the only higher in the latter case. Also, with DAS stations, devices that fo rm the dual ring backbone. This ben­ the network administrator is faced with the efit is very important for a large network. For exam­ impractical task of directly controlling 200 devices. ple, in a network supporting 200 end-user stations on a dual ring of trees topology, if 8-port concen­ trators are used to connect them to the dual ring Fault-tolerant Configuration options backbone, only 25 concentrators reside on the dual Twofa ult-tolerant configuration options are avail­ ring backbone. The reconfiguration of the backbone able: bypass relays and dual homing. Bypass relays is dependent on only 25 devices. Also, the network may be used with DASs directly attached to the dual administrator needs to control only 25 devices. In ring to provide fault tolerance in addition to the contrast, if the same 200 stations are DASs directly single fault protection provided by wrapping to attached to the dual ring, the reconfiguration of the the secondary ring. Dual homing is an alternative backbone is dependent on 200 devices. The proba­ mechanism which allows dual attachment devices bility of having two or more dis joint rings is much to have a redundant connection to a concentrator when installed in a tree topology. These two alter­ natives are examined in this section.

MC Bypass Relays To avoid the aforementioned reconfiguration prob­ lems with DAS implementations, the FDDI standard offers an option of using an optical bypass relay. While such relays are envisioned to alleviate some of the reconfiguration problems, they may induce more problems than they solve. The inclusion of relays in the network means added cost of components, cables, and connectors, loss of optical power, reduc­ tion in interstation distance, and an additional fail­ ure mechanism. These factors limit the use of such relays to possibly very small, physically collocated work group LANs and make the relay an unattrac­ tive solution fo r a large network environment. Figure 7 Implementation of a Tree Top ology As shown in Figure 9, the end-user stations A, B, in Structured Wiring and C are dual attachment with bypass relays, and

Digital Te cbt� icaljournal Vo l. 3 No. 2 Sp ring 1991 69 Fiber Distributed Data Interface

PERMANENT TELECOMMUNICATIONS OFFICE WIRING CLOSET

STATION A

DAS

I11 I I I I I I 14 STATION B (FAILED) 5

8

DAS

KEY:

DAS DUAL ATTACHMENT STATION I BYPASS RELAY CONNECTOR

Figure 9 Byp ass Relays

station B has fa iled. With B bypassed, stations A homing topology, shown in Figure 2 and described and C become adjacent. The total link loss between earlier in the paper, has none of the lim itations or these two stations, using a fiber-optic cable of 62.'5- problems that can be imposed by the dual ring. micron core diameter and 125-micron cladding This topology is beneficial in a large campus to diameter, must not exceed II decibels (dB) to com­ connect remote buildings into the FOOl backbone. ply with the FDDI standard. It allows standard radial wiring practices, with up Let us assume that the fiber-optic cable has a loss to 2 km between the campus hub and any given of 1.5 dB per kilometer (according to the EJA/TIA-492), building through multimoc.lefiber (MMF) links. The and the distance between the communication closet dual homing topology is especially useful with long and each station is 50 meters.' Since the cable length distance links utilizing single mode fiber (SMF), between stations A and Cis 200 m, the power loss which span distances ofup to 40 km.With the dual is 0.3 dB. A connector has a loss of0.7 dB (according ring topology each span has to be coun ted fo ur to the EIA). Since there are eight connectors times towards the fiber-optic path length maximum (labeled l through 8 in Figure 9) between A and C, of 200 km allowed by the standard, in the event the power loss is 5.6 dB. A bypass relay has a loss of that a wrap occurs. For a link of 40 km, 160 km has 2.5 dB. Since there are three relays between A and to be subtracted from the maximum of 200 km, C, the power loss is 7. 5 dB. The total link loss between thus leaving only 40 km fo r the rest of the network. stations A and C, therefore, is 13.4 c.IB, which is in Using a tree topology, the span is counted twice as excess of the maximum allowed by the FDDI stan­ it has only one active fiber pair. dard. Note that with the use of optical bypass relays. the effective distance is limited to less than Product Development 200 m, which is far below the maximum of 2 km The fo llowing sections examine the DECconcentrator allowed by the standard. 500 product development process from beginning to end, elaborating on details of the product func­ Dual Homing tionality as they were refined along the way. For some applications a tree or dual ring of trees Several key factors provided a smooth product may not meet the customer's requirements. The dual development process. First, the architecture fo r

70 ViJ I. .i No 2 Spring 1991 Digital Te chnical jo urnal TheDECc oncentrator 500 Product

the DECconcentrator 500 product was chosen as a ing the data link and A and B PHY port hardware, subset of the generality allowed by the AN.SI FDDI we ensured that any DECconcentrator 500 device standard. Several fe atures which would have signif­ installed in a dual ring of trees wo uld be manageable. icantly complicated the product without greatly The DECconcentrator 500 product also includes enhancing functionality were not included. a microprocessor to execute diagnostic and opera­ Examples of these are "roving MAC"and the ability tional firmware. To minimize the number of mod­ to allow stations to select the ring (primary or ules, we specified that the microprocessor and its secondary) to which to attach. Second, product support logic fit on the backplane. The use of an management established a clear list of priorities, active backplane eliminates the need fo r a separate requirements, and goals that allowed the develop­ control module in the card cage, thereby reducing ment team members to focus their efforts. The both cost and the vertical height of the box. The absence of significant changes in architecture or backplane also provides the token ring data path product requirements during the development function which interconnects any allowable config­ helped the team stay on schedule. The priorities uration of port and management modules. The num­ selected were to design fo r low cost and high relia­ ber of modules supported by the backplane is based bility, provide fo r ease of firmware upgrades, and on our evaluation of customer need. We decided strive fo r quick time to market. The emphasis on that 8 to 12 ports per concentrator is sufficient for simplicity and reliability enabled us to keep prod­ most end-user configurations. Two basic configu­ uct transfer cost to a minimum and to nearly meet rations are supported in the DECconcentrator 500 the time-to-market goal. And finally, the relatively product. A concentrator configured with one to small and tightly knit development team stayed three port modules (4 to 12 ports) can support a together from conception through field test and standalone work group but cannot connect in first product shipment. the dual ring of trees topology. A concentrator configured with a management module and one or Hardware Pa rtitioning two port modules supports the dual ring of trees As the hardware block diagrams were developed, topology and is remotely manageable. several concepts fo r partitioning the hardware into Another goal of the hardware team was to elimi­ modules were evaluated. The electrical, mechan­ nate the use of cables within the box. This goal was ical, and power supply designers worked closely consistent with minimizing cost and maximizing together to choose a suitable partitioning. reliability. The use of modular port and manage­ The initial high cost of the FDDI fiber-optic trans­ ment cards led the team to believe that the power ceivers led the designers to select modular hard­ supply could also be modular and plug into the ware partitioning. A modular design allows ports to backplane in a similar manner. To avoid potential be added according to the customer's needs, thereby safety hazards, the power supply module is not minimizing both the initial cost and the number of accessible without opening up the box; however, unused ports. Since Digital's networking products the interconnection of the supply with the back­ have traditionally used side-to-side airflow for plane is achieved with the same type of connector cooling, a card cage that supported horizontal mod­ used on the logic option modules. The only cable ules was chosen. Four ports per module was con­ used in the DECconcentrator 500 device provides sidered to be a reasonable number by which a power to the fans. customer could increment a system. This module, Figure 10 is a diagram of the various modules referred to as the port module, contains four sets that compose the DECconcentrator 500 hardware. of the FDDI physical layer chip sets, one status light-emitting diode (LED) per port, one module Power and Pa ckaging Trade-offs field-replaceable unit (FRU) fault LED, and a small Once we decided that three modules could support amount of support logic. 8 to 12 end users, we focused on the selection of One fu nction of the DECconcentrator 500 prod­ packaging. Two basic proposals were examined. uct is to provide support for network management; The first was to modify the Digital's NAC common this requires data link layer hardware. In addition, box which has been used in many of Digital's the DECconcentrator 500 device must connect to products. The second proposal was fo r a new box the FDDI dual ring; this requires the A and B port design, which allowed improved serviceability via types. The DECconcentrator 500 management mod­ quick access to all FRUs (field-replaceable units). ule was designed to meet these needs. By combin- The existing common box design was chosen to

Digital Te chtlical ]ourt1al Vo l. 3 No. 2 Sp ring 1991 71 Fiber Distributed Data Interface

CONTROLLER/BACKPLANE

DUAL FANS

Figure 10 DECconcentrator 500 Hardware Modules

minimize risk to the product development sched­ support the emitter-coupled logic (ECL) in the FDDI ule. The necessary modifications were the addition PMD. Fortunately the total power requirement was of a card cage fo r the port and management modules, similar to that of Digital's Ethernet-to-Ethernet provision fo r mounting the power supply and fans, bridge product, the LAN Bridge 200. The power and improvement of the airflow characteristics. supply group agreed to modify this existing high­ Cooling was also seen as a potential problem in volume supply to meet the requirements of the product development. The bipolar logic used in the DECconcentrator 500 product. The use of an exist­ FOOl physical medium dependent layer (PMD) dissi­ ing design was expected to result in fe wer bugs, pates a considerable amount of power in a small and this proved to be the case. Only one bug was area. Evaluation of the NAC common box showed found in system stress testing, and it was easily that the grill area in each side had approximately corrected with a minor design change. 35 percent open area. Analysis showed that signifi­ cant improvements in airflow could be achieved by Card Ha ndles increasing this percentage. Mechanical rigidity was The module hand le design was probably the single traded off against airflow improvement to reach a most complex part of the mechanical design, and it compromise of 50 percent open area in each side had the potential to impact the cost ofthe product. of the box. When prototypes were tested, we The mechanical design team recommended modi­ fo und the improvements in cooling fo llowed pre­ fying a handle design from an existing Digital prod­ dictions. The modification also yielded consider­ uct. This essentially meant stretching the handle and able reduction in acoustic noise levels, which making openings as required fo r l/0 connectors and allowed the use of off-the-shelf ball bearing fans LED displays. Handles fo r both DECbridge 500 and with good reliability. DECconcentrator 500 products are processed from Mechanical and electrical requirements could not the same mold since only 1/0 connector and switch/ be met by any of Digital's existing power supplies. indicator openings are different fo r the two products. The power supply height was limited; a unique Traditionally, these handles have been made from connector was required to interface to the back­ a plated cast alloy. Plastic offered the potential of plane; and the available area was determined by significant cost savings and weight reduction. How­ the size of the card cage on top of which the supply ever, there was concern about the quality of plating was mounted. Electrically, the supply had to pro­ with plastic, as well as about the structural strength. vide a relatively large amount of -5.2-volt power to The decision was made to start the development

72 Vo l. 3 No. 2 .\jJring I')')I Digital Te chnical jo urnal The DECconcentrator 500 Product

effort fo r plastic handles while using machined designs. When second-pass prototypes were tested, metal parts fo r the interim. When the final plastic only a single ECO wire was required in the product, product was available, it met all requirements. and the product shipped with the second-pass Another critical factor in handle design was elec­ designs on all modules. The fo llowing sections tromagnetic interference (EM!). Each FDDI duplex examine several areas of the logic design that are connector required a large, 1.4 em by 3.8 em, open­ unique to the concentrator fu nction. ing in the handle which posed the risk of emitting Concentrator Po rt An FDDI concentrator consists rad iation. A "waveguide-beyond-cutoff" structure of a group of seriallyconne cted ports, each of was evaluated. This structure is a rectangular exten­ which implements the FDDI PHY fu nctionality. sion to the handle with an opening the size of the Key to our product was Digital's PHY chip, which FDDI connector. The design attenuates all emissions implements the physical layer functionality and below the cutoff frequency and was predicted to supports the physical connection management provide excellent attenuation performance for all requirements fo r station management. In addi­ harmonics of the FDDI signals. Te sting of proto­ tion to the PHY chip, the CDC chips (one each fo r types verified the emissions problem created by transmit and receive) provide the serial/parallel the connector opening and proved the effective­ conversion, clock recovery on the receive side, ness of the waveguide structure in eliminating the and nonreturn to zero inverted (NRZJ) encoding/ emissions. This structure was then included in the decoding. The fiber-optic transceiver, however, design of the handles for each FDDI port. had to be purchased from an outside vendor. Since many mechanically incompatible devices are on Logic Design the market, we tested the products of many vendors. The logic design team developed prototype hard­ Fortunately fo r the product development teams, our ware as quickly as possible so that the diagnostic, optics group was able to identify pin-compatible operational firmware, and common node software transceivers fr om two vendors. Dual sourcing of (CNS) firmware teams could proceed with hard­ the transceivers used in our concentrator ports ware-based debugging. First-pass prototypes of the reduced the risk of shipping products based on controller/backplane module and the fo ur-port mod­ this relatively new fiber-optic technology. ule were in the laboratory within six momhs of the In ternal To ken Ring Data Pa th The FDDI PCM start of the project. The GenRad HILO simulation process provides fault coverage and topology rule software was used in the module design process. One type of bug was discovered in the first-pass checking between any two connected FDDI ports. This is essential to ring stabi lity since a token ring prototypes that was not caught in simulation. A is made up of a series of physical connections, through-hole component body was used in sche­ any one of which can bring down the entire ring. matic capture instead of a su rface mount body. As Because connections between ports within a con­ a result, the layout was wired according to the through-hole pinout. This error was not caught by centrator are not specified by the FDDI standard, any of the software that checks design rules. Thus, they are a critical design area fo r ring availability. The DECconcentrator design addressed this the control ler/backplane modules required engi­ 500 aspect of the product as described below. neering change order (ECO) wires to mount a com­ ponent onto the back of the module in a "dead • Data path (port-to-port) integrity. Adjacenr ports bug" fas hion. in the concentrator interface with a dual-symbol On the fo ur-port module, the differenrial ECL sig­ wide data path of 10 bits plus parity fo r a total of nal detect lines from the fiber-optic receiver to the 11 bits. This PHY-to-PHY interconnection is the clock and data conversion receiver (CDCR) compo­ same interface used to connect PHY to media nent were crossed. This logic was not included in access control (MAC) in a station. Parity check­ the simulation due to its analog nature. The problem ing was added to Digital's PHY chip to ensure in this case was an inconsistency in nomenclature that intermittent or hard fa ults cou ld easily be between the CDCR and fi ber-optic receiver chip detected. If a parity error in this data path bod ies used in schematic capture. occurs, the DECconcentrator 500 device is taken The strategy of bui I ding first-pass hardware proto­ off line to ensure that the entire token ring is types as quickly as possible to support early firm­ not corrupted. Without parity protection, a hard ware debugging was successful. Simulation played fa ilure on this internal path stops ring traffic a key role in guaranteeing fu nctionally correct altogether, in which case the SMT trace function

Digital Te chnical jo urnal Vo l. 3 No. 2 Sp ring 1991 73 Fiber Distributed Data Interface

might isolate the fault. However, an intermittent ware [MAC, CAJ.\'1 , ring memory controller (RMC), fault in a concentrator's internal data path that and packet memory] related to the data link is not protected with parity could arbitrarily layer. It also provides ring scrubbing in case the reduce ring performance and increase the risk MAC should have to leave or enter the ring while of undetected data errors and would not be iso­ the ring is operational. lated by any of the mechanisms built into SMT. Up gradeable No nvolatile Program Memory To sup­

• Fault detection/isolation. The controller/back­ port firmware upgrades over the network, the FDDI plane and the PHY chip design allow the products require electrically erasable program­ DECconcentrator 500 device to offe r continu­ mable read-only memory (EEPROM). Allcode in the ous service in the presence of hardware faults DECconcenrrator 500 product is executed directly by isolating the faulty hardware from the data out of EEPROM since the microprocessor's clock path. The diagnostics that are invoked at power rate is re latively slow. In the first-pass design we up or on command from firmware (as in the SMT used conventional 32K by 8 dual in-line package trace function) have the ability to isolate faults (DIP) devices as they were qualified within Digital. very close to the component level. The fault In order fo r the controller/backplane module to report is then passed to initialization firmware accommodate sufficient program memory, we which configures the DECconcentrator 500 needed a denser technology. At the time, compo­ product so that the faulty hardware is not nent engineering was evaluating flash EEPROM included in the data path. Two levels of bypass­ technology. Flash devices became available in sur­ ing are provided, one at the port level and one face mount packages with a density of 128KB that at the option module level. Bypassing is always met our needs. The flash memory proved to be a performed one level of hardware away from the robust technology; however, development of a flash detected fault. Thus if a fa ult is detected at the programming algorithm was challenging and CDC component level (using a CDC Joopback required extensive testing. The older EEPROM tech­ test), then that single port is bypassed in the nology had built-in logic to handle the details of PHY chip. If a fault is detected at the PHYlevel or the erasing and programming steps, but with the between PHY chipswithin a module, the entire flash memory these details were directly control led module is bypassed on the backplane. Note that by software. power-up diagnostics do not provide a PMD In order to upgrade firmware over the network, external loopback test except in a special manu­ a wel l-controlled procedure was developed. A firm­ facturing mode. Fault coverage of this hardware ware image plus the flash programming code is is provided by the SMT PCM process, which pre­ transferred over the network through multiple vents a faulty connection from being included packets and stored in packet memory. This down­ into the ring. Fiber loopback connectors are line upgrade is provided by a network device included with each product for isolating media upgrade (NDU) utility that was developed fo r the faults between the fiber-optic transceivers and FOOl products. Once the entire image is received in the fi ber-optic cables. packet memory, it is checked against a cycl ic redundancy check (CHC) included in the image. If • Internal MAC attachment. A MAC is not required the CRC is correct and the firmware image is of FOOl fo r an concentrator to function, however, the correct type (destined for this product), the it is included as an option to provide remote DECconcentrator 500 product takes itself off line. management. The internal MAC can be thought The 68000 microprocessor then executes the flash of as a "management station" attached to one of programming firmware directly from packet mem­ the concentrator ports whose job is to provide ory to load the new image. Once the load is com­ control/status of the local concentrator fu nc­ plete, the firmware forces a reset, and a power-up tion. This attachment is internal to the concen­ self-test is run that includes a CRC check of the trator, but must provide the same basic service contents of flash memory. as a physical connection to an external station. This service is provided by logic in the data path SoftwareDesign Issues referred to as the "null PHY.'' The null PHY pro­ Essential to the completion of the development vides a means of bypassing the internal MAC if process was rhe use of common software and the diagnostics should find a fa ult in any of the hard- field testing of the DECconcentrator 500 product.

74 Vu /. .) Nu. 2 !JjJring I'J'JJ Digital Te chr�icaljournal Tbe DECconcentrator 500 Product

Common Software Early in the development pro­ • Establishing and maintaining a close-knit prod­ cess it was clear that the aggressive time-to-market uct development team with good communica­ goals for the FOOl product set would require the tion channels. development team to be resourceful. In the begin­ Leveraging wherever possible from proven and ning of the development cycle a significant effort • avai lable designs, making incremental improve­ was made to identify ways to shorten the software ments as needed. development cycle. Code that could potentially be shared among the products, and code that could be • Providing a thorough testing process for both ported from previous projects was identified. These hardware and software which tested the prod­ early efforts resulted in the common use of the real uct in realistic environments with a process in time operating system (RTOS), common FOOl chip place to correct problems and verify fixes diagnostics, diagnostic error logger, diagnostic dis­ quickly. We had quite robust products at the patcher, and common node software (CNS). In time of first customer shipment as a result of our addition to the code that is common among the test/fix/verify process. FOOl product set, much time was saved by porting portions of the remote bridge management soft­ Acknowledgments ware (RBMS) responder and maintenance operation People from many organizations worked together protocol (MOP) from the J.AN Bridge 200 product. to achieve the development of the product on time. The management model for the data link and physi­ The key members of the engineering design team cal layer entities fo r both the DECbridge 500 and were Dave Benson, Gerry Capone, Stan Chmielecki, DECconcentrator 500 products was developed to Paul Ciarfe lla, Alison Coolidge, Janis Roth Cooper, ensure commonality between the two products. Joe Dagdigian, To m Ertel, Cheryl Galvin, Dave Hyre, John Iannarone, Bill McCarthy, Charlie McDonald, Field Te st The field test provided valuable infor­ Dick Muse, Luc Pariseau, Dave Sawyer, and Karen Shay. mation regarding the quality of the products. Several of the sites chosen to field test the FDDI References backbone products were technically knowledge­ able about networking. They were able to perform 1. FDDI Station Ma nagement (SMT) Preliminary specific testing while monitoring their networks. Draft Proposed American National Standard, As a result, detailed test information was provided ANSI X3T9/90-078, Rev. 6.2 (May 1990). to engineering. One engineer was assigned to each 2. Building Widng Standard fo r In dustrial and Com­ field test site as a "site parent." The site parents mercial Us e, ElA/TlA 568, PN1907-B (Washington, monitored their sites and channeled the info rma­ DC: Electronics Indust ries Association, Engi­ tion back to engineering. This structure for sup­ neering Department, forthcoming 1991). porting the field test enabled engineering to react quickly to the needs of the sites as well as act on 3. Detailed Sp ecification fo r 62.5-,um Core Diam­ problems fou nd. This testing and fe edback, coupled eter/125-,um Cladding Diameter Class Ia Multi­ with the capability to load new firmware revisions mode, Graded Index Op tical Wa veguide Fibers, over the network, was crucial to achieving quality AN SI/EIA/TIA-492AAAA-1989 (Washington, DC: prior to first customer shipment. Electronics Industries Association, Engineering Department, 1989). Conclusions The dual ring of trees topology is well suited fo r all FDDJ environments that require fau lt tolerance, scalability, and flexibility of configuration. This topology is the right choice fo r managing the ever­ growing local area networks through the 1990s. Several fa ctors proved to be crucial to meeting both functional and time-to-market requirements with a quality DECconcentrator 500 product.

• Establishing architecture and product feature requirements early and maintaining these with minimal changes.

Digital Te ch nical jou rnal Vo l. 3 No. 2 Sp ring 1991 75 Bruce E. Sweet I

DECelms-Managing Digitals FDDI and Ethernet Extended LocalAr ea Ne tworks

The DECelms software product provides extended local area network management fo r Digital's Ethernet/!EEE 802.3 andfi ber distributed data interfa ce (FDDI) bridges andfo r its FDDI wiring concentrator: Product development entailed keeping pace with a changing set of requirements. Th ese included the evolving ANSI FDDJ standard, the proposed Digital Network Architecture FDDI data link sp ecification, the Enterprise Management Architecture, the ability to extend the seruiceability of the products, and the aggressive schedules of the hardware and finnware development teams. DECelms development resulted in an improved network managementfu nctionality including fa ult, performance, and topologv management. These advancedfe atures required corresponding enhancements to tbe user interfa ce and dependable docu­ mentation. The development team met these challenges and successful�)' delivered the DECelms product to market as a part of Digital's FDDJ program.

DECelms Development mechanisms of Ethernet/802.3 LANs, such as fa ulty The DECelms product, Digital's extended local transceiver taps, continuously transmitting stations, area network (LAN) management software, pro­ and broadcast storms, are well understood. In add i­ vides remote network management fo r Digital's t ion to network management tools, powerful data­ LANBridge 100, LAN Bridge 150, IAN Bridge 200, scopes aid the network manager in recognizing DECbridge 500, and DECconcentrator 500 products. and correcting these problems. Moving into the The remote networks are included in the extended 100-megabit (Mb) world of the FDDI token ring LAN by means of transparent bridges. DECelms brought a new set of problems, some of which were functionality includes basic SETand SHOW capabil­ understood and others not even imagined. We real­ ities, fa ult management, performance monitoring, ized the need to offer a network management solu­ FDDI ring mapping, automatic device discovery, tion that was capable of performing the same and user alarms. The DECelms software runs as an fu nctions as our Ethernet/802.3 products and had application on a VAX processor running under the the additional functionality necessary to meet the VMS operating system. new challenges of the FOOl technology. Further, no When Digital embraced the new fiber distributed FDDl datascopes were available to aid in the devel­ data interface (FDDI) LAN technology, the role that opment of the FDDI product set. To correct this network management would play in the first prod­ deficiency, requirements fo r DECelms functionality uct set was unclear. As our understanding of the called fo r the software to be not only a good net­ technology grew, and we recognized the differences work management solution but a key development between the token ring architecwre and the car­ tool as well. rier sense multiple access with collision detection (CSMA/CD) protocol, the challenge became obvious. Key FDDI Differences Both technical and business reasons made network Several key diffe rences between the FDDI and management a priority. Ethernet technologies determined the specific On Ethernet/802.3 IANs, network management requirement s of the OECelms product. First, the tools provide the ability to monitor the networks physical topology and the location of stations and troubleshoot problems as they arise. The failure attached to an FDDI ring play a significant role in

76 Vu /. 3 No. 2 .)/1 ri11!: I'.J'Jf Digital Te chnical journal DECelms -Managing Digital'sFDDI and EthernetExtended Local Area Networks

how the ring operates. Each station is active and program team decided that the OECelms product must participate in connection management to fo rm would supersede the current bridge management a wo rking LAN.'On Ethernet/802.3 lANs, each station product, remote bridge management software is passive until it wishes to use the network. Thus, (RBMS), and incorporate RBMS functionality as a an Ethernet/802.3 IAN can operate with stations subset of the DECelms operating features. Thus, that do not strictly observe the protocol, since less DECelms software needed to provide basic SET stringent protocol requirements between stations and SHOW capabilities for the LAN Bridge 100, LAN are necessary to make the LAN work. This diffe r­ Bridge 150, and LAN Bridge 200 products, as well as ence places a high priority on the ability to manage fo r the new FOOl products, the DECbridge 500 and topology, a functionality that is less significant on DECconcentrator 500 devices. Ethernet/802.3 LANs. Requirements to build FDDI In addition to the ability to set and to show ring maps and examine third-party station manage­ system parameters, OECelms software had require­ ment (SMT) frame data grew from this priority. ments to provide automatic fa ult detection, auto­ A second difference is the need to manage the matic device discovery, performance monitoring, FDDI physical layer. This need arose mainly because and FOOl ring mapping. Automatic fault detection the DECconcentrator 500 product is a physical layer would be provided by a built-in polling mecha­ device and the primary building block of FDDI rings. nism and user alarms. Automatic device discovery To manage the FDDI physical layer, a network man­ would be accomplished by listening to system iden­ ager must be able to add or remove FDDI stations tification announcements broadcast by the devices from the rings via manipulation of the physical and registering these devices in the OECelms reg­ layer ports (PHYports) of the wiring concentrator. istry. Performance statistics could be calculated In addition, greater visibility was given into the from counters kept by the LAN Bridge 200 and quality of the physical medium, using the link error DECbridge 500 products and presented to the user monitor, for example.' in tabular format. FDDI ring mapping would be Another diffe rence between the two technologies accomplished by interpreting the data fo und in the is the order of magnitude increase in performance SMT status information frame (SIF) configuration of the FDDI ring over Ethernet/802.3. The ability to messages. These functions and their implementa­ transparently connect 100-Mb FDDI token rings to tions are described in more detail later in this paper. 10-Mb Ethernet/802.3 tANs using the DECbridge 500 product greatly influenced network management Development under Ti me-to-market requirements. Using these plug-and-play bridges Co nstraints makes it easier to create network topologies that The requirements of the DECelms capabilities were can funnel high throughput FDDI traffic onto the stabilizing. Program decisions were now influ­ lower bandwidth Ethernet/802.1 segments. A good enced by the time-to-market constraints. Advanced management tool is required to monitor and con­ development work on the DECbridge 500 and trol these topologies. OECconcentrator 500 products was progressing, A final key difference is simply the need to man­ and this work was converted into a full-fledged age the FDDI data link. The performance of the ring product development effort. When the network operation must be tuned, and having the capabil­ management discussions began, this effort was ity to modify FDDI media access control (MAC) well underway. Marketing and engineering man­ characteristics such as the valid transmission time agement clearly communicated to the developers or target token rotation time can provide the means the expectation that Digital would be the first ven­ to accomplish this tuning. Coupled with the modify dor with a complete FDDI solution. This pressure SHOW operations is a rich set of capabilities, which prompted the developers to be very creative in gives a detailed view of the FOOl data link behavior. choosing methodologies. As a result, the OECbridge 500 and DECconcentrator 500 teams shaved seven Extended Management Capabilities months off the original 18-month schedule. The Beyond the technical requirements to provide FDDI DECelms team was expected to meet these clul­ network management, DECelms requirements were lenging time constraints. driven by the need fo r better network management Reducing the time spent on development capabilities in general. Providing an integrated required the DECelms team to make many tracle­ product was a key project goal. Since managing offs. The first trade-off concerned whether the FDOI products included managing bridges, the product platform would comply with the Enterprise

Digital Te chnical jo-urnal Vi1i. 3 No. 2 Sp ring 1991 77 Fiber Distributed Data Interface

Management Architecture (EMA). The FDDI pro­ architectural soiution and fit in well with Digital's ject was concurrent with DECmcc development long-term network strategy. But the effort was (Digital's EMA-compliant director), but the FDDI just beginning as an advanced development pro­ program was three to sL"X months ahead. Given the ject, and the Phase V protocol specifications time constraints, the team decided that the FDDI were stiU in the review process. The memory program could not wait fo r the DECmcc program to constraints imposed by the DECbridge 500 and catch up. A point product was the only solution DECconcentrator 500 devices coupled with the that could be achieved in the short rime frame. A risk that the product might not meet the time­ second trade-off involved optimizing the develop­ to-market constraints caused this option to be ment effort. Having no EMA experience, the ream nlled out as a solution. used the expertise it had recently gai ned from The second option was to use the CMIP protocol working on the RBMS version 2.0 development .layered over a subset of the DNA Phase rv proto­ effort. They made the pragmatic decision to port cols. This subset is a streamlined implementation the available RBMS code in order to meet the time­ of the network services protocol (NSP) over an to-market constraints. The ream planned to start 802.3 data link. The network layer is null, so the

with the RBMS code, delete code that was not protocol is lim ited to the extended LAN . This solu­ applicable, ancl add code to provide the desired tion was small in size as dictated by the hardware, new functionality. offered guaranteed end-to-end deiivery service, But at the same time the RRMS code was being ancl would take less rime to implement than the refined and expanded, the DECelms team had to DNA Phase V option; but there were several draw­ keep the basic RBMS management code operational. backs. The bridge and wiring concentrator manage­ The firmware teams needed to use DECelms code ment entities were not defined in terms of the Hvi.A to cl ebug their new code, so the DECelms code had entity model, as was necessary to define the CM!P co evolve in such a way that new code would replace protocol structure. Using the Phase IV transport old and supersede its hwcrionality in a short protocol option did not bring the management period of time. This method of code development architecture closer to the open systems intercon­ led to a series of operations akin to "brain trans­ nect (OSI) model defined by DECnet/OSI Phase V, plants," where an interface was drawn in the ex ist­ and, in fact, would result in "throwaway" transport ing code and new code containing additional and network layer code. Thus, this option was functionality was added while the product was not suitable. kept running. The timing of add ing fu nctions had The third option was to extend the bridge man­ to coincide with the development of the peer hwc­ agement architecture and the RBMS protocol to tionaliry in the firmware. include support fo r the FDDI products. In terms of pure architecture, this was the poorest solution, Ne twork Management Architecture since it would merely result in the extension of With the product strategy taking shape, some dif­ an already limited protocol architecture. RBMS is ficult technical decisions concern ing the manage­ layered over the Ethernet/802.3 data link, has a ment architecture had to be made. Choosing a null network layer, and is therefore constrained network management protocol as well as support­ to the extended LAN. The RBMS transport is con­ ing transport and network layer protocols was a nectionless and, thus, does not offer guaranteed major challenge. end-to-end delivery service. In addition, no asyn­ chronous message support exists, so the delivery Choosing a Protocol of events or trars is unsupported. But RBMS is sim­ The team identified three options fo r the manage­ ple, being based on the IEEE 802.1 standard fo r ment protocol implementation. The first option network management, and easy to extend. Also, was using the common management information the LAN Bridge 200 development effo rt produced a protocol (CMIP) layered on rop of a paired-down new implementation of the management agent implementation of the Digital Network Architec­ that could he ported into the DECbridge 500 and ture (DNA) Phase V protocols. This implementation DECconcenrrator 500 products. Further, there was would include a subset of the DECnet Phase V ses­ the opportunity to port code from the RBMS ver­ sion, transport, routing, and data link protocols sion 2.0 product. Given the time constraints under used solei y fo r implementing management agents which the DECelms team had to operate, the RBMS in server products. This option wa� the most pure protocol was clearly the best possible choice.

78 Hi/. 3 No. 2 ,�jJring I()';) I Digital Te e/m ica/ jo urnal DECelms -Managing Digital's FDDT and Etbernet Extended Local Area Netwo rks

SMT as Co mpared to Ma nagement over DECbridge 500 and DECconcentrator '500 products, the Pro tocol Stack while keeping an eye to the fu ture of EMA. Several Another approach to network management was via key sources of information were available. One the FDDl station management frames defined in the source was the ex isting bridge management American National Standards Institute (ANSI) X3T9.5 architecture, which defined how to manage an I.AN FDDI working group draft standard ve rsion 5.1 . To Ethernet/802.3 bridge such as the Bridge 200 provide this alternat ive meant adding complexity device. Another information source was the ANSI X3T9.'5 to SMT that belonged in a more robust management emerging FDDI working group draft protocol such as CMIP. Further, the SMT standard standard, specifically the chapter concerning FDD! was unstable in this area; many vendors part icipat­ station management. Add itional information was DNA ing in the standards wo rk were advocating diffe r­ fo und in an early draft of the FDDJ data link ing FDDI fu nctionalities for SMT, thus, trying to specification as wel l as in the approved ve rsion of the DNA CSMA/CD data link specification. The final extend S1YI T beyond its originally intended scope. Digital's position was that network management set of inputs came from the product requirements should be performed using an OS! model where of the DECbridge 500 and DECconcentrator 500 management is an application that runs on top of devices. These requirements called attention to management capabilities that went beyond bridg­ the protocol stack and is widely available. SMT frames are below the MAC level and, therefore, do ing, wiring concentrators, FDDI data links, and not traverse a LAN beyond a local FDDI ring. Thus, Ethernet data Jinks including management of the management using SMT frames must come from a down-line-load upgrade fe ature and avai labi lity of (FRU) local station on the FDDI ring. The data provided by field-replaceable unit status.' ' the SMT frames, such as the SIF, is valuable to the Al l of the above data sources had to be assimi­ network manager. The appropriate mechanism to l.ated exped itiously because the chip designs were communicate this data is a management protocol nearing final fo rm. If the infrastructure necessary using an agent that can com municate across to allow fo r the extension of management fu nction­ extended I.ANs. Once again, RBMS could perform ality was not identiJied, either proposed fe atures this task. would be dropped from the products, or the chips would have to be respun. Keep in mind that the Ma nagement Model Definition strategy was to have two management structures AJter the protocol issue was settled, the next step in place: the first, based on the bridge management was to define the FDDI manageable entities and architecture to be used fo r the DECbridge 500 and attributes. The DECelms program team had decided DECconcentrator 500 products and managed using to use IUIMS, but the chosen long-term strategy was the DECelms product; and, the second, incorpo­ CMIP RBMS DNA to use with the E:\1A guidelines. The rating the FDDI data link architecture and a decision affected only the server products and future bridge and concentrator management archi­ not the FDD! adapters that were also under devel­ tecture based on the El\'lA . Design decisions regard­ opment. The adapters would be managed via ing network management entities and attributes VMS DECnet/OSJ Phase V on both the and the were made accordingly and are described below. UI.TRIX operating systems. The RBMS effort could The fo llowing discussion presents the manner in not derail the long- term management strategy of which the combi neu model fo r the DECbridge '500 El\'lA and the migration to OS!. Thus, there needed and DECconcentrator '500 products was developed . to be two management strucwres that offe red sim­ The proposed management model fo r the concen­ ilar management capabilities but used different trator was flattened out and inserted into the cur­ mechanisms. Later, a third management structure rent bridge architecture. The bridge architecture driven by the internet community would become was extended to include the FDDI data link in addi­ important, namely, the management information tion to the Ethernct/802.:) data link. Additionally, base (M ill) supported by the simple network man­ the PHY port entity was added, but instead of being agement protocol (SNMP). subordinate to the data link as in the management A series of FDDI network management meetings model fo r the bridge alone, the PHY port entity was

took place involving the FDD! implemenrors, the at a peer level, as shown in Figure I. Additional DNA architects, and members of the ANSI FDDI stan­ management attributes were defined to bring the dards wo rking gro up. The go al was to converge on visibility into the box as req uired fo r each product. a set of manageable ent ities and attributes fo r the This phase of designing a network management

I Dif!.ilal Te clm ical jo urnal VlJI. .i ,Vo l Sp ring I')') 79 Fiber Distributed Data Interface

CSMA!CD BRIDGE MODEL EMA CONCENTRATOR MODEL FOOl

COM BINED BRIDGE AND CONCENTRATOR MODEL FOR DECELMS

BRIDGE OR CONCENTRATOR DEVICE (INCLUDES NODE, FOOl, AND STATION ENTITIES)

I I I l CSMA/CD OR FOOl LINE (INCLUDES SPANNING FORWARDING SMT PHY PORT PORT AND LINK TREE DATABASE GATEWAY ENTITIES) I ,-----J � LINE ADDRess SPANNING '"mococ TREE WI I I I w

1 Figure Management Models fo r the DFCbridge 500 and DFCumcentrato1· 500 Products

archi tecture resulted in a delay in tilL development architecture that evolved to implement the SMT gate­ of the t<>nnal DNA architectu re . pending the re sults way was another extension of the RBMSprotocol. A of the continui ng ANSIstandar ds meetings. new e nti ty type called the SMT gateway was added. Gateway requests and responses rel ating to SIFs, Station Management Gatetuay neighborhood information frames (NJFs), and echo As mentioned earlier in this paper, s.v!T frames do frames were defined, fo r example, GET SJF, GET NIF, ECHO. not travel he yond a local FDDI ring. The:: DEC:elms and DO A set of timers was defined to al low the SMT product was expected to manage:: Digital and third­ the Rl3MS h ost, gateway, and the target sta­ parry FDDI stations on the FDDI ring using SMT tion to be synchronized and to avoid an ambiguous frames . This management had to be initiated from response. For example, ifa response is missing, it is the:: Ethernet/802.3 segment on which the DECclms important to know if the lack of response is caused host was located. The concept of the SMT gateway by gateway congestion, by no response from the emerge d from this situ ati on . target station, or simply by datagram loss on the

An S.Y1T gateway is firmwa re res iding on the Ethernet. With the SMT gateway architecture in place, DECconcenrrator 500 device (and s ubseque nt ly on a DECelms management station located anywhere the: DEC:bric.Jge 500 product) that cncapsulates SMT on the extended LAN can gather SMT frame <.lata frames into RB:VIS protocol data units (PDUs) and from any FDDI SMT ve rsion 5.1-compliant station t<>rwards these frames across the exte nded LAN . The on the same ring as a Digital SMT gateway stat ion.

HO Vol. J No. .l SjJring I')')I Digital Te cb nical jo unwl DECelms -Managing Digital'sf7JDI and EthernetExte nded LocalAr ea Networks

Management beyond Set and Show the state of each device in the OECelms registry and Several management fu nctions beyond basic SET reporting changes in that state to the network man­ and SHOW capabilities were introduced in an ear­ ager through user alarms and a log fi le. The infor­ LAN, lier section, namely automatic fa ult detection, auto­ mation includes the designated bridge on a matic device discovery, performance monitoring, the number of FDDI ring initializations, the num­ and FOOl ring mapping. These featuresadd value to ber of cyclic redundancy check (CRC) errors on a the network manager. They present network fa ult given data link, and, perhaps most importantly, the and topology data in a timely and concise manner fact that a station has become unreachable. which frees th<::: network manager from interpret­ ing the microlevel. details of the network . A pri­ Network In terface Multiplexer Process mary goal of the OECelms team was to develop a Both the device listener and the background poller set of tools that the network manager could use are integral parts of the OECelms process known to detect and correct a network problem before it as the network interface multiplexer (NIMUX). As was reported by a distressed user. implied by its name, NIMUX also provides the basic multiplexing of user protocol messages as they are Automatic Device Discovery sent and received by the data link driver. Designing The addition of a station to an Ethernet/802.3 LAN NIMUX to incorporate these three distinct functions is now a common and simple procedure. Using the was a considerable challenge. The design includes plug-and-play nature of Digital's bridge products, the a mailbox interface to the variable number of user connection of Ethernet/802.3 or FOOl LANsis also interface tasks. This interface provides fo r both the straightforward. The addition of stations to an FODI sending and receiving of user data and alarms. The ring is easy with the use of wiring concentrators. NIMUX design also provides the control. interface ln campus environments, network managers can fo r the background poller and the device listener typically, but not always, control physical access to and a service interface fo r the DECelms registry, the network backbone but have Jess control once a which is only written by NIMUX fo r synchroniza­ segment enters an individual department. Conse­ tion purposes. quently, there is a need to automatically discover Within NIMUX is a kernel that synchronizes these the "renegade" devices that can be added to such three basic fu nctions. Synchronization is performed networks. The OECelms product solves this prob­ using VMS asynchronous system traps (ASTs) and lem by providing a device listener that listens to event flags. When the NJMUX has no tasks, the process the maintenance operation protocol (MOP) system hibernates to conserve the CPU utilization. But, the identification messages periodically broadcast by NIMUX process may be woken up, fo r example, by the all of Digital's bridges and wiring concentrators. delivery of a mailbox message from a user process Whena new station is heard, the DECelms software or by the completion of a network 1/0 operation. queries the station via the RI3MS protocol to obtain The implementation of the background poller is more detailed information about the station and complex. The background poller gathers state infor­ automatically adds the station to the OECelms reg­ mation for each device in the DECel.ms registry and istry. The software also produces a user alarm that continua.lly circulates through the registry. This notifies the network manager of the existence of initial query is necessary to determine whether the this device and allows the manager to evaluate its device is currently active or inactive. To obtain the impact on the topology and performance of the complete state of a device, The OECelms software LAN.For instance, every bridge added between two must issue multiple protocol requests to that device. stations adds latency to the protocol communica­ In the case of a OECconcentrator 500 device, twelve tions between the devices. To o much latency may requests must be issued to accumulate all the negatively impact LAN performance by causing cer­ device data. The synchronization of these requests tain protocols to fa il. is a challenge, since it is possible fo r any of these requests to fa il because of a datagram loss or the Background Pa ller failure of the network or the device itself. Several The ability to quickly recognize changes in the event flags are required to identify when a query is state or counters of a network device can help a pending and then to identifywhether the query is a network manager avoid a degradation of service. success or a failure. When a single query is com­ The OECelms background poller provides this type pleted, the next step is dependent upon the device of fa ult recognition by keeping info rmation about type and the status of the previous query.

Digital Te clmicaljournal V/') /. _) No. 2 Sp ring 1991 81 Fiber Distributed Data Interface

The automatic addition of a device to the DECelms FDDI Ring Mapp ing registry is performed in the polling cycle. Once a The ability to map the FDDI ring is a key feature of device MOP system identification message is heard the DECelms product. The SMT gateway provides the and the device is recognized as new to DECelms, mechanism to obtain the raw SMT frame data nec­ the responsibility of performing the RBMS proto­ essary to build ring maps. Two types of ring maps col queries belongs to the poller. Three protocol were identified as possible fuoctionalities: a simple messages are necessary to determine the device logical ring map and a physical ring map. The logi­ type and MAC addresses. The recognition that these cal ring map could be built using the upstream two functions have this common thread helped to neighbor address (UNA) contained in the N!Fs. This simplifythe NIMUX design and limited the amount functionality would provide a list of the MAC of internal state information that it was necessary addresses in the FDDI ring to which the token was ro keep. passed, but would not provide info rmation regard­ In parallel to the background device queries and ing the number of PHY ports in a concentrator or the automatic device registration taking place, the details of the physical topology of the network. NIMUX also has to process user requests. These The preferred fu nctionality was the physical requests are given priority over those of the back­ ring map which can provide detail into the actual ground poller, since an interactive user cannot physical topology of the FDDI ring. Using the the­ be made to wait until all the background polling ory of wiring concentrators to build FDDI rings, the queries are completed. This processing is accom­ resulting topology is a ring of trees. Ty pically, this pl ished in the outermost loop of NIMUX. If a user configuration includes dual attachment concentra­ request is delivered via a mailbox, an event flag tors (DACs) located on the dual FDDI ring, with other indicates its arrival. This request is given pref­ concentrators and singJe attachment stations (SASs) erence over moving on to the next step in the connected to the concentrator on the dual ring in a polling process. tree-like fa shion:' The physical ring map is required An important part of the interface between to represent this topology showing the PHY port NIMUX and the user processes is the ability to attachments in addition to the MAC attachments. deliver alarm messages. Each user process has the This representation is accomplished using the option of enabling or disabling alarm messages data contained in the SIF configuration messages. fo r its process. The fact that multiple alarms can be The physical ring mapping starts when the user generated by a single device query from the back­ supplies the address of an SMT gateway on the tar­ ground poller, and multiple users need to receive get FDDI ring. The DECelms host then issues GET SIF these alarms, means the interface has to contain gateway requests to the gateway. The algorithm queues to buffe r the alarms and state variables to cal1s for the ring map to start with the gateway's SIF control the sending of the data. This problem was information. Then, the UNA of the gateway con­ solved, as were most of the NIMUX interface prob­ tained in the SJF configuration message indicates lems, by the addition of these data structures to which station in the ring to query next. Using the the bridge control process (BCP) tables. A BCP table station descriptor, station state, and path descrip­ is a robust data structure that keeps the NIMUX tor fields of the SIF response, the DECelms host can state info rmation for a single user process. One derive the detailed physical map. table is created and destroyed fo r each user process The basic mechanism to provide the physical that exists. ring map was understood, but the implementation The last job NIMUX can perform is giving write and the actual application mandated additional access to the DECelms registry. Since both the auto­ requirements. With the SMT standard still in a state matic device registration fu nction in NIMUX and a of flux, testing the ring map with third-party ven­ user process can add devices into the registry, syn­ dors disclosed that it was possible to interpret the chronization is necessary. The simplest solution is standard in different ways. Consequently, the ring to eliminate the need to synchronize by having map code had to be more flexible to interpret the only NIMUX do the registry write operations. The same data fields which often had slight variations mailbox interface commands were extended to in fo rmat clueto loose interpretations of the speci­ allow user registration, modification, and deletion fication. With increased testing the ring map could of devices in the DECelms database using NIMUX as interpret data from many vendors and thus became the server. a very useful and popular tool.

82 Vo l. .) Nu 2 Sp ring 1991 Digital Te chnical jo urnal DECelms -Managing Digital'sFDDI and EthernetExtended Local Ar ea Networks

Another feature was added to the ring map which is an advantage over simply presenting a table. which made it possible to build partial ring maps. In actual use, the map is easy to understand and pro­ The capability of specifying the ring map starting vides an intuitive picture of the network topology. address allows a user to pass a station that cannot be mapped otherwise (i.e., the station does not cor­ User Inteiface rectly respond to a SIF request) and manually con­ As described earlier, the DECelms code has its roots tinue the map with the next station on the ring. in the RBMS software. However, the RBMS user The display of the map was a noteworthy task. interface is a simple command line which lacks a Concepts fo r this implementation ranged from a number of fe atures necessary to make the product simple tabular fo rmat to color graphics. The time con­ more useful to network managers. To remedy this straints greatly influenced our decision. A simple deficiency, the DECelms team set the fo llowing and concise set of icons was developed to repre­ goals fo r the user interface. The software needed sent the stations on the ring. These icons are actu­ to provide intuitive commands to a previous RBMS ally depicted as stick figures displayed using ASCII user, shorter commands than RBMS, the ability to terminal art. Each icon is paired with the level at scroll through long output displays, the ability to which it exists in a tree, similar to the way in which input data from command files, and the ability to logic statements appear in a compiler listing. redirect output into files. During the process of Figure 2 is an example of a ring map display. This reaching these goals, the product had to be contin­ icon implementation was achieved in far less time uously available as a development tool. than a graphics display solution would have taken A screen-based, user interface solution, known as and offe rs the user a picture of the FDDI ring map, the DECelms screen manager (ESM), was designed.

FOOI Ring Map As of: 14-NOV- 1990 09:52:48 Name : Defcn Address : 08-00-28-13-E0-88

P - Primary Ring Concentrator : I I Station : S - Secondary Ring 7T\

Attachment s: trunk a s b -- == I - - I

Ring Station Station Name Station IO :S p Icon Station Type MAC Address (MAC#) v Active (concentrator) ports

I +== I I OEFCN !0=00-00-08-00-28-13-E0-88 I 7T\ 8-port 1-mac OAC 08-00-28-13-E0-88 I Active : 3, 4, 5, 6 I + -I 8RIOGE1 !0=00-00-08-00-28-14-03-29 SAS 08-00-28-14-03-29 I + -I WRKSTATION !0=00-00-08-00-28-14-03-87 SAS 08-00-28-14-03-87 I + -I 8RIOGE2 !0=00-00-08-00-28-14-00-BA SAS 08-00-28-14-00-8A I + -I 8RIOGE3 I0=00-00-08-00-28-14-00-C1 SAS 08-00-28-14-00-C1 I +== I I OEFCN2 I0=00-00-08-00-28-14-13-9E I 7T\ 8-port 1-mac OAC 08-00-28-14-13-9E I Active : 3, 4, 5 I + -I 8L0 18RIOGE !0=00-00-08-00-28-14-03-23 SAS 08-00-28-14-03-23 I + -I 3RDPARTYSAS !0=00-00-08-00-28-14-03-24 SAS 08-00-28-14-03-24 I 1 + -I OEFE8 !0=00-00-08-00-28-14-00-CO SAS 08-00-28-14-00-CO

End Of Map

Note that attachments a, b, and s are not illustrated in this example .

Figure 2 FDDI Ring Map

Digital Te chnical journal Vo l. 3 No. 2 Sp ring 1991 83 Fiber Distributed Data Interface

ESM uses VMS screen management (SMG) to provide priority and the motivation fo r the DECelms devel­ the basic screen manipulation. Two screen modes opment team to strive fo r excellence. were uevdopeu: an input mode and an output mode. In the input screen mode, commands are echoed Acknowledgments and error messages displayed. This moue provides The DECelms product was a significant accompl ish­ command line recall, command line editing, anu ment. completed in a short period of time. It was the support fo r command files. The output screen moue result a great team effort and personal sacrifice by is used to display the data from command requests. the DECelms development team. I want to recognize This mode provides the user with scrolling driven the fo llowing individuals fo r their efforts as members by the keypad and options to move the output data of the engineering team: Barry Colella, Mary Ellen inro files. The integration of ESM into the DECelms Monette, Phil Morano, Ken Pruyn, and Herman software was accomplished through a well-defi ned Levenson. In audition, I want to thank Andrew interface that could take the output of the original Shores fo r his diligence and endurance in produc­ RBMS output and convert it into the required format ing the DECelms user documentation. Lastly, I want ESM . fo r One by one, the old screens were replaced to rhank members of the FDDI program team, the by ESM, and at any given time during development DECbridge 500 and DECconcentrator 500 teams, there was a working user interface that provided and the NAC architecture group fo r their support debug support fo r the firmware developers. and contributions that made the DECelms product a success story. Summary This paper presents two important themes. The first References concerns the technical challenges and accomplish­ I. P Ciarfclla,D. Benson, and D. Sa·wyer, "An Over­ ments of the DECelms product. Among the challenges view of the Common Node Software," D(r;ital were defining the network management archi tec­ Te chnical.Journal, vol. 3, no. 2 (Spring 1991, this ture, including the protocol, the manageable enti­ issue): 42-52. ties and attributes, and the SMT gateway. Te chnical accomplishments induued the design of NIMUX 2. R. Kochem. ). Hiscock, and B. Mayo, "Develop­ and its multifaceted fu nctionality. The second ment of the DECbridge 500 Product," Digital theme was how the development team proceeded Te chnical.fournal, vol. 3, no. 2 (Spring 1991 , this to build the product. The time to market became issue): 53-63. the controll ing factor in many product decisions. Trade-offs were made in favor of product fu nction­ :). W. Tiffany, G. Koning, and J. Kuenzel, "l'he ality and meeting users' expectations rather than DECconcentrator 500 Product," Digital Te chn ical to promote and preserve architectural and design Jo urnal, vol. 3, no. 2 (Spring 1991, this issue): purity. Above all, product quality remained the top 64-75

H4 �-'<'>!. .> :vo. 2 Sp ri11g 1')91 Digital Te cb nical jo urnal Ursula Sinkewicz Chran-Ham Chang Lawrence G. Palmer Craig Smelser Fred L. Te mplin

UL TRIX Fiber Distributed Data Interface Ne tworking Subsystem Implementation

The ULTRJX operating system, Digital's version of the UNIX op erating system, supports the first implementation of a host networking subsystem with a fi ber distributed da ta interface (FDDI) network adapter. Digital'sFDDJco ntroller 700 adapter provides a single FDDI attachment fo r the reduced instruction set computer (RISC)-based, DECstation 5000 model 200 platform. Co mbined with the ULTRJX networking sub­ system, this adapter brings high-speed communication direct�y to the workstation.

Digital made the decision to adopt fiber distribu ted system, the ULTIUX operating system operates in data interface (FDDI) local area network (IAN) tech· user and kernel modes. A process running in user nology to follow Ethernet. With the FDDI system, mode can be preempted. Interrupts are run in the Digital is developing products to support improved context of the current process. A process running network performance such as the high-speed inter­ in kernel mode voluntarily rei inquishes control connection of workstations. of the CPU. ULTRL'X networks and communications The ULTRIX operating system supports Digital's device drivers run in kernel mode. first implementation of an FDDI host networking The ULTRIX operating system supports network subsystem. A key decision in the ULTRIX FDDI activity through a we ll-defined , layered hierarchy program was to design an adapter fo r reduced including user applications, system calls, and com­ instruction set computer (RISC)-based workstations. pile-time entry points to the protocols and com­ Consequently, the DEC FDDicontroller 700 network munication device drivers. The layered hierarchy is adapter was designed to support an FDDI single illustrated in Figure 1. attachment fo r the DECstation 5000 model 200, The user layer consists of applications (e.g., elec­ RISC-basecl workstation. This support covers the tronic mail) that use specific system ca lls to support Defense Advanced Research Projects Agency (DARPA) network activity. These interprocess-communication internet network protocols designed fo r the system calls incorporate the notion of a socket and , ARPANET packet-switched network. The DARPA hence, are referred to as socket system calls. Sockets internet network protocols include the internet are endpoints of communication containing infor­ protocol (IP), the transmission control protocol mation used by the operating system to associate (TCP), and the user datagram protocol (UDP). data with specific clients and servers. When exe­ This paper begins with an overview of the ULTRIX cuting in kernel mode, the socket system calls operating system. The sections that fo llow present perform the memory management, the security the implementation details of the network and checking, and the state management common to communication driver, review specific issues in all protocols. When the protocol-common process­ the ULTRIXFDDI implementation, and discuss both ing is complete, the operating system accesses a performance and future directions. protocol switch table containing vectors to protocol­ specific modules. These modules, in turn, access Overview of the UL TRIX Op erating communications drivers through the network Sy stem interface table. The ULTRIX operating system is based on the 4.3 The ULTRJX environment is characterized by a BSD system. (BSD refers to Berkeley Software large number of servers, i.e., SUN's NFS system, which Distribution, a popular version of the UNIX operat­ allows remote access to entire fi le systems, and ing system.) As in other systems based on the UNIX many network applications. The servers support a

Digital Te ch nical jou rnal Vo l. 3 No. 2 !>jJring 1991 85 Fiber Distributed Data Interface

groups to write requirements fo r FODI adapters fo r USER LAYER APPLICATIONS both ruse and VAX processors. Due to the evolving ULTRlX ULTRJX SOCKET SYSTEM CALLS emphasis on ruse-based solutions, the engineering group represented data structure, vir­ tual addressing, and performance requirements fo r RJSC processors, while the VMS engineering group UDP TCP LAT DLI AND represented the same requirements fo r VAX proces­ MOP sors. Approximately ten months after the initial network architecture presentations, the FOOl pro­ gram team drafted product requirements for the IP ULTRlX implementation, including support fo r an

LLC AND MAC FOOl workstation adapter. To provide a workstation solution, members of FDDI, ETHERNET, DDCMP COMMUNICATIONS DRIVERS the ULTRlX engineering group had already begun to work with the Low End Network Systems (LENS) KEY: Group on an advanced development project to UDP USER DATAGRAM PROTOCOL TCP TRANSMISSION CONTROL PROTOCOL define a workstation-based FDDI adapter. The team IP INTERNET PROTOCOL discussed alternatives fo r FDDI workstation con­ LAT LOCAL AREA TRANSPORT DLI DATA LINK INTERFACE nections, including the emerging OECstation 5000 MOP MAINTENANCE OPERATION PROTOCOL LLC LOG ICAL LINK CONTROL model 200 TIJRBOchannel bus, the OECstation 3100 MAC MEDIA ACCESS CONTROL plug-in option, and the industry-standard smal l com­ DDCMP DIGITAL DATA COMMUNICATION (SCSI) MESSAGE PROTOCOL puter systems interface bus. Six months into the adapter advanced develop­ Figure 1 ULTRJX Network Sub:>J'Stem Layering ment project, the internet community confirmed interest in TCP/IP implementations fo r FDDI require­ ments by issuing a draft of the request for com­ diverse range of activities such as managing mail and ment, R.FC 1103 (recently renamed RFC 1188), which ensuring that X Window System managers are avail­ defines the encapsulation of imernet packets on able to remote workstations. The underlying pro­ FOOl networks. Members of the FDDI engineering tocols fo r most of these servers are TCP, IP, and UOP. team were instrumental in providing direction for UL TRJX Supportfo r the FDDI Sy stem­ the internet FDOI task force meetings on R.FC 1103 Development Strategies and Issues and the rom network management information Presentations by Digital's networking architects in base (MIB). The draft of HFC 1103 prompted internet May 1988 brought the earliest news that Digital was vendors to hastily implement FOOl workstation­ pursuing a timed, token ring topology (i.e., FOOl) based products and the LENS group to publish plans in contrast to Ethernet, which employs a carrier for FOOl connectivity to !USC-basedworkst ations sense multiple access with collision detection with ULTRlX support. (CSM.A/CO) data link protocol. Digital's FDOI engi­ In October, at the Interop '89 Conference in San neering program began with product requirements Jose, Califo rnia. several internet vendors showcased fo r a wiring concentrator, a bridge to link Ethernet FOOl products. Although Digital did not show FDD! and FOOl networks, and an FOOl adapter to the VAX products at the conference, this event prompted computer. The FOOl program ream planned only Digital to design an architecturally sound, high­ high-end system direct connectivity to the ring. quality, FOOl solution to gain a competitive edge. Wo rkstations would be connected through the Soon after the conference, the FOOl Data Link existing Ethernet across a bridge. Specification and the project plan fo r ULTRIX sup­ However, the ULTRIX operating system running port fo r the FOOl system were released. Subsequent network applications on ruse workstations was ULTRIX development efforts to support the FDDI already saturating the Ethernet. The UI.TIUX engi­ system produced new networking code fo r the neering group advocated FOOl adapters, not only TURBOchannel device driver, the data link layer, fo r ruse-based servers but also fo r the increasing and the network layer. These efforts paralleled the number of high-end, ruse-basedwork stations. TURBOchannel adapter development efforts. ULTIUX and VMS engineering groups began archi­ A prototype lJLTRIX implementation successfully tectural discussions with the FOOl <.levelopmenr passed 802.2 frames over an Ethernet connection,

86 l'v l. 3 Nv. 2 .\jJ ring 1991 Digital Te e/m ica/ jou rnal ULlRIXFi ber Distributed Data Interface Networking Subsystem Implementation

as required by the American National Standards Operating system changes to improve the perfor­ Institute (ANSI) FDDI standard, to exercise the data mance of the network were made later. The next link and network layer changes necessary fo r FDDI two sections describe the implementation of the support. The product announcement for the link-level architecture and the device driver. Perfor­ TURBOchannel FDDI adapter assigned the official mance changes are discussed in the ULTRlX Network name DEC FDDicontro!Jer 700 to the adapter. Proto­ Performance section. types were delivered in May 1990; firmware integra­ tion was completed; and the first address resolution Data Link Support protocol (ARP) broadcast packet was sent over an The ULTRl.'C operating system implements both the FDDI ring from an ULTRIX host. internet protocols (TCP/UDP/IP) and the Digital Device driver and adapter interoperability prob­ Network Architecture (DNA) model, including the lems such as timing considerations, data corrup­ Digital data link interface (DLl) . In both the inter­ tion, and performance issues were solved promptly net and DNA models, the data link defines services by close cooperation between the software and known as the logical link control (LLC) and the media hardware groups. Several additional performance access control (MAC). A major challenge in the imple­ enhancements were added to the operating system, mentation of ULTRIX FDDI support was defining a bringing the performance of the ULTRJX and adapter set of common data link routines to satisfy the combination to nearly 40 percent of the entire frame fo rmat requirements of both internet and FDDI bandwidth-a factor of fo ur times greater DNA models in a heterogeneous LAN environment. than existing Ethernet implementations. Prior to the introduction of the FDDI system, all At its tracle show, DECWORLD 1990, Digital ULTRlX internet networking fo r LANs ran over announced the availability of its FDDI product offe r­ Ethernet networks using Ethernet V2 frame fo rmats, ings. These included the DECconcentrator 500 and even though ULTRlX DLI networking supported DECbridge 500 products, and the DEC FDDiconrroller both Ethernet V2 and 802.2 LLC frame fo rmats. 700 adapter, which runs under the ULTRlX operat­ Figure 2 illustrates the diffe rences among the V2 ing system. Ethernet and 802.2 Ethernet and the FDDI frame fo rmats. V2 and 802.2 frames include Ethernet UL TRIX Internals encapsulation; 802.2 frames consist of the MAC, the The implementation of FDDI support in the ULTRIX LLC, and data segments. When the 802.2 frame is sent operating system required the development of a over the FDDI system, the FDDI framing adds the link-level architecture and a network device driver. FDDI-specific encapsulation, as shown in Figure 2.

NUMBER OF BYTES 8 14 46-1 500 4 I PREAMBLEI MAC I DATA I FCS I V2 ETHERNET FRAME (MAC CONTAINS PROTOCOL TYPE)

NUMBER OF BYTES 8 14 3-8 38-1497 4 I PREAMBLEI MAC I LLCI DATA I FCS I 802.2 ETHERNET FRAME (MAC CONTAINS FRAME LENGTH)

NUMBER OF BYTES 4 12 3-8 0-44 75 4 2 I PREAMBLEIMACILLcl DATA I FcslsTATusl 802.2 FOOl FRAME (PREAMBLE CONTAINS START DELIMITER, FRAME CONTROL)

KEY

MAC MEDIA ACCESS CONTROL LLC LOGICAL LINK CONTROL FCS FRAME STATUS CONTROL STATUS FRAME STATUS

Figure 2 Frame Fo rmats

Digital Te e/m ica/ jo urnal Vo l. 3 No. 2 Sp ring 1991 87 Fiber Distributed Data Interface

In order to conform to the AJ'\J SI FDDI st:mdarcls, The port architecture defi nes the mechanisms to wh ich require 802.2 LLC frame fo rmarring, members transfer FDDI frames and control or status informa­ of the Internet Network Wo rking Group wrote tion between the communication driver and the Internet RFC 1188.' This RFC specifies the rules fo r port. The liLTRIX com munication driver interfaces 802.2 LLC encapsulation of internet frames on an to the adapter through six port registers, six port FDDI network. To meet the needs of both Ethernet memory rings, the driver data structures, and the and FDDI networks, we designed and integrated a driver data buffers, as shown in Figure 3. set of network-com mon routines. These ro utines, Each port register is represented by 16 bits in the net_output() and the net_read( ) fu nctions, sup­ adapter packet memory. These registers are descri bed port 802.2 encapsulation as required by RFC 1188. in Ta ble I. net_output() Function The net_ output( ) func­ Ta ble 1 Adapter Port Registers tion prepares packets fo r transmission by UI.TRlX network communication device drivers. If a urive r Register Name Written by Purpose requires 802.2 LLC support, the net_output( ) timc­ Port Reset Driver Forces the adapter tion supplies the necessary header, prefixes the MAC to reset header, and enqueues the packet to the appropriate Port Control A Driver Contro ls adapter operat ions communication driver fo r transmission.' The func­ Port Control B Driver Indicates that the tion, while supporting 802.2 l.LC encapsulation, driver is active does not preclude protocol modules from support­ Port Interrupt Adapter Notifies the driver of ing their own LLC fo rmatting. The encapsulation is Event important events switch-driven so implementors can add special rou­ Port Status Adapter Indicates the adapter tines to the switch to either replace or bypass the status 802.2 LJ.C encapsulation. Port Interrupt Driver Masks the adapter Mask interrupt events net_read() Function The net_read() fu nction is called by the communication drivers and prepares The adapter uses six queues, cal l.ed port memory received packets fo r delivery to protocol modules. rings, ro exchange data, events, and commands with This fu nction first identifies the protocol type from the driver. These port memory rings, represented as information contained in the i'v!AC and l.l.C headers. circular lists of descriptors, are described in Ta ble 2. places the packet on the corresponding queue, and Each descriptor is associated with a cl ata bu ffe r in finally schedules a software interrupt to alert the adapter packet memory or in driver memory. appropriate protocol module of the arriving packet.

Co mmunication Driver Ta ble 2 Port Memory Rings The FDD!controller 700 adapter connects directly Ring Name Purpose Description with the DECstation 5000 model 200 ·nmBOchannel Host Receive Data flow Identifies driver data bus. The FDDlcontroller 700 is a 100-megabit (Mb)­ Ring buffers to receive per-second, timed , token ring adaptn that supporrs incoming packets an FDDI single attachment fo r the DEC:stat ion 5000 RMC Tr ansmit Data flow Identifies adapter data model 200. The adapter provides a host interface Ring buffers containing packets to transmit with the fo llowing fe atures: a packet memory inter­ SMT Receive Used by the driver to fa ce (PM I) gate array for receive direct memory access Data flow Ring route SMT frames to (DMA) data transfer; a packet memory subsystem the adapter for that contains one megabyte (MI3)of dynamic random­ processing access memory (DRAM) fo r packet store and fo rwarcl: SMT Transmit Data flow Used by the driver to and the ability to handle .FDDl ring initialization, Ring route SMT frames to the adapter for recovery, and SMT frame processing. (SMT refers to processing the ANSI-specified FDDT station management.') The Command Ring Control Used by the port driver adapter is controlled by a microprocessor and uses to initialize, configure, Digital's FDDl chip set, which includes ring mem­ and monitor adapter ory control (R.J\1C), media access control, and the operat ions elasticity bu ffe r and physica l link manager (EI..\1). Unsolicited Control Used by the adapter Events Ring to notify the driver of The liLTRIX communication driver interfaces to unso lic ited events the adapter's port architecture.

\1•1. .) ,Vu. .! .\jJ ri11,� J')')! Digilal Te ch uicnl jou rnal ULTRIXFi ber DistributedData InterfaceNe tworking Subsystem Implementation

NETWORK COMMON LAYER

�------l SMT : -- I QUEUE COMMUNICATION DRIVER I t : : I I t------J I LLC � I LLC __ _- I " f-1��------r------I :: SMT y I f�

SMT RMC UNSOLICITED COMMAND HOST SMT TRANSMIT EVENTS RING RECEIVE RECEIVE TRANSMIT RING RING RING RING RING

L__ I L__ I ------I ------KEY:

SMT STATION MANAGEMENT LLC LOGICAL LINK CONTROL RMC RING MEMORY CONTROLLER

Figure 3 DEC FDD/controller 700 Port Architecture

The packet data flow between the adapter and initialize the adapter, change the adapter state, mod­ the ULTRIX communication driver is also shown in ify and read the packet filter address table, read data Figure 3. For incoming FOOl packets, the adapter link counters, and read data link status. In addition, uses a direct memory access engine to move the due to the evolving state of the ANSISMT specifica­ packets from the adapter memory to the receive tion, the driver function now al.lows the on-line data buffe rs of the driver. These buffers are allo­ upgrade of adapter firmware.' Finally, the driver sup­ cated as 4-kilobyte (KB) pages in kernel memory. ports the ability to recognize unsolicited adapter Each host receive ring descriptor is associated with events communicated through the unsolicited events two receive buffe rs to handle the maximum FOOl ring. When received, these events are logged and frame (4500 bytes). To achieve the maximum receive reported through the console. throughput, the driver performs packet filtering. If the incoming packet is an LLC frame, the driver UL TRIX Ne twork Peiformance processes it and calls the net_read() function. Performance is a key factor in the success of Digital's Otherwise, the driver fo rwards the packet to the workstation FOOl offering. A great effort was made adapter's SMT receive ring, and then the adapter to characterize the OECstation 5000 machine per­ queues the packet to the SMT transmit ring after fo rmance by using the earlier OECstation 3100 processing. The driver is then notified by the workstation perfo rmance resu lts to help set realistic adapter to move this packet from the SMT transmit goals. Both the characterization and the measure­ queue to the RM C transmit ring fo r transmission. ments were essential to predict the performance When the net_out put() function requests to trans­ goals. This section discusses the level of perfor­ mit packets, the driver copies the packets from mance achieved by the OECstation 5000 model 200 driver memory to the RMC transit ring, signaling running the ULTRIX operating system with FOOl the adapter to transmit the packets. support. We present details of the historical prece­ The communication driver controls and requests dence fo r predicting FOOl performance, the evolu­ information from the adapter by issuing com­ tion of the ULTRIX networking code, and the mands through the command ring. These commands performance of applications with the FOOl system.

Digital Te chnical journal Vo l. 3 No . 2 Sp ring 1991 89 Fiber Distributed Data Interface

Historical Precedence Evolution of the UL TRIX Internet Code

We expect FDDI perfo rmance to develop similarly The early implementations of ULTRIX internet net­ to that of Ethernet. Early Ethernet hosts were unable wo rking code were based on robust BSD network­ to utilize more than 20 percent of the available net­ ing code. In 1987, the lJLTRJX internet networking work bandwidth because of the limited throughput code was updated to incorporate performance capability of existing processors. The graph shown enhancements available from a recent BSD release. in Figure 4 illustrates the historical performance Later, UI.TRJX network performance was fu rther of several different processors using Ethernet improved to include the implementation of a adapters. Note that since 1983, network through­ dynamic buffe r allocation policy to replace the put has increased significantly At some time after inefficient static allocations. With FDDI systems, the middle of the decade, it was possible to reach a the challenge then became adapting the code to throughput of 10 Mb per second, a rate high enough effectively use the higher network throughput. to saturate Ethernet with a single host. We attacked this problem by isolating and opti­

Figure 4 also shows that, in nearly all cases, the mizing each networking subsystem. The ULTRIX achievable throughput is limited by the speed of networking code is divided into three major sub­ the processor. In addition, an experimental constant systems: sockets, protocols, and drivers. Each of of 1 mips/Mb is measured in cases where the proces­ these subsystems can be further subdivided: sock­ sor is saturated. (1 mips equals one million instruc­ ets into the system call interface and the socket-to­ tions per second.) This constant means that 1 mips of protocol interface; protocols into the IP and UDP processor speed is needed to generate 1Mb of net­ components; and drivers into the ARP, buffe r man­ work traffic. For example, the 1-mips VAX-11/780 agement, and driver interrupt components. We used processor is able to generate about 1Mb of network kernel profiling, a means fo r making run-time mea­ traffic. The data presented in Figure 4 shows that a surements of time spent in system-level routine processor fo llows the I mips/Mb ratio unless the calls, and known benchmarks to track progress. adapter becomes a limiting factor, as in the case of Using an unrel iable protocol without error the DECstation 3100 system. The 1 mips/Mb ratio recovery, such as UDP, instead of TCP with relia­ allows us to predict that a 100-mips processor is bility and packet sequencing fe atures, the packet­ required to satu rate the FDDI system. Thus, the per-second (pps) rate of each component can be FDDI system satisfies the throughput requirements easily determined. Figure 5 shows the sizable packet of available processors and allows fo r the growth rate measured on the DECstation 5000 model 200 of fa ster processors. Finally, if the present trend fo r both the nonoptimized and the optimized of doubling processor speed every two to three ULTRIX network, i.e., before and after performance years continues, the FDDI graph will resemble the improvements are incorporated. Note that the pps Ethernet graph of the 1980s, with the saturation of rate of each layer reflects improvements in the the FDDI system possible in 1996 or 1997 layers below.

40 f-o :::l z a.. o DECSTATION Iu 3 VAX 5000 0 � 8500�� � MIPS <.:>w 3 M IP :::l (IJ 25 � Oa: 1-; a:w � a.. 20 MICROVAX II p �(/) ����;�T���O- a:'= --_:_�� - - - - - S�N�-- ETHERNET ooo � MAXIMUM 10 \- -- � MIPS �<>: VAX "' 9 f-<.:> 11·780 ww MIPS z� 1 � 0 3 1980 81 82 8 84 85 86 87 88 89 90 91 YEAR

KEY:

... OBSERVED THROUGHPUT RATE 0 POSSIBLE THROUGHPUT RATE BAS ED ON PROCESSOR SPEED MIPS MILLION INSTRUCTIONS PER SECON D

Figure 4 Historical Performance of Processors and Ethernet Adapters

90 vul. 3 No . .2 Sp ring 1991 Digital Te c/Jnicaljo unwf UL TRIX Fi ber Distributed Data InterfaceNetwo rking SubsystemIm plementation

RATE (PACKETS PER SECOND) into the kernel, the IP layer checksums the data, LAYERS OF THE ULTRIX OPERATING SYSTEM NONOPTIMIZED OPTIMIZED and the driver start routine copies the data to the adapter. We fo cused our efforts on the socket-to­ SYSTEM CALL 16000 16000 protocol layer and fo und that considerable proces­ sor time was spent allocating kernel buffe rs to SOCKET-TO-PROTOCOL 1750 2650 hold the data. Reworking this code to buffe r the data more efficiently resulted in the performance UDP 1700 2570 change between the optim ized and nonoptimized columns shown in Figure 5.

IP 700 1400 Application of the FDDI Sy stem ARP 610 1230 The greatest long-term benefit of end-node access to FODI will probably come to those utilizing a

DRIVER START 600 1200 distributed computing environment since this paradigm relies heavily on the performance of

DRIVER DONE 300 600 the underlying network. While Ethernet currently serves this growing set of applications well, it is

NOTE: THE DECSTATION 5000 PLATFORM RUNNING UNDER THE ULTRIX expected that as the applicat ion demand fo r net­ OPERATING SYSTEM HAS A 4096-BYTE PACKET-PER-SECOND RATE. work service increases, so wi ll the total network KEY: bandwidth and performance requirements of the UDP USER DATAGRAM PROTOCOL participating end node. IP INTERNET PROTOCOL ARP ADDRESS RESOLUTION PROTOCOL Even today, some users of distributed network file systems (e.g., NFS) will notice a significant per­

Figure 5 Op timized and No noptimized Pa cket fo rmance improvement as a direct result of using Rates fo r ULTRIX Network Co mponents FOOl. This improvement is particularly evident on cached file reads and writes, where no disk access Packet rate values fo r the system call through the is required but the aggregate bandwidth advantage ARP components are determined by processor speed of FOOl is beneficial. However, overall NFS perfor­ and code. Rates below the ARP depend on adapter mance is currently lim ited by CPU speed and disk speed and not processor speed. The packet rate value write capabilities; so we expect that with improve­ for the ARP is the maximum packet rate fo r a proces­ ments in processor performance, disk access, and sor. With an optimized packet rate, Figure 5 shows a data caching, network performance at the level maximum rate of 1230 pps fo r the OECstation 5000 provided by FOOl will soon be essential. workstation. Since each test packet contains 4096 Ta ble 3 contains FOOl performance measurements bytes, this rate is equivalent to 40Mb per second, with respect to Ethernet fo r various applications which is a 40 percent FOOl bandwidth utilization. and transports at the application layer. The spray A significant amount of work is performed at the application is most often used to measure how an socket-to-protocol, IP, and driver start layers because unreliable transport performs. Applications such in each case, an effective copy of the data is per­ as the BSD rep (the remote file copy program over fo rmed. The socket layer copies data from the user TCP/IP protocols), the file transfer protocol (FTP),

Ta ble 3 Application Performance in Relation to Ethernet

Rate (Megabits) Rate (Megabits) Transport Application (UDP Checksum On) (UDP Checksum Off) Change

TCP rep 18 18 2X TCP FTP 5 5 1X UDP/NFS NFS Read 20 30 2X (3X) UDP Spray 22 35 2.5X (3.3X) TCP TICP 18 18 2X UDP TICP 22 38 2.5X (4X)

Digital Te clmical jo urnal V!;l. 3 No. 2 Sp ring 1991 91 Fiber Distributed Data Interface

and the test TCP (TTCP) program all measure per­ Future fo rmance of a reliable transport protocol. An NFS This section describes some areas of research that test is used to measure how FDDI performs as a file may impact the use of the FDDI system. Included server. Note that, in all cases except FTP, perfor­ are discussions on protocol alternatives, future per­ mance improves by at least a factor of two. FTP does fo rmance work, and how this system will facilitate not take advantage of the larger buffering gained new software technologies. by using the FDDI system and, thus, shows no per­ formance change over Ethernet. New Protocols Another aspect of network performance is the As illustrated in Figure 4, processor speed is the routing function. Using the DEC FDD!controller 700 current bottleneck in FDDI throughput. While adapter, the DECstation 5000 model 200 can perform processor speed continues to increase, emerging FDDI-to-FDDI routing, FDDJ-to-Ethernet routing, or protocols are making efficient use of available pro­ both, fo r internet traffic. With a built-in Ethernet cessing power and are yielding additional gains in port and the ability to accept up to three addi­ network performance. tional TURBOchannel adapters, a 5000 model 200 A development relevant to this discussion of pro­ can connect to as many as fo ur different networks. tocol alternatives is the versatile message transport The performance of such a host-based router protocol (VMTP) research project from Stanford is difficult to characterize. A wide range of factors University. ' VMTP demonstrates that a rei iabJe influences this performance, including the proto­ transport is achievable with no greater overhead cols routed, the efficiency of the routing algo­ than existing unreliable transports. VMTP, therefo re, rithms, the system load, and the available data link represents an alternative to TCP that would nearly bandwidth. Nonetheless, it is usefu l to consider the double the throughput of some remote procedure performance of the 5000 model 200 serving as an call (RPC) applications. Also, knowledge gained from FDDI router because we expect this product fea­ the VMTP research may influence fu ture internetor ture to be popular. Ta ble 4 shows the performance open systems interconnection (OSI) directions. results fo r a DECstation 5000 workstation perform­ ing FDDI-to-FDDI routing and FDDJ-to-Ethernet Future UL TRIX Network Performance routing, both under minimal system and network Wo rk load. Note that the data presented fo r the TCP­ In addition to examining new protocols, perfor­ based applications shows that throughput is lim­ mance work is continuing with our existing ULTIUX ited by the way TCP calculates its flow control protocols. One area being studied is data copy. Cur­ window when data is destined fo r a remote net­ rently, user data is copied twice as it traverses the work. All current TCP implementations have this internet protocol stack. One copy occurs in the same limitation because all nonlocal connections socket subsystem, and the other one takes place in must have a small flow control window size of the device driver. Data copies account fo r 50 per­ 576 bytes to avoid the saturation of intermediate cent of CPU utilization time when large amounts of gateways. Since both Ethernet and FDDI systems data are transferred. Eliminating one copy can yield can transmit frames larger than this flow control increased performance, with savings of at least 25 window, the advantage of transmitting maximum­ percent of the total processing time. sized frames is lost. UDP does not have this limita­ An FDDI adapter optimized fo r the internet proto­ tion; thus, throughput numbers are only slightly col stack may also provide improved performance. lower than in the nonrouting case. This decrease in processing time may result from

Ta ble 4 FOOl Routing �or a OECstation 5000 Workstation

Ethernet-to-FDDI FDDI-to-FDDI Transport Application Rate (Megabits) Rate (Megabits)

TCP rep 4.8 4.8 TCP FTP 2.7 2.7 UDP/NFS NFS Read 8.0 16.8 UDP Spray 9.0 18.0 TCP TICP 6.4 6.4

92 Vo l. 3 No. 2 Sp ring 1991 Digital Te cbnicaljournal ULTRIXFiber Distributed Dat.alnte1jace Networking Sub�J'Stem Implementation

calculating internet checksums in the adapter or Acknowledgments from moving the complete protocol stack to the We would like to thank Kathy Wilde fo r her early adapter. For example, researchers have proposed participation on the project, Kent Ferson fo r gen­ protocol engine chips that would off-load all pro­ eral support and encouragement, and Fred Glover tocol processing to customized chips. With such fo r his help in reviewing the document. Nolan Ring real-time protocol engines, ex isting processors patiently edited the early draft. Julia Fey helped could easily outpace current FDDI speeds. with the collection of performance data and with Conclusions the data analysis. We wou ld especially like to thank the staff of the Digital Te chnical jo urnalfo r their Digital brought FDDI to the ULTRlX workstation to help ancJ support. satisfy the growing network demands of its cus­ tomers. The number of network-intensive applica­ References tions that run on ULTRlX workstations is growing at a fast pace. Graphics and imaging applications have 1. A Proposed Standardfo r the Transmission of iP the potential of generating megabytes of network Datagrams over FDDI Networks, Internet Net­ data. Also, multimedia applications can strain FDDI work Working Group, RFC 1188 (October 1990). networks and are not practical using Ethernet. The 2. Fi ber Distributed Data Interfa ce (FDDI) -TrJken best scenario fo r a live motion video application Ring Media Access Control (MAC), A.i'\JSI X31:)9-1987 is the requirement of at least a 1-MB, continuous 3. ANSI FDDI Station Ma nagement Standard, ANSI network connection, enough to easily saturate X3T9.5, Revision 5.1 (September 1989). Ethernet. Even a live audio application will require 4. Mason, "VMTP: A High Performance ·rransport a 200- to 300-KB-per-second network connection. W Protocol," Conne.xions, vol. 4, no. (June Clearly, with applications that demand these data 6 rates, FDDI bandwidth is necessary. 1990): 2-10. The DEC FDDicontroller 700 adapter brings FDDI to the desktop. The adapter is well matched to the General References DECstation 5000 model 200, joining a 25-mips pro­ cessor to a 100-Mb-per-second data link. As the next A Standard fo r the Tra nsmission of IP Datagrams generation of LAN, FDDI extends the base fo r net­ over IEEE 802 Networks, Internet Network Wo rking work applications by allowing those appl ications Group, RFC 1042 (February 1988). that run on Ethernet to run faster and by providing ANSI/IEEE Standards fo r Local Area Networks: the bandwidth fo r a whole new generation of appl i­ Logical Link Co ntrol, ANSI/IEEE Standard 8022-1985 cations. FDDI is the network of the '90s, as Ethernet (New Yo rk: The Institute of Electrical and Electron­ was the network of the '80s. ics Engineers, Inc., 1985).

Digital Te ch nical jo ur11al Vo l. .i No. 2 Sp ring 1991 9.1 I Further Readings

The Digital Te chnical]ournal Distributed Systems publishes papers that explore Vo l. 1, No. 9,]une 1989 the technologicalfo undations Products that allow system resource sharing of Digital'sma jor products. Each throughout a network, the methods and tools journalfo cuses on at least one to evaluate product and system performance product area and presents a Storage Technology compilation of papers written Vo l. 1, No. 8, February 1989 by the engineers who develop ed Engineering technologies used in the design, the product. Th e contentfo r manufacture, and maintenance of Digital's storage the jo urnalis selected by the and information management products jo urnal Advisory Board. CVAX-basedSystems Vo l. 1, No. 7, August 1988 To pics covered in previous issues of the Digital CVAX chip set design and multiprocessing architec­ Te chnicaljo urnal are as fo llows: ture of the midrange V�'C 6200 family of systems and the MicroVAX 3500/3600 systems Transaction Processing, Databases, Software Productivity Tools and Fault-tolerant Systems Vo l. 1, No. 6, February 1988 Vo l. 3, No. 1, Winter 1991 To ols that assist programmers in the development The architecture and products of Digital's of high-quality, reliable software distributed transaction processing systems, with information on monitors, performance VAXcluster Systems measurement, system sizing, database availability, Vo l. 1, No. 5, September 1987 commit processing, and fa ult tolerance System communication architecture, design and implementation of a distributed lock manager, VAX 9000 Series and performance measurements Vo l. 2, No. 4, Fall 1990 The technologies and processes used to build VAX 8800 Family Digital's first mainframe computer, including Vo l. 1, No. 4, February 1987 papers on the architecture , microarchitecture, The microarchitecture, internal boxes, VAXBI bus, chip set, vector processor, and power system, and VMS support fo r the VAX 8800 high-end multi­ as well as CAD and test methodologies processor, simulation, and CAD methodology

DECwindows Program Networking Products Vo l. 2, No. 3, Summer 1990 VOL. 1, No. 3, September 1986 An overview and descriptions of the enhancements The Digital Network Architecture (DNA), network Digital's engineers have made to MIT's X Window perfo rmance, L<\Nbridge 100, DECnet-ULTRIX and System in such areas as the server, toolkit, interface DECnet-DOS, monitor design language, and graphics, as well as contributions MicroVAX II System made to related industry standards Vo l. 1, No. 2, March 1986 VAX 6000 Model 400 System The implementation of the microprocessor and Vo l. 2, No . 2, Sp ring 1990 floating point chips, CAD suite, MicroVAX work­ The highly expandable and configurable midrange station, disk controllers, and TK50 tape drive family of VA)(. systems that includes a vector pro­ VAX 8600 Processor cessor, a high-performance scalar processor, and Vo l. 1, No. 1, August 1985 advances in chip design and physical technology The system design with pipelined architecture, Compound Document Architecture the I-box, F-box, packaging considerations, signal Vo l. 2, No. 1, Win ter 1990 integrity, and design fo r reliability The CDA family of architectures and services that support the creation, interchange, and processing Subscriptions to the Digital Te chnicaljo urnal are of compound documents in a heterogeneous available on a yearly, prepaid basis. The subscrip­ network environment tion rate is $40.00 per year (four issues). Requests

94 Vo l. 3 No. 2 Sp ring 1991 Digital Te chnlcaljounzal should be sent to Cathy Phil lips, Digital Equipment A. Garde! and P Deosthali, "Using Simulation in Corporation, ML01-3/B68, 146 Main Street, Maynard, Semiconductor fabrication," Electro '90 (May 1990). MA 01754, U.S.A. Subscriptions must be paid in U.S. C. Gilbert, "Improving Distance Learning in dollars, and checks should be made payable to Industry," College Industry Education Co nference Digital Equipment Corporation. (February 1990). Single copies and past issues of the Digital H. Hvostov anu M. Luksic, "On the Structure Te chnicaljo urnal can be ordered from Digital of H-Infinity Controllers," American Control Press at a cost of $ 16.00 per copy. Co nference (May 1990).

Technical Papers by Digital Authors A. Incorvaia, "Case Studies in Software Reuse," IEEE Software Applications Conference (November 1990). E. Atakov, ·' Electromigration Resistance of TiN­ layered Ti Doped A1 Interconnects," !ELE VL'il K. Mistry and B. Doyle, "The Role of the Electron MulUleuel Conference Oune 1990). Trap Created in Enhanced Hot-carrier Degradation during AC Stress," IEEE Electron Device Letters R. Bansal, H. Fu, C. Haggerty, anu P Posco, "Hyper­ Oune 1990). info rmation Systems and Te chnology Transfer,'' ASMT: Computers in Engineering (August 1990). C. Nofsinger and Z. Cvetanoric, "Parallel As tar on Message-passing Architectures," International D. Benson, P Ciarfe lla, and P Hayden, "FDIC: Co nference on Sy stem Sciences Oune 1990). Meeting the lnteroperability Challenge," IEEE Local Co mputer Networks (October 1990). S. Novotny, "Performance Evaluation of a Giffo rd­ McMahon Refrigerator fo r Cryogenically Cooled D. Bhandarkar and R. Brunner, "Vfu'{ Vector Computer Applications," 1-Therm II(May 1990). Architecture," International Sy mposium on Cornputer Architecture (May 1990). P Riley and T. Clark, "Integrated Deposition and Etchback of Tungsten in a Multichamber, Single G. Champine and D. Geer, "Project ATHENA as a Wafe r System," IEEE VLSI Multilevel Conference Distributed Computer System," IEEE Computer Oune 1990). (September 1990). M. Sidman, "Parametric System Identification on R. Col lica, "The Physical Mechanisms of Defect Logarithmic Frequency Response Data," American Clustering and Its Correlation to Yield Model Control Co nference (May 1990). Parameters fo r Yield Improvement," Advanced Semiconductor Manufacturing Co nference E. Zimran, "Generic Material Handling System," (September 1990). InternationalCon ference on C!M(May 1990).

R. Csencsits, ). Rose, N. Riel, and ). Dion, "Cross Section Preparation of CU/PI Interfaces fo r TEM Digital Press Analyses," Electron Microscopv '90 (August 1990). Digital Press is the book publishing group of Digital D. Dalal, "A 500 KHz Multi-output Converter with Equipment Corporation. The Press is an interna­ Zero Voltage Switching," IEEE Applied Power tional publisher of computer books and journals Electronics (March 1990). on new technologies and products fo r users, system and network managers, programmers and other S. Das and W Chaurette, "Self Heating Effects on MR professionals. Proposals and ideas fo r books in Head Performance," IEEE lntermag '90 (April 1990). these and related areas are welcomed. B. Doyle and K. Mistry, "The Generation and Char­ acterization of Electron and Hole Traps Created by VAX/VMS: Wri ting Real Programs in DCL Hole Injection during Low Gate Voltage Hot-carrier Paul C. Anagnostopoulos, 1989, softbound, Stressing of N-MOS Transistors," IEEE Tra nsactions 409 pages ($29.95) on Electron Devices (August 1990). X WINDOW SYSTEM TOOLKIT: TheComplete A. Enver and). Clement, "Finite-element Numerical Progranuner's Guide and Specification Modeling of Currents in VLSIInter connects," IEEE Paul J Asente and Ralph R. Swick, 1990, softbound, V/51 Multilevel Conference Oune 1990). 967 pages ($44.95)

Digital Te ch nical jo urnal Vo l. .l No. 2 Sp ring 1991 95 r:urther Readings

UNIX FOR VMS USERS TECHNICAL ASPECTS OF DATA Philip E. Bou rne, 1990, softbound, COMMUNICAT ION, Third Edition :)Mlpages ($2H 9'5) john E. McNamara, 1988, hardbound, :383 pages ($42 00) VAX ARCHITECTURE REFERENCE MANUAL, Second Edition LISP STYLE and DESIGN

lli chard A. Brunner. Ed itor, 1991 , softbound. Molly M. J\ll ilkr ami Eric Benson, 1990, softbound, '560 pages ($44.9'5) 214 pages ($26 9'5)

SOFTWARE DESIGN TECHNIQUES FOR lARGE THE VMS USER'S GUIDE

ADA SYSTEMS James F Peters III and Patrick ,I. Holmay, 1990, William E. Byrne. 1991 , hardbound, softbound. 304 pages ($28.95) :314 pages ($44.9'5) THE MATRIX : Computer Networks and INFORMATION TECHNOLOGY STAN DARDIZA­ Conferencing Systems Worldwide TION: Theory, Pract ice, and Organizations John S. Quarterman, 1990, softbound,

Carl F Cargi ll, 1989. softbound, 719 pages ($49.9'5) 2'52pages ($24.9'5) X AND MOTIF QUICK REFERENCE GUIDE THE DIGITAL GUIDE TO SOFTWARE Randij. Rost, 1990, softbound, DEVELOPMENT 369 pages ($ 24.9'5) Corporate User Publication Group of Digi tal FIFTHGENER ATION MANAGEMENT: Equipment Corporation, 1990. softbound, Integrat ing Enterprises Through Human 2:)9 pages ($279'5) Networking DIGITAL GUIDE TO DEVELOPING Charles M. Savage, 1990. hardbound. INTERNATIONAL SOFTWARE 267 pages ($289'5) Corporate User Publication Group of Digital A BEGINNER'S GUIDE TO VAX/ VMS UTILITIES Equipment Corporation, 1991, softbound, AND APPLICATIONS 400 pages ($28.9'5) Ronald M. Sawey and Troy T Stokes, 1989, VA X/VMS INTERNAlS AND DATA softbound, 278 pages ($26.95) STRUCTURES: Ve rsion 5-2 X WINDOW SYSTEM, Second Edition Ruth E. Goklenherg ami Lawrence J Kenah. with Robert Scheifler and James Gettys. 1990. the assistance of Den ise E. Dumas. 1991, hardbound. softbound. 8'51 pages($ 49.95) approximately I .:300 pages ($124.95) COMMON LISP: The Language, Second Edition COMPUTER PROGRAMMING A.l'IID Guy Steele .Jr., 1990, 1,029 pages ($:18.9'5 in ARCHITECTURE: The VA X, Second Edition L softbound, $46.9'5in hardbound) Henry M. Levy and Richard H. Eckhouse_lr., 1989, hardbound. 444 pages ($:18 00) WORKING WITH WPS-PLUS Charlotte Te mple and Dolores Cordeiro, 1990, USING MS-DOS KERMIT: Connecting Yo ur PC softbound, 23'5 pages ($24.95) to the Electronic World Christine M. Giaoone. 1990. softbound. To receive a copy of our latest catalog or info rma­ 244 pages, with Kermit Diskette ($29.9'5) tion on rhese or other publications from Digital THE USER'S DIRECTORY OF COMPUTER Press, write or ca ll: NETWORKS Digital Press Tracy L LaQuey, 1990. softbound, Department DTJ 6:)0 pages ($ 34 95) 12 Crosby Drive SOLVING BUSINESS PROBLEMS WITH MRP II Bedford, MA 01730 Alan D. Luber, 1991. hardbound, 617/276 -1'536 3:33pages ($34.95) Or, to order, call DECd irect at VMS FILE SYSTEM INTERNALS 800-DIGITAL (800 - 344 -4825). Kirby McCoy, 1990, softcover, 460 pages ($499'5)

96 llril. ,) Nn. 1 .\j1ri11g 199 1 Digital Te ch nical jou rrwl .\

ISSN 0898-901X

Printed in U.S.A. EY-H876E-DP/91 03 02 16.0 MCG/BUO Copyright © Digital Equipment Corporation. All Rights Reserved.

------