Imperial College of Science and Technology, University of London, Department of Computing.

HIGH EFFICIENCY, CHARACTER-ORIENTED, LOCAL AREA NETWORKS

by

Martin Cripps

This thesis is submitted in partial fulfilment of the requirements for the degree of Doctor of Philosophy and the Diploma of Imperial College of Science and Technology, January 1988. For Clare

Attempt the end His reasons are as two grains of wheat but never stand to doubt hid in two bushels of chaff. nothing's so hard You shall seek all day ere you find them but search and when you have found them will find it out they are not worth the search.

Robert Herrick (1591-1674) William Shakespeare (1564-1616) 1

ABSTRACT

This thesis explores the problem of interconnecting character-oriented devices over local area networks by investigating significant aspects of hardware, software, protocol and operational factors. It proposes effective and efficient solutions which were tested during a full-scale experiment The results of that experiment demonstrate convenient, cost-effective and reliable operation. The novelty of this investigation arises from its character-oriented approach. Much work has been carried out by others on local area networks which transfer blocks of data efficiently, however, a large majority of installed devices operate on a character-by-character basis and will continue so to do for some considerable time. This study is approached through analysis of the low efficiency of international standard networks for this class of device which defines the scope of this work. An original analysis of the potential mechanisms which can be used to give high efficiency and low delay for this class of transfer is then derived. This leads to the proposal of an entirely new structure for interconnection with its associated protocols.

A unidirectional, baseband bus is developed and shown to have a unique combination of highly desirable properties which are not available together in any ring, mesh or simple-bus structure. These properties are: simple cabling, unbroken network media, densely packed data on the media, low delay, fault location in normal operation and reconfiguration in the event of partial failure. The protocols which are proposed are specifically aimed at small-data transfer and it is shown how, by using contention-free access, they approach optimal efficiency while maintaining deterministic behaviour. The logical and electrical characteristics of the new bus structure (Synchronet) and its access mechanisms are presented and their performance demonstrated. The Synchronet structure also has more general capabilities, such as for large block transfers, and could be the subject of further work.

A full-scale experiment (Sonet), set up to demonstrate the overall viability of the proposals, is discussed with its hardware and software aspects. Specific modifications for a Synchronet to operate at extremes of speed and efficiency are also shown. The experimental results are described including the novel fault detection and fault tolerant aspects of the system. The value of Sonet was illustrated during the four years (1983-87) that it formed the major network in DoC at ICST. The general principles remain valid, however, as with all scalable technologies, improvements in technology are directly mirrored in improvements in the overall network. The use of the new technology and protocols proposed here provides for efficient small to medium sized, character- oriented networks. It is shown how they can be used to provide distributed multiplexer feeders to existing standard networks with simple, single-path wiring giving a cost-effective and convenient way of connecting dispersed, character-oriented devices. 2

ACKNOWLEDGEMENTS

As I have been working in this area for, to use the Registry's phrase, some considerable time, there are many more people I wish to thank for their advice, assistance and kindness than I can possibly list here, but they all have my continuing gratitude.

I particularly wish to thank my longtime friend and former supervisor Peter Throsby for all his guidance and help. I also wish to thank my colleagues at Imperial College during this time: John Benbow and David Willis of the Centre for Computing and Automation; Doug Barnes, Ed Davies, Don Monro, Mike Reeve, Chris Sowden and Ian Stinson of the Department of Computing; and Barry Belcher of the Department of Aeronautics for their specific contributions to this work and to my sanity.

I would like to record my gratitude to the Wolfson Foundation for the generous grant which established the Wolfson Microprocessor Research Unit at Imperial College. Some of the work described in this thesis was greatly facilitated by the equipment available in the Unit, and I doubt that it would have been undertaken at all if the Wolfson Unit had not existed.

I would like to thank Professor Bruce Sayers for his supervision, support and for giving me the incentive to submit this thesis.

Finally I would like to thank the External Examiners for their valuable suggestions for improvements which have been incorporated into this thesis. CONTENTS Abstract 1 Acknowledgements 2 Contents 3

1.0 INTRODUCTION 8 1.1 Locality Of A Network 9 1.2 Shared Access To A Network 10 1.3 Geographical Coverage Of A Network 10 1.4 Network Topology 12 1.5 Network Data Rates 13 1.6 Allocation Of Network Bandwidth 14 1.7 Operation Of Network Algorithms 15 1.8 Multiplexing as a Network Mechanism 19 1.9 Network Servers and Functions 20 1.10 Operating System Support, History, Software & Protocols 21 1.11 ISO Protocols 23 1.12 Interconnection of Networks 25 1.13 Reliability and Fault-Tolerant Aspects of Networks 26 1.14 Security Aspects of Networks 27 1.15 Cabling for Networks 28

2.0 EFFICIENCY OF NETWORKS 30 2.1 Efficiency of CSMA/CD (Ethernet) 32 2.2 Efficiency of Token-Passing Ring 35 2.3 Efficiency of Cambridge Ring 37 2.4 Efficiency of Mesh Networks 38 2.5 Efficiency of Multiplexer-based Solutions 39 2.6 Summary of Network Efficiencies 40

3.0 IMPROVING EFFICIENCY AND MINIMISING DELAY 41 3.1 Unit of Transfer 43 3.2 Synchronisation of Transfer 44 3.3 Minimising the Overhead of Routing 45 3.4 Minimising the Overhead of Control 45 3.5 Minimising Redundancy for Error Control 46 3.6 Summary of Methods for High Efficiency and Low Delay 47

4.0 THE PROPOSED SOLUTION: SYNCHRONET 48 4.1 Synchronet Topology and Data Flows 48 4.2 Synchronet Dynamic Bus Timing 51 4.3 Synchronet Connection Unit 51 4 4.4 Synchronet Frame Formats 52 4.5 Link Management 53 4.6 Lazy Scanning Mechanism 53 4.7 Connection Forcing Mechanism 55 4.8 A Uniform Synchronet 55 4.9 Implementing a Synchronet 56 4.10 Additional Features of Synchronet 57 4.11 Summary of the Proposed System 58

5.0 THE SONET EXPERIMENT AND HISTORY 59 5.1 Sonet Topology and Data Flows 60 5.2 Sonet Servers and Central Facilities 63 5.3 Interfaces to the Network Control Units 64 5.4 Flow Control and Break Conditions 65 5.5 Establishment and Control of Virtual Channels 67

6.0 SYNCHRONET AND SONET HARDWARE DESIGN 70 6.1 Media and Driver Design 70 6.2 The Sonet Line Driver 72 6.3 The Synchronet and Sonet Receiver Designs 73 6.4 Sonet Timing and Data-transfer Circuits 73 6.5 Sonet Network Server Unit's Circuits 76

7.0 THE SONET SYSTEM SOFTWARE 79 7.1 Network Server Unit Software 79 7.2 Network Connection Unit Software 82 7.3 Communication Between NSU and NCUs 85 7.4 Self Testing Facilities in Sonet 87 7.5 Network Protocol Generation 89

8.0 EVALUATION OF SYNCHRONET AND SONET 90 8.1 Efficiency of Synchronet 90 8.2 Efficiency of the Experimental Sonet 91 8.3 Cabling for Sonet 93 8.4 Interconnection with Synchronet and Sonet 94 8.5 Security Aspects of the Experimental Sonet 94 8.6 Fault Monitoring and Finding in Synchronet 95 8.7 Fault Monitoring and Finding in Sonet 96

9.0 APPLICATIONS TO EXISTING NETWORKS 98

10.0 CONCLUSIONS 100 5

11.0 CITED REFERENCES AND BIBLIOGRAPHY 102

APPENDIX A Symbols and Abbreviations 107 B Protocol Generation ROM Code 108 C Original NCU Primary Self Test 109 D NCU Store Arrangement 111 E NSU Store Arrangement 114 F Sample Test Procedure for an NTU 115 LIST OF FIGURES

Figure 1.1 Categorisation of Networks by Area Served. Figure 1.2 Categorisation of Networks by Topology. Figure 1.3 Total Network Data Rates Figure 1.4 User Interface Data Rates Figure 1.5 Hubnet topology Figure 1.6 Extending a Star Network using Multiplexers Figure 1.7 Comparison of KERMTT and NET File Transfer Regimes Figure 1.8 Layers in the ISO Standard Model Figure 1.9 Typical OSI Packet Showing Peer Headers for MAP Figure 1.10 Repair of Ring Using Duplicate Path

Figure 2.1 Ethernet Frame Format Figure 2.2 Ethernet Efficiency (for 2, 5 and 64 competing stations) Figure 2.3 Token-passing Ring Protocol Efficiency Figure 2.4 Token-passing Ring Frame Format Figure 2.5 Token-passing Ring Efficiency (for 2,5 and 64 competing stations) Figure 2.6 Cambridge Ring Frame Format Figure 2.7 Typical Mesh Frame Format (eg X-25) Figure 2.8 Summary of Typical Efficiencies Showing Area of Interest

Figure 3.1 Addressing Mechanisms for Networks Figure 3.2 Network Characteristics by Unit of Transfer Figure 3.3 Path Access Methods

Figure 4.1 Synchronet Unidirectional Bus Figure 4.2 Synchronet Data Flows Figure 4.3 Synchronet Data Transmission in Slots Figure 4.4 Synchronet Frame Format Figure 4.5 Partial Scan - Action Table Figure 4.6 Sample Commands for Uniform Synchronet Figure 4.7 Synchronet Station Hardware

Plate 5.1 Sonet NCU and NTU and Main Cable Plate 5.2 Sonet Network Connection Unit Construction Figure 5.1 Overall Design of Sonet Figure 5.2 Sonet Data Flows Figure 5.3 Sonet Frame Format Figure 5.4 NTU to NCU Connection Figure 5.5 Sonet RS232c opt D Interface Figure 5.6 Modes for NCU/Device Communication Figure 5.7 Protocols for NCU/Device Communication Figure 5.8 Sonet NCU Use of the RS232c Interface Signals Figure 5.9 Sonet NCU Block Schematic

Figure 6.1 Overlapping of Transmissions in a Synchronet Figure 6.2 Synchronet Driver Design Figure 6.3 Sonet Driver and High-protection Additional Circuits Figure 6.4 Synchronet and Sonet Receiver Circuits Figure 6.5 Sonet Protocol and Timing Circuits Figure 6.6 Phase Encoding and Master Bit Timing Circuits Figure 6.7 Receive Selection and Phase Decoding Circuit Figure 6.8 Frame Limitation and Bit-timing Circuit Figure 6.9 Transmission and Reception Waveforms Figure 6.10 ROM Circuit for Timing Generation Figure 6.11 Battery Backup for RAMs

Figure 7.1 NSU Software Structure Figure 7.2 NCU Software Structure Figure 7.3 NCU/NSU Communication Protocol Figure 7.4 Coded Message Expansion Code

Figure 8.1 Synchronet Frame Format and Timing Figure 8.2 Sonet Frame Format and Timing Figure 8.3 Summary of Efficiencies with Synchronet and Sonet Figure 8.4 Comparison of Delay and Throughput with Offered Load Figure 8.5 Partial Repair of Loopback Bus Figure 8.6 Repair of Loopback Bus into Simple Bus Figure 8.7 Error Distribution Caused by Cable Break Figure 8.8 Error Distribution Caused by Low Impedance Driver

Figure 9.1 Switching between CSMA/CD and Pre-Allocation

Figure A.l Primary Self Test Syndromes. Figure D.l NCU Store Map. 8 1.0 INTRODUCTION

This thesis covers significant aspects of hardware, software, protocol and operational factors in exploring the problem of interconnecting character-oriented and character-by-character devices using local area networks. It proposes effective and efficient solutions which were tested during a full-scale service experiment and the results, which demonstrate very cost-effective and reliable operation, are presented. Following an introduction to the historical background and more general problems of interconnecting this class of device, an extension to standard analyses of efficiency and delay for a variety of networks is used to delineate the problem area.

An original analysis of the potential mechanisms which can be used to give high efficiency and low delay is then derived. It is shown how overheads can be reduced by a novel combination of electrical, topological, protocol and operational factors. This leads to the proposal of an entirely new structure for interconnection, the unidirectional bus, and protocols which allow optimal operation.

The unidirectional, baseband bus is shown to have a unique combination of highly desirable properties which are not available together in any ring, mesh or simple-bus structure. These properties are: simple cabling requirements, the ability to tap devices onto the network without media breaks, the ability to densely pack data on the media, the ability to transfer data with low delay, the ability to locate faults in normal operation and the ability to reconfigure in the event of partial failure. Such a bus structure could be used for large block data transfers, with reasonable efficiencies, but that is not the purpose of this thesis, and is not pursued. The protocols which are proposed are specifically aimed at small-data transfer and it is shown how, by using contention-free access, they approach optimal efficiency while maintaining low delay and deterministic behaviour. The logical and electrical characteristics of the new bus structure (Synchronet) and its access mechanisms are presented and their performance demonstrated.

A full-scale experiment (Sonet), set up to demonstrate the overall viability of the proposals, is discussed with its hardware and software aspects. Specific modifications for a Synchronet to operate at extremes of speed and efficiency are also shown. The experimental results are described including the novel fault detection and fault tolerant aspects of the system.

The specific implementations discussed naturally have a short lifetime as improvements in technology in this area are so rapid, Sonet forming the major network in DoC at ICST for four years before being replaced. The general principles remain valid, however, as with all scalable technologies, improvements in technology are directly mirrored in improvements in the overall network. The use of the new technology and protocols proposed here provides for efficient small to medium sized, character-oriented networks. It is possible to use the proposals of this thesis with international standard networks to improve worst-case performance and this is described. In addition, by being used as a distributed multiplexer with simple, single-path wiring, the Synchronet approach remains the most cost-effective and convenient way of connecting dispersed, character- oriented devices to larger scale networks for the foreseeable future. 9 The majority of work carried out on local area networks has been for the transfer of large blocks of data between interconnected mini- and mainframe computers, which were dominant some years ago. A large majority of available devices currently operate in a character-oriented manner and it is expected that a majority will continue to do so for some considerable time. These devices transfer only small amounts of data at a time, but require very low delay and latency. The block-organised local area networks have traditionally operated with very low efficiencies when used to connect character-oriented devices and in some cases this can lead to catastrophic failure. This chapter provides a categorisation of networks and an explanation of the key factors in local area network design for character-by character transfer. This differs from other categorisations [Kas79,BrH83] by concentrating on factors which give convenience in implementation, installation and operation.

1.1 Locality Of A Network

A Communications Network is a set of computers and other devices interconnected by circuits to form an overall structure which permits data and transactions to be interchanged. Networks can be analysed and compared in a wide variety of ways. The comparisons are usually chosen to show the advantages of a particular technological solution, an approach normally fails to provide networks that meet what should be their primary requirement which is to satisfy their users.

The main purpose of any network is to make a facility or resource which is remote appear to be local, transparently. The success of any network can be judged by how well it achieves this purpose for the range of facilities it is intended to interconnect. A first categorisation into different levels of network can be identified by considering w hat they are meant to make appear to be local:

processor (e.g. coprocesso) store (mainstore) backing store Machines archival store (filestore) peripheral (e.g. printer) } terminal (possibly graphis) Users user (e.g. telephone) >

These are ordered by the data rate required to satisfy the requirement to appear local, with the highest first There is a huge difference in these rates from more than 100 million bits per second per processor [Fie85] to a few thousand bits per second per terminal.

It is not intuitively obvious that a single type of network will be suitable for all of these purposes. The first five devices may be categorised as m achines, and networks to interconnect them would be expected to run autonomously, unattended and with no consequence if the communication protocols were extremely complex and encoded for efficiency rendering them unintelligible to human users.

For connecting terminals and users, however, the network must appear simple to use, the protocols must be obvious and easily understood and the performance characteristics must be compatible with 10 human interaction input and output Considering that there are currently many times more network connections for terminals or users than there are for all other devices [Cha86,Egg86] it is surprising how much research has been done on machine interconnection networks and how little on networks for people. Work on networks has been going on for a long time [Bar64] but as far as can be ascertained this is the first thesis to propose solutions to provide the correct characteristics for user interaction: convenience, transparency, and low delay and latency.

1.2 Shared Access To A Network

The second main purpose of a network is to provide shared access to a facility or service. The shared access is justified on grounds of economy or access. Sharing is arranged either because it is cheaper or else because it provides some benefit simply because of the shared access, as for example with databases. Shared access can be provided either by a multiuser computer or by a network and the successful provision of shared access will also be considered in this thesis. For stations which provide a common shared service there is a choice of the style of service provided:

• Logged on session (multiple transactions) • Single coarse-grained transaction • Single fine-grained transaction

These exhibit different performance characteristics to an individual user and to a population of users. These differences make each style suitable for some operations but very unsuited to others. Different levels of security protection and frequency of security checks are required for each service style. Different delays will be met by users of each style of service. For a logged-on session a communications port is blocked for the entire duration of the session. The user interface is simplified, however, as security protection is provided by a single check which gives session locking usually by a 'logon' password. For file transfer then a single coarse-grain transaction is preferable. It is protected at the file lock level and the communications port is released immediately upon completion of the whole file transfer. A similar level of protection would be applicable for remote access to a centralised database. Each database transaction, consisting of a number of transfers, would require locking. For shared file access, such as to a distributed database, then no larger transaction than a record will be acceptable. Protection must be provided on every record and access to records (and use of a communications port) can be interleaved at will.

1.3 Geographical Coverage Of A Network

A third simple categorisation of networks is made by the area they serve. This is useful because the first difficulty encountered in linking multiple workstations to form a network is actually reaching all the necessary points. If a public highway has to be crossed and no private tunnel exists then the monopoly power of the PTTs (Postes, Telephonique et Telegraphique) is met. In the United 11 Kingdom this means that either British Telecom or Mercury will have to supply the necessary link(s). This may be taken as a rough division between wide-area networks over which the user can exercise little control, and local-area networks (LANs) over which he can exercise control. To complete the categorisation metropolitan-area networks, which can provide higher performance than wide-area networks but with the same basic constraints, and small-area networks, which may extend no further than a single room, car or hi-fi set should be included. This is shown in figure 1.1.

Figure 1.1 Categorisation of Networks by Area Served.

Many techniques and behaviours will be categorised in this thesis as wide-area even when used, or suggested for use, for implementing local area networks. The reasons for this are now summarised. Once a wide area is covered the distances are long and so transmission delays are necessarily higher. The investment made in communication paths is high due to their length and inaccessibility. Strenuous efforts are therefore made to improve utilisation and especially chargeable use of the communications facilities to recoup this high investment [Hem79,Coo85,Jos86]. The problems of this investment in wide-area services are exacerbated by the essential requirements for high reliability and easy maintenance of installed paths to provide high availability of the chargeable services. As many of the paths have difficult physical access for repair, and as there are many switching exchanges (more than 4000 in the United Kingdom PSTN [NCC82]) redundancy is essential to provide high availability of service.

Thus wide-area operation is characterised by m esh interconnection of switching nodes, store and forward (delayed) data transfer, and path sharing whenever possible. Wide area networks thus take a particular set of options which are different to those which could be taken for local networks. They often use standard protocols such as X25 as these are supported by the Postes Telegraphique et Telephonique (PTT's). They choose the options which provide highest line utilisation whilst local networks can choose to disregard utilisation if this provides a better service for the user. The former set of choices will be termed wide-area throughout this thesis. Local networks, which are the subject of this thesis, may cover a single floor, a building, extend for many kilometers over a large plant, or even extend for many tens of kilometers throughout the roadways of a mine. 12 1.4 Network Topology

A fourth categorisation of networks is by their topology or cable layout. Many possible topologies have been proposed or used to interconnect network stations [BrH83] but there are only two generic types, all networks having either mesh topology or bus topology

All mesh topologies are characterised by stations which break the communications media to insert nodes or switching stations. They must operate on a store and forward basis even if the data is stored for only a single bit-time.

All bus topologies are characterised by basically continuous communications media with nodes tapped onto it They operate in a broadcast mode [Abr70] where data is received by many (or all) stations and then switching is arranged by applying selectivity in the receivers.

The topology is not to be confused with the protocols as some mesh topologies operate in a broadcast mode in addition to the required store and forward mode, and so have receivers which are also partially selective. Similarly, a bus could have store and forward operation superimposed upon its broadcast mechanism. It is easy to see how other common topologies fall into these generic types as shown in figure 1.2. A star or centrally switched network is a mesh with only one switching station [ScW74]. A ring is a mesh with only one path connecting the nodes [Pie72]. A tree with non-switching repeaters is a collection of interconnected busses operated as a single bus. A hub-based network has been devised to be particularly suitable for use with fibre optic cables [Lee83]. This is tree structured but with switching nodes at the hub points. Even though these hub switches are very simple they must be inserted in breaks in the media and so a hub comes into the mesh category. A further topology for fibre optic cables is the star bus. This is a physical star layout but is operated with essentially unbroken media. The fundamental operation for various topologies is discussed further on.

MESH BUS

ring star h u b

Figure 1.2 Categorisation of Networks by Topology. 13 1.5 Network Data Rates

The total network data rate is the most frequently quoted factor to (supposedly) convey the quality of the the technology used It is, of course, highly confusing as it does not convey any idea of the actual usable data rates available to users, nor of the amount of bandwidth lost in protocol overheads. However, figure 1.3 is useful to determine approximately how many users a given network could support at specified user data rates. Examples of commercially available networks are shown for comparison with those introduced in this thesis.

.sbO Q bO 1 § pO w •E 2 CO o o a> a 8I •9a u 6 J§ § n o £2 o3 "S § 0 & < c/a u u w

Current WAN Low-speed LAN | Medium-speed LAN \ Future WAN Future MAN

0 bps 50k 500k 5M 50M |______|______|______|______|

Figure 1.3 Total Network Data Rates

The total data rate is of little interest to an individual user. The rate s/he sees at the device interface is far more important. This rate has a direct impact on the performance of the microcomputer and the complexity of the interface which will link the microcomputer to the network. Of course the total rate is of some interest as, taken with the user's interface rate, it determines the number of simultaneous transfers which may proceed without degradation. The data rate which a micro­ computer may accept is dependant on the acceptable processor load for the network operations. For a single user machine it may be permissible to allocate the entire machine to servicing transfers. This is not possible with a shared system, nor if the user wishes to continue word processing, for example, while sending a file over the network for printing as a background task.

cto to .5 c* .5 w> & A lm ost all Networks .*9 CL, o have this as one way c •sE M 8 U 3 O ' of connecting CL, U H I

RS232 interface Cache Buffer ij Big Cache +DSA| :| Interrupt per Character Interrupt per Line i Interrupt per Block |

0 bps 50k 500k 5M 1 ______I______I______I

Figure 1.4 User Interface Data Rates 14 The availability of custom designed interfaces for microcomputers to connect to networks used to be very limited [Mal81]. The situation can only improve as more integrated circuit implementations of the IEEE standards for Ethernet (CSMA/CD), token passing bus and ring structures [IEEE83a-e, Rao86] appear. Once the cost reduces sufficiently it may become common practice for machines to have a communication connection provided by a network interface. To fill the gap before this happens many networks, for example Gandalf [Gan85a], Sonet [Cri85], Multilink [Gra84] and PABX (telephone exchange) switches [Kas79,JuN83,Coo85], have provided the ubiquitous RS232 interface. It is limited in speed of operation to only approximately 50kbps, but is very cheap and almost universally supported by mainframes and microcomputers alike.

1.6 Allocation Of Network Bandwidth

It is perhaps a trifle obvious to say that it is easy to throw away bandwidth and performance, but that is just what many currently popular network systems do when run in very obvious ways. Some proportion of the available bandwidth of a network is taken up by arranging for the rest to be shared out between the various users. A further loss of bandwidth is caused by the sw itching function which allows the selection of end users to connect Further losses may also be caused by control and maintenance functions and by the addition of redundancy to enhance reliability. By using a block-oriented network for individual character transfer a further overhead is introduced.

If a network operates solely in a concentrating, demand-based manner then it must, under heavy loading conditions, ultimately fail to deliver some user's data. If bandwidth is not utilised, and could (or would) not otherwise be used to improve performance for users then it can not be considered to be wasted. In fact, if a system is not to fail under heavy loading conditions then it must act in a multiplexing rather than a concentrating manner and must have sufficient w asted bandwidth. That is, the sum of the individual, worst case channel requirements must not exceed the bandwidth of the network available for data transfer. It should be noted that if the allocation algorithm is not fair then a local, rather than a global, failure would occur. Adequate bandwidth could exist in the network as a whole but only be available in the wrong place! Complex routing algorithms are employed in wide-area style networks to attempt to overcome this problem. There are a wide variety of mechanisms for permitting access to, and hence utilisation of, available network bandwith and it is instructive to compare them. Two primary mechanisms exist for subdividing the bandwidth:

Frequency division or Time division.

This division also gives the spit into Broad-band and Base-band networks [HoM82]. Broadband networks [Dat81] use frequency division and a modulation technique to move originating data into the chosen frequency bands. This permits a wider use of the frequency spectrum at the expense of the extra modulation and demodulation stages. It also permits the simultaneous mixing of signals of disparate type [Hal87], for example data and television video, on the same paths. Baseband networks include zero frequency in their used spectrum and allocate by time division. This means 15 that at any instant only one signal can occupy any particular section of connection media in the network and so mixing of disparate signal types is not necessarily possible. With a deterministic time-slicing mechanism adequate bandwidth may be provided to permit sharing, but if a random access mechanism is used then sharing is not normally possible. Broadband networks have higher throughput over longer distances and provide better noise immunity because the modulation process moves the data to less noise susceptible regions of the spectrum.

1.7 Operation Of Network Algorithms

There are a wide variety of algorithms for controlling the access of data onto the network media. These range from no algorithm at all as in the case of a simple circuit-switched star network or PABX, to complex token-passing arrangements for bus-based networks. A simple categorisation is by the degree of random ness of allocation the algorithm exhibits from fully random to fully deterministic.

A fully random algorithm exists in Carrier Sense Multiple Access with Collision Detection (CSMA/CD) as used in the Ethernet [Met76,Dig80,Cra80,IEEE83c]. This relies for its fair allocation of network bandwidth on the symmetrical nature of its media and the randomness of the algorithm. A station wishing to transmit first listens to the single bus media and only attempts to transmit if the bus is not already in use. A minimum blank gap between consecutive transmissions from stations is mandatory to ensure other stations have a chance to observe an idle network. A station wishing to transmit, observing that the network is idle, starts sending its frame. The frame layout is shown in figure 2.1. If no other station commenced sending at the same time then the packet is continued and is seen by all receivers. The chosen receiver operates selective reception, seeing its address in the frame, and copies the data into its buffers. No acknowledgement is mandatory. If two or more stations wish to transmit at a given time then they will (all) observe the idle bus and start transmitting. This will cause corruption of data, called a collision, and is detected by all transmitting stations, also listening to the bus to check to see if they hear what they send.

A minimum packet length is essential in such systems as, if a collision occurs, all stations on the bus must see it as a collision. If too small a packet were allowed then two stations at the far extremities of a network could send, but due to the delay of the network media would not see the collision. Receiving stations located at the centre of the network would, however, see a collision. If collision is detected then the colliding stations all back off for a random time and retry. The random period is determined by their station number, the load on the network and a genuine random number to ensure they should not collide again. Of course, if the load is high, the stations may collide with other stations when they retry. Time is wasted by the collision mechanism and to keep it to a minimum, when a collision is detected stations abort transmission and transmit a jam m ing signal to inform other stations immediately. As is shown in chapter 2, this algorithm works well with lightly loaded networks (<15%), has shortcomings under medium loads, and fails completely under heavy loads. A bus based network may also have its access controlled by a deterministic, yet not necessarily fair, 1 6 algorithm based upon token-passing [IEEE83d]. This is currently only applied to broadband bus arrangements but there is no fundamental reason why it is not found on baseband busses. Access is granted to the single station which holds the token. If a station does not wish to transmit, or on completion of the transmission of its frame, then it releases the token and passes it on to the next device. The problem is 'which is the next device'? Each station must therefore hold the address of its successor device in the token-passing chain. This chain is the key to the whole operation of a token-passing bus. If it is completely fair then it can be initially established in arbitrary sequence, and ascending order of station address will be quite fair. If it is to be biased in favour of some stations by giving them more than one turn per 'cycle' then it would initially be established by the network manager (a person). Problems occur if any station is switched off or fails. The previous station in the chain will try to pass the token to the failed station and an acknowledgement will not be received. After a failed retry or two the station holding the token will have to try to repair the chain. It sends a request onto the bus asking: 'which station had as its previous station in the chain'?

If all stations hold the address of the station before them as well as the station after them then one will reply and the repair can be affected. If the failure is more serious then a complete rebuild of the chain may be necessary and this has to be done by making requests such as: 'any stations having addresses in the range please respond?

Following single responses that station can be inserted into the chain. If a garbled response is received then the range must be reduced. This is a most time consuming process and the efficiency of the token-passing bus arrangement relies on the high reliability of the hardware to ensure that the chain-rebuilding is a rare occurrence.

The fundamental token-passing algorithm can be considerably simplified if the topology is restricted to give a single connecting path direction between all stations. The ring topology provides just this restriction and hence token-passing ring standards have been promulgated [Dix83,IEEE83e]. Nodes wishing to transmit on token-passing rings wait until they have control of the token. This is passed round stations on the ring in turns fixed by the physical layout The format of the token and data packets is shown in figure 2.4. Once a station has control of the token it transmits its packet and waits for the acknowledgement of correct reception in the frame status. It then passes the token on around the ring to the next station. Should a station fail then hardware mechanisms are required to repair the ring. It is possible to lose, or even duplicate, the token due to a station failure and so a monitor station is essential for this algorithm to operate reliably.

There are other algorithms for operating rings and two are considered serious alternatives to token passing. These are the em pty-slot and register insertion algorithms. Register insertion [Gra84] operates by a station in the ring simply reading data in from the previous station and writing it out to the successor station. If the station wishes to transmit then it lengthens the ring by inserting a register between its read and write circuits. This register is of the length of the packet to be transmitted. Consequently the delay (in bits) of the complete network is now adequate to 17 accommodate the new packet in addition to the existing traffic. Once the packet has traversed the ring, picking up a changed acknowledgement bit on the way it again occupies the transmitter's register and is switched out of the ring. This algorithm can be operated without requiring a monitor station, though commonly one would be included for maintenance purposes.

The classic example of the empty-slot ring is the Cambridge ring [Wil75,Lar82,Sha82,Hop86] losely based on an earlier design by Pierce [Pie72]. A Cambridge ring continuously circulates an integer number of minipackets with the format shown in figure 2.6. A station wishing to transmit waits until it sees a minipacket with its full!em pty bit set to empty. The station changes this bit to full and inserts its data address and control bits into the empty frame as it passes through the station. The minipacket is passed on through intervening stations until it reaches its destination. There the appropriate acknowledgement bits are set and the minipacket continues around the ring back to the transmitting station. The transmitting station knows how many minipackets there are on the ring and so can set the full/empty bit of its own minipacket back to empty, prior to checking the rest of its contents. For this algorithm to work stations must have one (exactly) bit of delay so that they can both see and change the full/empty bit and any acknowledgement bits. A monitor station is essential to ensure an integer number of minipackets will fit on the ring and to ensure that a failed station does not leave a full minipacket endlessly circulating around the ring. The monitor passed bit is used for this purpose.

Mesh based networks [CCI80,Fol82,Bar83] operate by accepting the packets from the user devices and then checking their destination. Based on this a routing algorithm determines to which node of the mesh the packet should be sent next There the entire packet has to be read in and checked before it can be further routed towards its destination. Though such networks can easily cope with network line or switching-node failure, by simply chosing an alternative route, they must impose high delays by comparison with the other algorithms. To reduce the delay through the network short packet sizes should be chosen, so that they can be read in, checked, routed and forwarded quickly and the next packet got underway on the previous link. Most mesh-based networks operate on wide-area principles and do not use very small packets for bulk data transfer. The storage requirements of the network are also considerably higher than for the other types. The upshot of this is that the mesh networks have a very high latency and high-level protocols which rely on acknowledgement, or attempt any form of real-time interaction will perform poorly. A major claimed benefit of mesh networks is their high line-utilisation, but it has been amply demonstrated [Abr78] that local congestion will commonly occur, increasing delays, and reducing line utilisation in other areas. For local cable runs high line utilisation is not an essential requirement

The algorithm for operating the Hubnet mentioned earlier, and shown in figure 1.5, minimises delay and wasted bandwidth whilst providing deterministic behaviour [Lee83]. The network is formed of hubs each having a double fibre-optic connection to a number of stations and a similar (station-like) connection to the next hub. A station wishing to transmit commences sending its packet to the 'sub-hub' to which it is connected. Each sub hub arbitrarily selects one of the incoming bit streams to switch to its output. The hub has no processing power to examine the content of the data and has 18 a delay of only nine bit-times (20ns) in the existing system. If an idle hub starts receiving traffic within its discrimination time (1/4 ns) then it arbitrarily selects the lower numbered input The sub-hub next in the path again selects one of the offered packet bit-streams to forward, and so on until the central hub is reached. Here the selected packet passes through the 'golden link' and back to the other half of the central hub. The return half of each hub simply broadcasts any data it sees on its input to all of its outputs.

Thus all stations see the chosen packet, and the one station which originated it continues to send it to completion. All the other stations which were attempting to transmit stop, wait for a fixed retry time, and then recommence sending their packet The retry time can be set to optimise performance and to give stations a uniform chance of access. To ensure deterministic behaviour however a single bit is used and on packets from each station it is alternated. Thus the first packet has a T , the second a 'O’, the third a T and so on. The hubs can then be made to select one type (say T ) until all of that type has been transmitted successfully and then switch to sending the other type ('0'). In fact only the central hub need perform this selection for the system to work, but if built into all hubs then the net can partition on failure into separate functioning, but disconnected, networks.

The final major type of network is the star, epitomised by private automated branch exchange telephone systems (PABX), used to transfer data [Hem79,JuN83,Coo85,Egg86] and specific star bit switches such as the Gandalf [Gan85a,b]. These sample the data lines connected to them at sub-bit times and provide a circuit-switching function to transmit these samples onto a selected output line. A major advantage of the PABX approach is that the existing telephone wiring in a building is used. The data rates which can be supported are much lower than for other network types, but the low delay and ease of operation are particularly attractive for interactive users.

However, for low cost the Ultimate Low Cost Network (ULCnet) [Cle81] is a do-it-yourself system at less than five pounds per connection. It operates satisfactorily for very small numbers of nodes only. It has a bus topology which simply connects the RS232 ports of the devices to be networked 19 in parallel. The common wires are run at RS232 speeds (up to 19.2K baud) and so only one device can operate at the full user data rate at any instant The addition of a common interrupt wire (from the RS232 'request to send' control signal for example) can be used to avoid any collisions. A slightly more complex but, at 25K - 100K bps, a faster version of this idea is implemented as the Infinet [Eid82].

1.8 Multiplexing as a Network Mechanism

Simple networks have been constructed using multiplexers, and multiplexers are used to simplify parts of larger networks. A pure multiplexer combines a number of low data-rate streams into a single higher data-rate stream where the maximum sum of the individual rates is never greater than the capacity of the higher rate channel. Multiplexers provide an efficient method of connecting multiple devices because there is no need for flow control or error control to operate on the high rate shared channel. Firstly no flow control is required because there is guaranteed to be adequate time on the higher rate channel to carry the sum of the individual, worst-case data rates. Secondly no error control is practical as should an error be detected on an individual sub-channel adequate time would not exist to resend the entire frame for all channels. Also those channels on which an error had not been detected would not know that a retransmission had been requested. In both cases any control has to be organised on a per channel basis. This implies that although multiplexers themselves are very useful and efficient devices, a higher level of protocol will always be needed operated on an end-to-end basis by each individual channel.

Figure 1.6 Extending a Star Network using Multiplexers

Some synchronisation information is always required in each combined data frame. This is usually in the form of added-digit framing, the most common form being an alternating-bit protocol. A bit is added to each frame and by alternation provides a lazy clock signal which can be used to maintain synchronisation between source and destination. Multiplexing is possible in bits, bytes or in larger units, though as these are outside the scope of this thesis they will not be considered further. Bit multiplexers can only operate synchronously, wheras byte multiplexers may operate synchronously or on demand when they are ususally termed Statistical multiplexers. In this latter case a single bit is added to each byte-part of the major frame to indicate whether a byte for that channel is present or absent The performance of multiplexers is considered in chapter two. 20 The major problem with multiplexers, however, is that they require the devices which are to be connected to be co-located. On demultiplexing at the end of the combined channel the same restriction is apparent It is of course quite practical to consider using multiplexers as part of another network scheme. The most obvious example would be to reduce the complexity of wiring needed for a pure star network. By interposing multiplexers at convenient points fewer, more costly cables can be substituted for a larger number of cheap ones. This is shown in figure 1.6 and is the arrangement we installed in DoC at IC to replace the experimental Sonet (chapter five). A central star switch connects to many devices directly but also to a number of multiplexing 'shelves' using fibre-optic cables. Though this is a more complex solution than those proposed here, particularly distributed multiplexing, it remains a competitive technology with which to compare this work.

1.9 Network Servers and Functions

A network provides two types of function which are similar to those available from a telephone. It either provides a communication channel to another arbitrary point or access to one of a (small) number of centralised, or shared, services. Typical examples from the telephone system are videotex, the speaking clock and sports results. A digital network will provide a wide range of services. These could, in theory, each be fully distributed amongst many nodes but in practice this has not happened.

Consider, for example, the problem of connecting to a node by its logical name. A request to translate the logical name 'Martin' to node number '42', say, is needed. If the name is in fact a generic name attached to a set of node numbers then it is necessary to determine which is the first free node of the set and translate the name accordingly. If all of the set of nodes were already busy then this call would be queued until one of them became free. The problem, if distributed, has reached epidemic proportions. The initial call for name translation now requires a separate queue and entry in each of the generic named set of nodes. The first to become free must respond and remove the entry from all the others. Of course with the latency in a network two may attempt this simultaneously. As a result of complexities like this all current networks use servers to centralise shared functions, simplify the provision of shared resources and ease problems of consistency. Servers will commonly need to be provided for: • Logical name translation (directory) • Status and help information • Initialisation of node parameters • Gateway access to other networks • Network management and maintenance.

In addition value-added service functions attached to the network are also provided in a similar fashion: • E-Mail, memos • Processing • Printing and other output • Filestore, both database and archive 21 1.10 Operating System Support, History, Software and Protocols

Simply connecting a computer to a network via suitable hardware affords no solution to the general problem of communication. The user must somehow be able to link to (or between) application programs. To establish an editing session remotely, for example, may need four communication 'channels'. The file being edited may be located remotely so channels to read and write to it would be necessary. The user may be located remotely and so his input and output must be handled. Thus a need to support network communication directly from programs has been identified and this, in turn, presupposes some integration between the communications hardware and the machine's operating system.

One of the early experiments in distributed, multi-user operation was conducted at Imperial College in the late 1960's. An IBM 7094, running the IBSYS version 13 batch operating system [IBM63] was connected by a high speed communications link [Cri70] to a DEC PDP-9 computer [PDP68]. The IBSYS operating system and the Purdue University Fast FORTRAN Translator (PUFFT) [Ros65] were modified to permit the remote machine to access the main 1301 disc file store to extract, modify and store program and data records. The PDP-9 was run using its standard operating system with the addition of the handler for the two million bit per second communications link, a terminal multiplexer and its handler [Cri72], and a multi-user interactive editor. Initially simple ASR33 teletypes were used, but early visual display terminals were also tested successfully. Files were held on the IBM's 1301 disc unit and records were passed up to the PDP-9 for editing. Once an interactive editing session had been completed in this fashion, the job created was inserted at the head of the PUFFT batch queue. As batch job time limits were quite small during the daytime interactive sessions the program(s) would be quickly run and results returned to the 1301 disc unit The user could monitor the progress of his job via the data link and, once the job had been completed, could interrogate the compilation output and results. This gave the user the appearance of a very powerful, fully interactive system with the benefits of distributed operation. Files could be freely transferred between the two systems and the restrictions of the mainframe, batch operating environment were overcome.

This project did, however, require extensive modification and additions to the operating systems. Had it been used for providing service for a longer period then the maintenance of the systems, as they were updated with new releases being introduced, would have been a considerable undertaking.

With the diversity of machines and operating systems which mushroomed in the 1970's a less demanding mechanism for communication and distributed operation was required. Two approaches have become apparent to permit easy support for networks. The first is to design protocols which can fit on top of any operating system. These must make the minimum possible assumptions about the transparency of the path from electrical signals to the protocol program. These are called application-supported networks. The second approach is to get international standard protocols agreed which all manufacturers will be forced to incorporate into their operating systems by the weight of user pressure. 22 To demonstrate that portable, application-supported protocols were a practical solution, while international standards for the integration of communications into operating systems were awaited, the CCDNET protocol was devised in collaboration with Ian Stinson [Sou78]. This protocol allowed file transfer, remote job submission, remote printing and messaging between computers ranging from the IBM370 [Dav78] and DEC PDP-15 [Bro78] to the Texas Instruments 960A [Sou78] and Motorola M6800 microprocessor [Sau79]. These implementations provided a fully secure transfer mechanism with sequence and individual packet checks. No implementation required any modification to the operating system other than the addition of a serial communications port driver where one did not exist. The protocol was later simplified to run over the Sonet network to provide file archiving, printing and remote job submission. This reduced 'NET protocol was implemented originally in Pascal with very small assembler routines for I/O [Csg83]. This student facility went into full service in January 1983. It provided a fully reliable continuous service and was extended to give staff support, running over one hundred nodes in January 1986. The system included the IBM4331, DEC PDP-11, Vax 11/750, Onyx and Corvus computers, twenty-five Northstar microcomputers, four MX 100 printers and links to other networks for student use. The IBM-PC implementation, added by the author for staff users, was programmed entirely in Pascal.

The NET protocol operates complete network functions on a single transaction basis, with its commands having the form:

I list,send|

A> NET | get,print| filename.xtn filename type discno USERID(options | er a s e run | V.____ J J V ------local name remote name if different

Other protocols for file transfer, based on the same principles, have gained popularity for use over wide area networks, for examples Xmodem [Cla83]. The linking of one such protocol with a popular terminal emulation (VT52 or VT100) gave rise to the Kermit system designed by Bill Catchings and Frank DaCruz in 1981 [Cat83,DaC84]. This is implemented on dozens of different micro, mini and mainframe computers. It has been fully supported by Columbia University, USA, and has become a de fa c to standard. It only provides a logged-on capability, however, and so is not very efficient in communications port usage for file transfer. A comparison of a file transfer under Kermit and NET is shown in figure 1.7 and highlights the difference between the logged-on and single transaction styles discussed earlier.

Excluding the password check, ten separate commands must be entered to perform the file transfer under the logged-on regime, wheras only one command is needed for a single transaction server, which demonstrates the convenience of such servers for user oriented networks. 23 Kermit style

A> kermit i Invoke local kermit: MSkermit: c 1 S91R Sonet (DoC) ? c ivax >1 Connect over local area network S21I Connected to 15 DoC Ivax login: UMACH33 X Logon plus security check password: xxxxxxxx 1 $ kermit Invoke remote kermit CKermit vxx: server -l *1c i Instruct local kermit to transfer MSkermit: get filename xtn 1 xx kbytes xx packets Status messages shown on screen completed MSkermit: remote bye »!> Automatic remote logoff S22W Disconnected from 15 plus local termination MSkermit: quit >1 A>

NET style

A> net get filename.xtn UMACH33 <1 invoke NET NET..... enter password: xxxxxxxx X Security check Connecting to filestore S21I Connected to 56 Establishing Waiting for VM connection Status connection Connected to VM Messages receiving data ¥* ¥ * ¥ * ¥ * Blips to confirm transfer complete transferring data S22W Disconnected from 56 Disconnection A>

Figure 1.7 Comparison of KERMIT and NET File Transfer Regimes

1.11 ISO Protocols

By agreeing to international standard protocols, manufacturers are forced to incorporate them into their operating systems by the weight of user pressure. The International Standards Organisation (ISO) took on the task of achieving agreement on such protocols for wide area networks. It has been assisted by the Institution of Electrical and Electronic Engineers in determining standards for the local area network specific parts of these standards [eg IEEE83]. To achieve agreement on standards in such a complex area the task was split up by first agreeing on an overall seven-layer model shown in figure 1.8 [Zim80]. Subsequently agreement would be achieved for the individual standards for each layer and the interfaces between them [eg IS079].

There are problems attributable to this approach, however, the most serious of which is the resultant speed-efficiency of the protocols. Also the functions may not initially be correcdy split between layers, as turned out to be the case where duplication of function has occurred in the OSI protocols. Additional computation is required to force data into the patterns dictated by the interfaces between levels. The pipeline effect of layered protocols increases the latency of communication of any network using them. As a very rough guide, an order of magnitude loss in data rate is introduced by 24 each of the three stages: firstly, dividing the network out amongst users; secondly, interfacing the network to users devices; and finally, receiving data through the layered protocols.

Access point to communications system for 7 application applications programs Reformatting of data from application program 6 presentation into a form common to the network Name/address translations, access security 5 session synchronisation and management of data 4 transport Transparent virtual link between end-nodes

3 network Establishment of connections between end-nodes - routing 2 data link Reliable data transfer - media access and error checking 1 physical Encoding and decoding of electrical signals onto physical medium. Figure 1.8 Layers in the ISO Standard Model

The overhead of this standardised set of protocols can perhaps best be seen from a complete frame as shown in figure 1.9. Even with the null presentation layer the amount of computation required for this General Motors Manufacturing Applications Protocol version of ISO is apparent If run on a Digital Equipment VAX11/750 over one quarter of the processor's time can be taken up in handling the communication protocols.

C/3D Oa u 8P X> '& P S3 e P 8 £ co U G VO CO O 3 o CS C/3 oo ooo 3

USER DATA C £ o • C/5vH jp CO 14-^ 1 3 p p I I CO £ a T3 a & 8 & p 8.

Figure 1.9 Typical OSI Packet Showing Peer Headers for MAP

Such protocols will not be universally acceptable until high-speed, purpose-designed integrated circuits perform the functions rather than their being a load on the main processor. The foregoing arguments imply that, even with very high speed optical networks and interfaces, the losses attributable to the layered protocols will be so severe as to provide the actual limit on individual data rate and latency. The future for high speed networks would appear to be best served by the agreement of much simpler and lighter international standard protocols. 25 1.12 Interconnection of Networks

A single local area network is always limited in the distance it can cover, or by the number of stations that can be linked. Having found the benefits of local communication the desire to extend to the wide area will also need to be accommodated. Standardisation of the interlinking of networks is yet more complex than the standardisation of the networks themselves [Qua86], however, a number of units have been designed to perform these linking functions at a variety of levels of complexity.

A repeater is a device which operates at the Physical level (1) and amplifies or conditions signals received from one transmission media section and drives them onto a similar piece of transmission media. The repeater does not read or alter the address, data or control content of any frame it passes and it requires no 'logic' circuits. It is sometimes called a Level 1 (or a physical level) relay.

A bridge is a device which operates at the link level (2). It may modify the address and control information of the data link layers to permit the bridge to provide a link between two similar networks running the same protocols but which may have addressing conflicts. A frame of data is read in and the address and control information is decoded. Any changes are made and the frame is recoded with any check redundancy being added. The frame is then transmitted onto the second network. A bridge is sometimes called a level 2 (or a data link) relay.

A router is a device which operates at the network level (3). It receives input from a physical level interface and decodes it according to datalink (2) and network (3) level protocols. One can be used to link two networks which operate different level 1,2, or 3 protocols but employ the same standards for the transport level protocols (4) and above. A more common use, however, is to switch or route traffic around in a large local network or a wide area network. A directory table is used in the router to decide on the next link (there may be more than one possibility) on which to transmit a packet. The directory entries are periodically updated to represent the delays and costs associated with particular routes to allow avoidance of congested areas and prevent saturation. This mechanism is also used to establish virtual channels. Once established, routing is performed using abbreviated virtual channel tables indexed by the logical channel number. Once the network level parameters have been determined then the packet is recoded by the same or different level 3 protocols, is wrapped in similar or different forms of level 2 headers and trailers. It is then transmitted onto the new network or the next segment of a larger network. A router or level 3 relay is sometimes called an intermediate node or a switching node.

A gatew ay permits connection between two or more dissimilar networks and is operated at the transport level (4), or possibly at a level above this depending on the requirement. If the networks to be linked are entirely dissimilar then all layers of protocol of one network must be decoded to recover just the original data. This is then recoded according to all the protocols of the second network and transmitted onto it The gateway thus translates any to all seven layers of protocol and so is the most complex of the linking devices. 26 1.13 Reliability and Fault-Tolerant Aspects of Networks

There are a wide variety of possible fault mechanisms in networks and methods are needed to predict, detect or locate them before any recovery can be effective. It is commonly necessary to locate a fault before any repair can be attempted but it is not necessary to locate a fault to get over its effect if some loss of performance is allowable. Failures occur at many different levels within a network though most may be attributed to the physical nature of communication connections and interference on them. The lowest level fault is one which interferes with the communication media causing a fault of long duration. The probability of failure should be broadly similar for all cables of ruggedised construction. The capability of operation after a failure is higher for lower technology or lower speed networks, some being capable of partial operation even in the presence of short or open circuit bus failures. To protect the cabling, some form of separate ducting or strain relief is highly desirable, as will be apparent in discussion of the experiments carried out.

Detection of a break or short in a link of a m esh is quite easy if suitable protocols have been adopted. The absence of a received signal for a sufficient period prompts a request transmission to check the integrity of the link. In a multiply-connected mesh, packets can be rerouted but with noticeable loss of performance.

For a ring the problem is more difficult as there is only a single path between stations. A redundant ring must be included to allow self-repair reconfiguration as shown in figure 1.10. This renders the ring invulnerable to a failure at any single point, regardless of the severity of the fault The main reason for such redundancy in practice, however, is to allow new stations to be inserted without causing disruption to existing patterns of communication. Cambridge Rings and their derivatives commonly need to use their monitor stations to handle self-repair. For example, when a break is detected the station dow nstream of the break informs the monitor and switches its input to the auxiliary path. The monitor adjusts the ring length to an integer number of minipackets and informs the station upstream of the break to feed its output to the auxiliary path.

Figure 1.10 Repair of Ring Using Duplicate Path 27 A system of wiring which seems, at first sight, to have considerable advantages is the star-ring. It is in current use with the token-passing ring protocol to form the IBM token-passing ring local area network. The failure of any individual section of wiring or remote node can be detected at the central star-point The wiring and distribution clusters are so arranged that any failed section can be removed from the star ring to leave an operable system. The star ring however, like any other star organisation, is vulnerable to failures at its centre. No amount of redundancy can overcome this and so precautions must be taken to ensure the security of the centre.

Location of this level of fault is very difficult for all simple buses. The reason for this difficulty is the same as the reason for most of the benefits of a bus, namely the absence of a forced structure on the paths of the communication media. But it is this very lack of an imposed structure which makes it difficult to locate faults. A simple bus system may be able to to use the distribution of error rates at operational stations to provide the probabilities of location for bus-level faults while the network is live. However, a novel bus system which retains the flexibility for paths of a simple bus, yet imposes just enough structure to make bus-level fault location much easier, is possible. Both of these methods are demonstrated in chapter eight. $

1.14 Security Aspects of Networks

Networks, due to their widespread nature, are more susceptible to security problems than centralised computing facilities. The problems come in three main forms:

a) Eavesdropping, or the interception of messages which are intended to be secure, b) Disruption, of an individual message or to (sections of) an entire network, c) Authentication, or the acceptance of an intruder as a valid user.

In general any broadcast technology is insensitive to monitoring by tap connection. Even ring and tree structures are usually operated with protocols which do not differentiate between momentary failures from different causes and so amy fail to appreciate the significance of an intrusion. Thus eavesdropping and authentication are handled by encryption and signature techniques which are basically common to all computer systems. It should always be asumed that all networks are open to eavesdropping in varying degrees, so the only protection is a high level protocol which includes encryption. Similar factors apply to authentication.

A network can give no absolute guarantee of the source of a message. To lessen the risk of unauthorised access to the network it is possible to accept only connection requests from devices which have been nam ed in the currently active name server. Thus, though connecting a suitable spying device to the network would permit eavesdropping, it would not permit unauthorised access as a valid user. A secure terminal is needed for the name server to prevent an intruder gaining access by validating an invalid name. Deliberate or accidental disruption of a (section of) network can be curtailed by monitoring and self-repair circuits in the network. This is discussed later, but as much 28 work has been pursued elsewhere [Dav84] this thesis does not concentrate on security aspects. We now turn to consideration of the cabling itself and this can have a considerable bearing on the security of a network [Phi85].

1.15 Cabling for Networks

And so, to introduce the most serious problem of installing and maintaining a network: the cables linking the stations! The difficulty of installing cables, particularly if they are of high-performance, coaxial, or fibre-optic types is considerable [Cam86,Eos85]. The use of more advanced signalling techniques or the use of simplified but more efficient protocols permits cheaper and more convenient types of cable to be installed. Practically all modem buildings, and of course all old buildings, were built without any thought of the consequences of the rapid increase in data transmission requirements. The designs of the most modem buildings do concede that a greater provision is needed for data communication cables than for the usual power and telephone lines. There are three areas to consider. The primary routes are vertical, usually in risers. The secondary routes are horizontal and are usually housed in ducting of some kind. Where the two meet the interconnection should be housed in distribution areas, commonly called closets.

Standard ducting is too small for most network cables, and particularly is too small to accommodate bends around comers. For example, the minimum bend radius of standard IEEE802.3 cable is 6cm. For any cable an absolute minimum bend radius of twenty times the diameter is required. Housing transceiver units close to the cable is also a problem. Transceiver units for IEEE802.3 are connected either by PL259 UHF connectors in line with breaks in the cable, or by so-called vam pire taps which fit around the cable with sharp prongs connecting through the insulation to the conductors. Typical transceivers of either design are 15cm x 10cm x 5cm and so will not fit in standard ducting. One proposed solution which is quite common is the use of false ceilings. False ceilings are, however, often poorly constructed and are almost always inconvenient to remove and replace, thus making maintenance as well as installation difficult Large rooms have the additional problem of needing cable drops from the ceiling. Ducting around the periphery of a floor is only satisfactory if the building is less than about 15m deep. The test site for this research was the Huxley building at Imperial College and, completed in 1975, is a classic example of all these faults. The only really satisfactory solution for larger buildings is a flexible arrangement of false flooring.

Ensuring the adequacy of vertical risers, floor distribution rooms {closets) and the horizontal distribution system on each floor is essential if cabling is not to be the Achilles' heel of a network. Up to two percent of a buildings floor area can be required for risers and distribution. Problems with the reliability of cable installations and fault tolerance of specific topologies are discussed in more detail further on.

As well as the difficulty with the layout of the signal cables there is the problem of mains power. The number of socket outlets as well as their combined power rating will usually need to be 29 increased to support a large network. Two solutions are apparent to this problem. The first is to rely on distributed cabling (as for bus or ring topologies) and power the network connection units either from the network itself, or by daisy-chaining from the connected device. The second solution is to adopt a star layout and locate all network connection units and cable distribution racks in the floor distribution closets. The closets need to be larger to accommodate this, at least four square metres per closet being necessary. The advantage of the first solution is that installation only occurs when and where it is needed. The second solution allows safe, possibly secure housing of communication equipment in convenient racks with centralised power supply. The centralisation of wiring is both an advantage for ease of maintenance and a disadvantage as it causes a large increase in the total cable and more complexity at the closets. The route the cables take must be considered carefully to avoid routing near fluorescent luminaires, motors, power distribution cables or lightning conductors. This is easier to arrange with the first (distributed) solution.

The choice of cable: fibre-optic, high or low grade coaxial or high or low grade twisted pair, is usually determined by the choice of network [Abr82]. Obviously the cost varies depending on this choice, but so does the convenience and size of the installation. The basic unshielded twisted pair (UTP) such as is used for telephone and most star-based networks is capable of carrying over 2 Mbps as is demonstrated further on. Simple shielded twisted pair cable (STP) is capable of still higher data rates and has the added advantage of more rugged construction though this is offset somewhat by marginally more complex connections though using similar insulation-displacement techniques. Low grade (thin) co-axial cable, such as is used for cable television, is nearly as simple to install as STP, but the high grade (thick) co-axial cable required for IEEE 802.3 standard networks is much more costly and difficult to install. Even greater care is required with fibre-optic cable. In medium or large scale installations, of which the test site is fairly typical, cabling costs as much as 30% of the total hardware cost. Thus techniques which can simplify cabling or permit a cheaper cable to be used,such as those proposed here, are obviously desirable.

Following on the cost of initial installation, however, the cabling in a building must be regarded as an asset to be audited and maintained with a proper regard for the likely cost of subsequent operations. All installations should be designed to be flexible and capable of easy expansion.

To recap, the major attributes to consider in selecting a network are: capacity, coverage, configuration and cabling, convenience, security, reliability and, of course, cost. The following chapter will show the original motivation for this work by seeing how well standard networks can cope with character-at-a-time data transfer. 30 2.0 EFFICIENCY OF NETWORKS

As the thesis title includes the word efficiency it is necessary to know what this means in this context, and how it is measured. Many attempts have been made, over the years, to provide measures and comparisons of widely varying network proposals [e.g. Ros76, KuR78, Pf082, Sta86]. A fairly-chosen selection are derived or reproduced here. The comparison, particularly for the internationally standardised networks shows the reason for, and area of interest of, this thesis.

There are three absolute measures of efficiency for networks:

• the delay in transfer introduced by the network, • the throughput of data from end to end, or the 'bandwidth', and • the efficiency of use of the network technology, i.e. the cost.

The first metric is naturally the difference between the time of entry of a bit or a character to the network and the time of it being available for use after exit from the network, both measured on the same leading or trailing edges. The second metric can either be considered for an individual user or as a total summed over all users. The third metric is a ratio of the actual throughput to the maximum possible throughput of the network. It is a cost/performance measure, showing how well a given network architecture uses a chosen network technology, and the maturity and convenience of that technology. For many network designs these metrics are hard to measure and so are not known deterministically.

To understand some of the problems involved in measuring or estimating efficiency consider a simple visual display terminal transmitting data at bits per second to a computer. A frame of data of length Fd bits contains Ud user data bits. If the computer is d metres away and the velocity of propagation of the signal along the connecting medium is v metres per second then the the first transition will arrive after d/v seconds. The entire data frame will have been transmitted and read into the computer in: Fd d T, = (— + —) seconds td Rd v Note that writing out of the terminal and reading into the computer overlap to some extent as long as:

Unless the data rate Rd is increased then this is the minimum delay, and the throughput efficiency is: U, n = — ...the Throughput efficiency for the User's device

To give a feel for the magnitudes involved, consider the terminal sending a single character of data at 19.2 K bits per second isochronously. A frame of data of length 10 bits will contain 8 user data bits. If the computer is one kilometer away, and the velocity of propagation 0.7C, then the transfer takes 0.554 milliseconds. If frames are sent continuously then the throughput efficiency ^lt = 80%. As 31 this is the best this device and link could achieve any reduction in this performance must be due to network inefficiency. Introducing the simplest conceivable network means that, instead of the single transmission, data is read into the network, transmitted across the network and then retransmitted to the end user. No overlap is possible between writing out of the terminal and reading into the computer and so the delay is at least doubled. Many networks based on wide area style mesh connection increase the delay by as much as a thousand times.

The delay attributable to a network is caused by a combination of three factors:

• delay due to transmission Tn(j, • delays due to protocol overheads Tpr» and • delays in input and output of data at the end stations Tj0.

The basic delay due to transmission is determined from the data rate on the network (R„) and the frame size (Fn), and the distance between stations and the velocity of electromagnetic waves on the chosen network media. F d T = ( — + — ) seconds. nd v r v n n

The delay due to protocol overheads comes in two parts. The first is the Frame Efficiency of the protocol and is computed by the ratio of user data bits to the total number of bits which are transmitted: Ud T| f = ( -jT- ) ....the Frame efficiency of the network n

Further protocol delay is introduced in some networks due to their operating mode. For example, if an access permission ’token' must be held, or a given 'slot' must occur, before transmission is permitted then there may be a delay caused by other stations being offered the chance to transmit first In CSMA/CD based protocols if other stations wish to transmit at the same time then a collision will occur and a random ’back-off delay must elapse before transmission is retried. As is obvious from a cursory consideration of the examples, this extra protocol delay is variable from nearly zero to nearly infinite time and is different for each network. F n = — 2— the Protocol efficiency *P F + O n n

Equations for the protocol efficiency have been derived below for the various cases by deducing the network overhead (On), but to allow comparison of efficiencies, three specific points have been defined: • minimum delay, • best, worst case delay, and • worst, worst case delay. 32 The minimum delay is taken assuming that immediately after data has been read into the first network node the node is permitted to transmit, without hinderance, to an adjacent destination.

The best, worst case delay assumes that all other stations also wish to transmit, but only minimum length packets, and that they all may transmit before the chosen station. The stations are assumed to be at opposite ends of the network.

The derivation of these values also shows that some protocol and topology combinations do not scale. A network scales if as the performance of the technology employed increases the network's performances increases to match. All networks increase the delay as the length of the network media is increased. Networks like Ethernet, Token-passing ring, and Sonet also lose efficiency as the data rate increases or the length of the network is increased. Networks like Hubnet, Cambridge ring or Synchronet do not lose efficiency as the length of the network or its data rate are increased.

The delays in input and output of data at the end stations Tj0 are dependant on the interface hardware employed. If the interface is a standard serial link using interrupts to handle a character at a time, as are most computer access ports for users, then the delay will be an order of magnitude higher than for an interface using small cache buffers and direct store access. To a first approximation, for any given connection, this should be assumed to be a constant as all networks could employ the same type of interface hardware. The problem area, namely using standard networks for individual character, or other small, transfer will new be shown.

2.1 Efficiency of CSMA/CD (Ethernet)

An Ethernet transmits successfully when one, and only one, station attempts to send. If more than one station is waiting to attempt to transmit following the end of the current transmission then a collision occurs. Stations which collide back off for a weighted, random time where the weighting is increased as the loading and frequency of collisions increases. The calculations of probabilities are derived from the simple model proposed in [Met76], but yield surprising results for the small sizes of transfers which are being considered in this thesis.

Destination Source Preamble Type Data (46-1500 bytes) CRC Address Address 1

64bits 48bits 48bits 16bits 32bits ?ap ◄------► minimum Cyclic Redundancy Check covers these fields

Figure 2.1 Ethernet Frame Format

Assuming N stations are ready and wish to transmit with Fn bits in a packet and a data rate of ^ bits per second gives the best, worst case calculation. There are N ways in which a single station may 33 transmit alone each of which has probability 1/N, whilst the other (N -l) stations wait with a probability of (1-(1/N)). Thus the probability that an Ethernet transfers data is:

/Oxt -J 1 n .-M /i 1-1 )xN-i =(1-^) / i 1 xN-l = P

If the ether is not satisfactorily acquired by a single station then there is a wasted period. The length of this period for a single collision is the time necessary to guarantee that the collision will be detected by all stations on the network. This is twice the single full trip time from one extremity of the ether to the other. This time sets the minimum packet size and so after any collisions a time equal to an integer multiple of minimum packet transmissions is lost The probability of any of the stations not having to wait is shown above. The probability of wasting a single 'slot' before a successful transmission is thus: P*. (1- P^) and the probability of wasting i slots is thus:

(i-i)tw-(i-(i4 )Nv

The mean wasted time is the mean of this geometric distribution:

d - d - i ) ”'1) W avg = ( l - l ) * 1 v N J Thus the proportion of time for which an Ethernet will carry good transmissions is: F (— ) VR * n where T is the time for a minimum length packet T1 = F ( _ i + Wavg • T ) Rn 1 F . But rj. _ mm hence T| = R Fn + W avg .F mm . Note that though the data rate R^ seems irrelevant to this calculation, F ^ is actually determined by the length of the Ether, the velocity of signal propagation, the specification calling for a minimum of 0.77.C, and the data rate Rn.

For minimum length packets (i.e. those containing up to 46 bytes of data) then Fn = F ^ and so this simplifies to:

( i - - ) N-x Ethernet ^ = 1 + Wavg v N J This is a somewhat surprising result as it only depends on the number of competing stations, but remember that the minimum packet size under consideration is fixed by the maximum round-trip delay, the velocity of propagation, and the data rate. This minimum packet size is 512 bits plus a 64 bit preamble for an Ethernet operating at 10Mbps.

Ethernet is, of course, much more efficient for long packets than for short ones. If the packet contains 4 Kbits then the protocol efficiency is normally 98%. This drops to 91% for two competing stations with 0.5 Kbit packets and to 86% for large numbers of stations competing with packets of 34 this size. However, for small packet sizes (1-46 bytes of data) the maximum efficiency for competing stations is 50%. If there are five competing stations, with these sizes of packet, the maximum protocol efficiency is 40% and this drops asymptotically to 36.8% for large numbers of stations. It should be noted that the losses above are simply due to the operation of the protocol and the frame efficiency must be included to give the overall efficiency.

As can be seen from figure 2.1, the data field in an Ethernet packet varies from 46 to 1500 bytes. The overhead for addressing and control is 32 bytes but there is also a minimum interframe gap equivalent to 120 bytes at 10Mbps. The maximum frame efficiency occurs with 1500 bytes of user data per frame when: 'Hf = 90.8%. Figure 2.2 shows the protocol, frame and total efficiencies for Ethernet for a range of user data sizes and with 2, 5, and 64 competing stations.

User bytes 1 10 40 512 2 nodes 50% 50% 50% 98.8% 5 nodes 41% 41% 41% 98.3% P 64 nodes 37.1% 37.1% 37.1% 98%

Ilf 0.51% 5.05% 20.2% 77.1% 2 nodes 0.25% 2.53% 10.1% 76.2% 5 nodes 0.21% 2.07% 8.28% 75.8% 64 nodes 0.19% 1.87% 7.49% 75.6%

Figure 2.2 Ethernet Efficiency (for 2,5 and 64 competing stations)

Proposals have been made and demonstrated for a wide variety of alterations to the basic CSMA/CD bus to improve the delay and/or efficiency characteristics. The use of dual ethers where contention is restricted to one bus thus permitting deterministic and more efficient data transfer on the other, overcomes half of the problem, the variability of delay, but does not solve the problem of low efficiency for very small transfers. Though worst-case efficiency is improved significantly, the media and transceiver cost is doubled. The system is not cost-effective for direct connection of character-oriented devices. An alternative solution, using an unchanged single ether, which can improve the worst-case efficiency is presented in chapter nine, but it still does not provide a cost-effective way of connecting individual character-oriented devices to the network.

The use of prioritised transfers on ethemets [Ono78] again solves part of the problem, but it is inherent in any collision-based bus system that there must be a minimum size of packet. As the speed of the network is increased this minimum size increases and so small size transfers become steadily less efficient. It should be borne in mind that part of this inefficiency can be removed if devices are co-located as the individual small transfers from a number of stations can be multiplexed into a single packet An even better solution which does not require co-location is to use a distributed multiplexer and this is considered further in chapter four. 35 2.2 Efficiency of Token-Passing Ring

When token-passing protocols are used with a physical ring a token, or access granting message, is circulated around the ring. It is circulated freely in the absence of stations wishing to transmit In the IEEE token passing ring standard [IEEE83e] adopted by the IBM corporation, the token is 24 bits long. When a station wishes to transmit it waits until it is offered the token. It then captures the token, marking it busy, and transmits a data frame onto the ring. The frame is passed on by intervening stations with a small delay (2.5 bit times per node in the Texas Instruments set of chips) to the destination station. The data is copied from the frame and acknowledged by setting the ’address recognised’ and ’frame copied' bits. The frame then traverses the remainder of the ring back to the originating station which removes the frame. If the acknowledgements are correct then the token is set 'not busy' and transmitted to the next station on the ring. Thus a station takes:

( ) seconds to acquire the token from its nearest neighbour. n n Circulating a minimum length frame around the ring takes: 2.5 N + seconds. ( R n Thus the delay in sending a best, worst case message is: , F . + F . + 2.5 N N , d N N (_mm------) + (N+1)(_il) secs. R v n n Hence the proportion of the time during which useful data frames are being sent in general is :

.... Token-passing ring T1*p = (N+1)R .d (F„ + F,ok + 2-5N) + N.v

As would be expected the protocol efficiency increases with increased user data in a packet as the overheads are spread over a larger number of user data bits. For a lKm ring with a data rate of 4Mbps and velocity of propagation of 0.7C the maximum efficiency for 4K bits of user data in a packet is 98.7%. Curiously this occurs for three competing stations and the protocol efficiency is actually slighdy less for two competing stations. This maximum at three stations applies to all user data sizes for the given network parameters. The reason for this maxima is the crossover of the delay through one station and the delay when the token is passed around the ring between stations. The protocol efficiency drops to 95.4% for 64 competing stations but is 86.2% for 256 stations which interestingly is the same as for the CSMA/CD described above.

For small packets, however, the protocol efficiency of a token passing ring varies dramatically depending on the number of competing stations. Again, making best worst-case assumptions for a single character of user data at each station of the network described above, the protocol efficiency never rises above 75.6% (again at 3 stations). This drops to 70% with twelve competing stations, to 50% with 54 stations and crosses the CSMA/CD asymptote at just over 100 stations therafter performing worse. 36 /------\ User bytes 1 10 40 512 ri lp maximum 75.6% 81.3% 89.6% 98.7% Stations @ ^ p =50% 54 82 178 >86% at 256. V ______/ Figure 2.3 Token-passing Ring Protocol Efficiency

As for the CSMA/CD case these losses are simply due to the operation of the protocol and the frame efficiency must be included to give the overall efficiency. The protocol losses include all of the token overheads so the frame efficiency is again the ratio of user data to overall frame length.

Token Frame Start Access Ending Delimiter Control Delimiter / \ PPP = priority bits PPP T M RRR T = token busy/firee bit Address M = monitor bit accepted Data Frame bits 3 1 1 3 RRR = priority reservation bits Message accepted Start Access Frame Destination Source :;Data (variable) g CRC Ending Delimiter Control Control Address Address Delimiter 8bits 8bits 8bits 48bits 48bits 32bits 8bits 8bits

Cyclic Redundancy Check covers these fields

Figure 2.4 Token-passing Ring Frame Format

The frame format of the IEEE802.5 standard is shown in figure 2.4. The overhead per frame is constant at twenty-one bytes. Combining the protocol and frame efficiencies for various numbers of competing stations yields the results in figure 2.5. The token passing ring performs better than an equivalent Ethernet for all categories. The main reason for this is the 9.6 microsecond gap between frames in Ethernet. The token passing ring will also exhibit stable, deterministic behaviour when the load is taken to extreme levels.

User bytes 1 10 40 512 2 nodes 75.3% 81.2% 89.4% 98.7% T| 5 nodes 74.8% 80.7% 89.2% 98.6% p 64 nodes 46.7% 54.9% 70.6% 95.4%

i i f 4.76% 32.3% 65.6% 96.1% 2 nodes 3.42% 26.2% 58.6% 94.8% Tjrj, 5 nodes 3.40% 26.0% 58.5% 94.7% 64 nodes 2.12% 17.7% 46.3% 92.3% ______J

Figure 2.5 Token-passing Ring Efficiency (for 2,5 and 64 competing stations) 37 2.3 Efficiency of Cambridge Ring

A Cambridge ring continually circulates an integer number (1-15) of 40-bit long minipackets. A gap of two or more zero bits must be present at one and only one point in the cycle of minipackets to identify the complete circulating slot ring structure. Each node contains a single bit register and exhibits a delay of one bit so that any bit of a minipacket can be read in and modified if necessary prior to transmitting it onwards. A monitor station is included both for error recovery and to ensure the ring has sufficient delay, in addition to that in the stations and the interconnections between them, to accommodate the extra bits necessary to complete the last minipacket Cable interconnection exhibits a delay of approximately one bit per 25 metres.

If a station wishes to transmit it waits until it is passing a minipacket which has the Full/Empty bit set to zero to indicate that the rest of the minipacket is empty. It then sets this bit to full (one) and transmits it. It sets the Monitor Passed bit to one and transmits it, followed by the desired destination address and its own source address. It then sends out the sixteen bits of data that it is transferring, the two type bits and the response bits, both set to one. These are followed by an even parity bit. The node then continues circulating other minipackets until it receives the packet it sent out It must then mark the packet as empty. It may not attempt any other transmission from the time it first marks the packet as full until it has marked the same packet as empty and forwarded all of its bits to the next station. In the process of passing these bits it examines the response bits. In fact the CR82 specification [Lar82, Sha82] insists that a node shall pass the next minipacket through aswell, regardless of its Full/Empty status. Though this arbitrarily restricts the point-to-point bandwidth it does not alter the overall efficiency. The Monitor station sets and maintains the slot-framing bit and the gap. It also sets the Monitor Pass bit to zero when it passes a packet so that it can clear any packet which it sees has not been emptied by its transmitting station, and so is still full. The gap and leader bits are needed to ensure that synchronisation is maintained.

Destination Source 1 f/e mp Data R esp P Address Address Type

1: Slot framing bit = 1 f/e Minipacket full or empty mp Monitor passed P Even parity check Response: ll=Ignored, 10=Not selected, 01=Accepted, 00=Busy Addresses: 8 bits each Data: 16 bits Type: 2 bits Figure 2.6 Cambridge Ring Frame Format

As slots are densely packed the protocol efficiency is only reduced from 100% by the gap and the fact that each station delays the data by one bit Thus each minipacket must move on by 1 bit time, carrying no useful data, each time it is to be filled with usable data by the next node. So if there are M minipackets on a given ring, ignoring any cable delay, the protocol efficiency is:

M.Fn .... Cambridge Ring = (M(Fn+l) +2) 38

The protocol efficiency is controlled by the number of minipackets. For a single minipacket it is 93%, rising to 95.2% for two minipackets and to 97.2% for the maximum of 15 minipackets.

The frame format for a Cambridge ring is shown in figure 2.6. As can be seen there are sixteen data bits in a total frame of forty bits so the frame efficiency is constant at Tj f = 40% if both bytes are used but drops to rj f = 20% for the transfer of a single byte. Hence the best possible efficiency for a Cambridge ring is rj T = 38.9%, and the worst possible efficiency is T| T = 18.6%. If the best, worst case assumption is taken and there are N nodes on the ring then the point-to-point bandwidth which a node can achieve in best worst case is 4/(M+N) Mbps assuming that all receivers can take the data sent to them and none show 'busy' status.

2.4 Efficiency of Mesh Networks

There are many possible configurations and operational protocols for mesh connected networks. They all have the same basic property, which is that a packet must be read in to a node in its entirety before it can be checked, and routed for onward transmission. Though high line utilisation figures are always claimed for mesh networks their extremely poor delay characteristics usually outweigh their notionally higher total efficiencies for local area use. R .d Delay (min^ = N.Fn+ ■ n n .... Mesh (ViaNnodes)

If point-to-point bandwidth is considered then any interactive use immediately suffers from the store-and-forward delay. It is unfair to attempt to quote overall efficiencies for mesh networks as they are usually designed to have more redundant links than other local area network configurations. Thus if these are included the figures produced are too low to be fair to the mesh, and if they are not then the high load performance of the net would be adversely affected and the system would be unworkable in practice. Suffice it to say that simple measurements on all available networks gave efficiencies which were always lower than the token-passing ring, but no further attempt was made to calculate rj p for meshes.

It is reasonable to calculate the frame efficiency, though, as the majority of modem, mesh networks use Open Systems Interconnection or formats with similar overheads. Fig 2.7 shows the typical CCnT X-25 packet format [IS079,Fol82].

0111 CONTROL CONTROL ADDRESS QDOt Logical Chan 1110 0 N (s) PF N(r) 0 P(s) M P(r)

8 8 8 4 12 8

Figure 2.7 Typical Mesh Frame Format (eg X-25) 39 As can be seen there is a minimum overhead for transferring data down an established logical (virtual) channel of 72 bits per packet. This is usually increased by higher levels of protocol, extended modulo for sequence numbers, extended control or the bit insertion process, but if it is not then the frame efficiency can never be better than r\ f=10% for a single byte of data, 81% for forty bytes, and 99% for large frames.

2.5 Efficiency of Multiplexer-based Solutions

It is possible to construct networks from multiplexers or to use multiplexers as part of a network solution. The efficiency and delay of individual multiplexing elements will be considered prior to linking them into, or to form, a network. The most efficient time division multiplexers for small numbers of channels are the Bell T1 (DS-1) system multiplexing 24 channels onto a channel of 1.544 Mbps or the equivalent, but slightly less efficient, European CC1TT level 1 system multiplexing 30 channels onto a channel of 2.048 Mbps. Other international standards, capable of being carried on twisted pair cable, are the T2 (DS-2) system putting 96 channels onto a channel of 6.312 Mbps or the equivalent European CCITT level 2 system which puts 120 channels onto a channel of 8.448 Mbps. Above this co-axial cables have to be used and the efficiencies are reduced as they multiplex multiple multiplexed channels onto the cable.

As a typical synchronous system assume the more efficient T1 multiplexer. To gain the highest efficiency an alternating bit protocol is used which adds a single bit per major cycle of 192 bits to give a framing efficiency of 99.5%. One can assume than no additional protocol is required if a simple multiplexed path is used. However when used for digital data, rather than pulse code modulated voice channels, only 23 of the 24 channels are used for data. The last byte of each frame holds a synchronisation pattern to permit faster and more reliable reframing following any framing error. This reduces the frame efficiency to q f =94.5%. If any control information is to be included into the channels, as is common practice, then a further bit is required and the frame efficiency drops tori f =83.4%.

Turning to statistical or demand based multiplexers each channel has an additional bit to indicate if the frame is used or does not exist Assuming a similar scheme of 24 channels, framing now takes a minimum of 25 bits. For just a single active channel there is a very large improvement as 83% fewer bits are transferred. In the best worst case however the efficiency drops to 87% of the equivalent synchronous system. These efficiencies are for a single level of multiplexer but for a full network of any size there will be two or more levels. Also, no allowance has been made for switching which is included in the other alternatives and would reduce protocol efficiency.

The delay introduced by each multiplexer link is nearly minimal for a network at 2,Ttd. However with two or more stages required, as above, then the delay increases in integer multiples of Ttd. Other multiplexing schemes, based on HDLC frames exist and their efficiencies are the same as those already derived for mesh networks in section 2.4. 40 2.6 Summary of Network Efficiencies

A summary of the total efficiencies calculated for the international standard types of local area network is shown in figure 2.8. As can be seen the shaded area is not covered by any existing standard network and is the major reason for the further investigation presented here. Both star and multiplexer solutions would appear to offer the opportunity for operation in the shaded area. The network parameters which contribute to efficiency and delay performance, particularly for the small-sized transfers which are of interest, are analysed in the following chapter. 4 1 3.0 IMPROVING EFFICIENCY AND MINIMISING DELAY

As was apparent from the preceeding chapters, small-data devices are not catered for efficiently in networks which are also convenient for interconnecting them. The main overhead in transmission of packets is the number of non-user bits in each network frame. To be pedantic it is the number of non-user transitions, and the number of transitions (if more than one) per user data bit which form the overhead. This overhead mainly reduces efficiency, but it also increases delay. The other important overhead identified by the preceeding analysis is the operation of the chosen protocol. Whilst this also decreases efficiency it is the main contributory factor to increasing delay. We now turn to analysis of the non-user data. It is of three types:

• Addressing, • Control, and • Redundancy for error control.

First consider the requirement for addressing. Nodes within a network are assigned physical addresses to distinguish one from another. An important feature of local area networks is their ability to handle logical naming so that a connection could be made to MARTIN instead of having to remember that he uses box 3025. A particular advantage of logical naming is the possibility of generic naming. Generic names permit a number of nodes to have the same name, where connection to or from any one would be acceptable. The mapping from logical to physical names must be performed transparently for the user.

In theory each packet could carry the full logical address of the required destination and this could be used for all routing. This carries an enormous overhead, such as to make the problem intractable. Of course, the mapping from logical to physical names may change at any time and so central servers are always used to overcome the problems of logical name mapping. There are two approaches to transfer of data which affect the format of the physical addressing of packets.

A datagram packet, one which is a complete and separately-routable entity, must contain a full destination address. If the network is linked to the wide area network, or if itself is large, then the address range is large. For a datagram both destination and source addresses must be carried to ensure correct delivery and acknowledgement. Many of the network standards allow 48 bits for each address. This is a considerable overhead for short packets. However, the majority of network transfers are not isolated entities and this fact can be used to reduce the overhead in them.

If a virtual channel is established then a table can be set up in each station which needs to route packets for the connection. Packets only need to carry the reduced form of the address, i.e. a logical channel number which is the index into the table, to allow them to be successfully routed. As a bidirectional channel (set of table entries) may be established then only the reduced form of the destination address is needed in a packet to give a further saving. 42 If a virtual channel is established then it is possible to go further and infer the address for routing a given packet without needing to carry it in either full or partial form. For example, in a star-organised network such as a Gandalf [Gan85,Gan86] where all routing is carried out at a single point then only a single table is needed to show the destination port for any given source port Thus the address is inferred from the address of the line on which the data came in and its associated table entry. The destination for any input which is not connected via a virtual channel to an output is a (queued) allocater port This is used by the input port to convey the desired destination port (once only) so that the allocater can make the necessary table entry to permit subsequent transfers to be addressed implicitly.

For a distributed network to gain similar benefits, the address inference method has to be made inherent in the mechanism used to subdivide the network among the users. The principle is that logical 'slots' used by a connected device, which is transmitting, are known to any device wishing to receive from it. The network may be divided by time or frequency as discussed in section 1.6 and so logical 'slots' may be mapped onto physical separations. It is then only necessary to be able to delineate the physical 'slots' and use an absolute or incremental method of chosing the right one to transmit in or receive from. If frequency division is used then an absolute value (the frequency band) can be known, but for time division only reference points and positions relative to them can be known.

On a time divided network the delineation of slots can be done in a simple way by having a separate channel on which to carry 'slot markers'. This has a disadvantage of possible skew between the marker and data channels as well as the overhead of the extra channel. More efficient methods use synchronisation markers distributed between some (or all) of the data frames thus saving the extra channel. The most efficient method is to distribute synchronisation markers between major cycles only, in a manner similar to that described for multiplexers and use the accuracy of local crystal- controlled clocks to delineate the positions of minor cycle data frames.

r ADDRESS

CARRIED INFEF>RED

/ FULL PARTIAL TABLE INHERENT ---- Separate Channel Destination Logical (star only) (in slotting) and Source Chan. no. 1 1------Distributed Sync. v______V- J V DATAGRAM VIRTUAL CHANNEL ---- Master Sync plus local clocks J

Figure 3.1 Addressing Mechanisms for Networks 43 The main methods of carrying slot-delineation information which have been studied here are: • An infrequent synchronisation pattern used to initiate a local timer to locate the slot. • Synchronisation patterns distributed throughout the data to remove the timer but giving rise to problems of transparency. • Synchronisation patterns sent by electrically separate means e.g. voltages or frequencies • Frame and packet synchronisations sent on a separate physical channel.

These options have the benefit of simplicity but at the expense of bandwidth, which notionally could have been used for data, being used for synchronisation. Both the first and last methods above have been tested for this thesis and are described further on with the appropriate hardware and algorithms.

3.1 Unit of Transfer

The size of data input from a user device and the size of data transferred across the network are not necessarily the same. It is the size of individually routable entities on the network which has an important bearing on the performance of the network both in terms of potential data rates and user convenience. Four sizes are identified: • a bit or subpart of a bit, • a byte or character, • a line or block of from 1 to 100 bytes or characters, or • a block of from 1 to 1000 or more bytes.

One may then see the delay introduced by transmitting and switching these entities and the possibilities of flow control and changes of data rate or data fo rm a t by the network. A summary of these possibilities is shown in figure 3.2.

T ra n s fe r D e la y Flow Control Speed Change Format change

B i t B its End to End Not possible Not possible h possible B y te B ytes but not Possible 0 ) P ossible e s s e n tia l

L in e Lines Possible (2) P ossible 11 Node / Node B lo c k Blocks r essential Possible (2) P ossible j

Notes: (1) restricted to character level (2) if user-device aswell as node to node flow control employed

Figure 3.2 Network Characteristics by Unit of Transfer 44 If each entity must be completely stored by a node prior to forwarding then, for a given rate of transmission over a network link, the smaller the entity the lower the delay in transmission over an entire network. This does, of course, assume that no additional overheads are incurred by utilising a smaller unit of transfer. Some attributes, however, will be very difficult to incorporate without significant additional overheads. As an example consider the problem of error detection when the unit of transfer is a bit or subpart of a bit The question of overheads in the control of data transfer is discussed in section 3.2.

Smaller minipackets also require less buffering at end nodes, and at intermediate nodes in store- and-forward networks. They also reduce the necessity for flow control and give more even store- access patterns as they can operate with continuous small cycle steals, rather than with long waits followed by long bursts of store access.

For machine to machine communication, throughput is more important than initial delay, but low delay is a crucial factor for networks used by people. If a single character, for example a cursor movement, is input by the user then it must traverse the network, be input and processed by the target machine and then the appropriate response must be output to traverse the network back to the user. The throughput required is negligible yet two initial network delays are involved. If the delay is higher than the low-limit of human operation (or perception) then the 'system' will be disliked by most users and will be unacceptable to many.

From the particular characteristics shown, a good case can be argued for either bit or byte units for user networks. For m achine networks there is an excellent case for blocks of 1000 bytes or more to give better throughput and utilisation. It is not surprising that all of the ISO standard networks adopy this size. The major drawback of the bit size is the difficulty of providing any error checking mechanism at the mini-packet level. Any such mechanism would impose a large overhead. Error checking at this level is not provided on PABX type switches or multiplexers.

The only size for which no justification appears to exist is the line, however, most packet assembler/ disassemblers (PAD's) used for terminal concentrators work with this sized unit! A packet is sent when a carriage return character is input or the buffer (of 80-256 chars) is full. This is inadequate to support single key cursor controls [e.g. <= ft li => ] and all "line" unit based systems such as PADs have to have a 'mode' which is capable of sending individual characters. This is inefficient and usually has to be arranged by some sort of timeout mechanism, e.g. after a set time has elapsed then the buffer is sent regardless of its contents.

3.2 Synchronisation of Transfer

For data to be transferred reliably at the bit level each bit must either be clocked or else be synchronised by a handshake mechanism. For the fastest transfer of data, a separate transmitted clock accompanies the data on an additional path or channel. Without losing any speed the same two 45 channels can be transition encoded so that a 'one' is signalled by a transition on one channel whilst a 'zero’ is signalled by a transition on the other. The use of distinct transmit and receive clocks with the receiver resynchronising its clock to the transmitters regularly obviates the need for the extra channel. The cost is two bit-times for every resynchronisation and a clock accuracy of (100/n%) for n bit blocks with this isochronous technique. The use of an embedded transmit clock gives only half the maximum data rate for a similar baud rate, but obviates the need for the extra 'clock channel'. It has the great advantage that clocking errors of up to 25% can be tolerated with data still being received reliably. The fully asynchronous handshake transfer mechanism is quite unsuited to use in local area networks as two extra channels are needed and the data rate can not be more than a third of the fastest synchronous method. For the Sonet experiment a modified phase-encoded, embedded transmit clock system was used and this is described in chapter 6 with its implementation details. Full details of the possible bit synchronisation methods are given in [Cri87].

3.3 Minimising the Overhead of Routing

There are two low-level mechanisms employed to arrange for a given packet to reach a given destination. These are Store-and-forward routing and Selective reception routing. A complete 'unit-of-transfer' may be read in, analysed and then be routed by sending it out down a chosen link. This is the mechanism used by all wide area networks and those local area networks using wide area principles. The majority of local area networks, however, use selective reception. Here many packets are seen by a given receiving node and it is the function of that node to ensure that it only takes, and acknowledges, packets with the destination address matching its own reception address.

Obviously to permit bridges and gateways to be made there must be promiscuous devices which will transfer packets with any address. Where a gateway is to be made selective, as to the packets it will forward, then it must read at least the start and destination address from a packet before it can decide whether to forward or ignore the packet. It should be noted that in store-and-forward routing it is the transmitter which carries the overhead of routing, wheras the receiver carries the overhead for selective reception. The latter minimises delay as no overhead is required of the transmitter before it sends out the data packet

3.4 Minimising the Overhead of Control

Again, two activities contribute to the control overhead: • Acquiring channel 'bandwidth' on the network to permit data transfer • Transferring control information across the network or between nodes.

The aquisition of a channel from source to destination simply requires queueing if store-and-forward routing is being used. One of three methods: Active Contention, Passive Contention or Contention avoidance is chosen for selective reception. The attributes of the these are listed in figure 3.3 46 CSMA/CD Token Passing Time Division or Empty Slot Multiple Access

• Active contention • Passive contention • N) Contention

• Unstable • Stable • Stable • Burst rate only • Burst rate and • guaranteed rate guaranteed minimum

Figure 3.3 Path Access Methods

3.5 Minimising Redundancy for Error Control

The addition of redundancy for error detection and control is an integral part of all networks. There is a wide variation in the amount of redundancy which is added and the level in the network protocols where it is calculated and checked. It varies from none added by the network in a Gandalf star-switch [Gan85] through a single bit in Sonet and a Cambridge ring to thirty-two bits in a token-passing ring and an Ethernet. The redundancy is added for two different reasons.

The first is to permit the performance of a network to be monitored to enable preventative maintenance to be scheduled before faults become apparent to users. The second reason is to give the user some reassurance that his data has been correctly transferred across the network. Of course there can be no absolute guarantee that it has been sent correctly, even if it was returned for checking. The user's data, being sent in individual packets needs two types of checks. One on the data in each packet and a further check on the sequencing of packets. Some network protocols are guaranteed not to be able to mis-sequence packets on delivery. The level of reassurance provided, that the data has been correctly transferred, can be appreciated from the figures for the CCITT V41, sixteen bit cyclic redundancy check. All burst errors of 16 bits or less are detected and over 99.99% of longer burst errors [Dav84]. However, four isolated bits in error may easily go undetected.

The probability of an error occurring in a block naturally increases as the block length increases if the overall rate of occurrence of errors on the transmission channel is constant. Large blocks are transmitted less frequently, but if a block is received in error then the whole block has to be retransmitted and this constitutes a greater increase in load over small packets. This is counteracted in some part because the overhead of redundancy checks for a given level of protection increases as: O ( log{no of bits}). Ideally if a single bit was in error only that bit would be retransmitted, but of course this is not practical! Similarly an increase in the data rate used over a particular link means that if a burst error of given duration occurs, more bits will be corrupted, and so more redundant bits will need to be transmitted to give the same level of confidence that the data was received correctly. This is the reason for the 32 bit cyclic redundancy check used by IEEE 802.3 and 802.5 by comparison with the 16 bit cyclic redundancy check used by X-25. 47 3.6 Summary of Methods for High Efficiency and Low Delay

A summary of the contributions to efficiency (or lack of it) and delay presented to character-oriented and interactive-user data by a network follows. These are a combination of the overheads due to the operation of protocols and the overheads in the chosen frame format.

* Minimising protocol overheads • Dense packing of frames on the network media • No contention, or at worst passive contention for media access • Virtual channel rather than datagram operation • Selective reception rather than store-and-forward routing to minimise transmitter delay • Small unit of transfer (minipacket) rather than large packet or message

• Minimising frame overheads • Inherent addressing rather than direct (full or reduced) addressing • Minimise synchronisation bits by use of internal clocks and infrequent global resynchronisation • Control carried in separate frames rather than as an overhead on every frame • Only error check when it is of direct use and higher level checks are not in operation • Minimise error check on individual minipackets (only useful for maintenance). • Use the lowest feasable data rate on the network to reduce the number of bits corrupted by a given burst error and hence minimise redundancy required. 48 4.0 THE PROPOSED SOLUTION: SYNCHRONET

Having considered the methods described in chapter three to reduce overheads and increase the efficiency of networks for character-oriented devices, a new type of network structure and protocol was proposed by the author [Cri79]. The major design aim for this network, called Synchronet, was that it should provide high efficiency and low delay for short packets. Furthermore, it was essential that it achieve stable operation with delivered traffic being a non-decreasing function of offered load. It must also be reliable and simple to maintain yet permit frequent changes to the configuration, and be suitable for implementation with shielded or unshielded twisted-pair cables. The network was to be designed to support all serially interfaced device types using standard interfaces existing at the time, and their a d hoc flow control schemes. Finally it was to be able to operate with application level protocols without modification of existing operating systems and as a transparent feeder to other networks.

These aims were completely achieved by using a bus topology for the connection media required, but so arranged that frames of data could be densely packed and transmitted without collisions. The bus topology was chosen to simplify and minimise the cabling, and to permit incremental growth. To increase efficiency the proposed protocol was based on the virtual channel principle with implicit addressing. To minimise delay a very small basic packet size was chosen with as small an amount of redundancy as was essential to support the virtual channel operation.

4.1 Synchronet Topology and Data Flows

Two difficulties are presented by the choice of a bus topology, common to all simple-bus layouts. First, a method for chosing which station takes control of the bus must be selected. Secondly, however, regardless of this choice, if two stations located at opposite ends of the bus are to transmit consecutively then a blank time would normally have to elapse. This blank time must be at least equivalent to the delay between them. This blank time would prevent dense packing and reduce efficiency, but is overcome by the Synchronet proposal.

The key feature of mesh topologies which permits them to densely pack data frames is that the transmitter always precedes the receiver and so the direction o f data flow is fixed. This was a fundamental observation which led to the achievement of dense packing on an unbroken media. To reduce addressing overheads the time-sliced slot mechanism described earlier was employed and this also contributes to overcoming the restriction of blank times.

By constraining a bus to have transmitters preceding receivers, unidirectional data flow can be imposed on the bus. There are two logical, unidirectional flows on a Synchronet. The first is the users' data frames and the second is the synchronisation information required to exactly delineate the time periods (or slots) for the data frames. As will be seen other desirable characteristics are found once the direction of data flow is known. 4 9 A bus layout with separated data and timing channels which can give unidirectional information flow is shown in figure 4.1. The two logical flows which are shown in figure 4.2 are easily combined on a single loop-back channel.

There are three points of particular interest on the unidirectional bus: the start, the loopback point and the end.

Figure 4.1 Synchronet Unidirectional Bus

The station located at the start of the Synchronet unidirectional bus provides the correct termination to match the characteristic impedance and originates the synchronisation pulses to determine the major cycle slot timing. Any of the mechanisms described in section 3.0 could be used though the preferred method would be to use a phase-violating pulse of sufficient duration (more than two cell-times) to provide the synchronisation at the start of every major cycle. Each station on the bus detects this violation and synchronises to the end of the pulse subsequently using its own internal, crystal-controlled clock to give minor cycle (slot) divisions until the next major synchronisation.

synchronisation pulse

data slots Figure 4.2 Synchronet Data Flows

The bus runs from the start past all transmitters which tap onto it, to the loopback point, which is at the furthest extremity that has to be reached. The return side (cable) of the bus then has all the stations' receivers tapped onto it until it reaches the end which is physically the same station as the start Here the bus is again terminated to match the characteristic impedance and the end station can 50 continuously monitor the synchronisation pulses to ensure cable integrity. The start/end station also performs an important function, described further on, to establish optimal minor cycle timing. It should be noted, however, that all stations can be given the ability to act as the start/end station as the additional cost is minimal. This is discussed further in chapter eight.

As all transmitting stations are located between the start and loopback points, and all receivers are located between the loopback and end points, the direction of data flow is fixed and dense packing of frames can be achieved. This would present no problem for fibre-optic cable as the light is constrained to propagate in the forward direction only, as shown in figure 4.3. Fibre-optic cable is currently not an implementation option because of its cost and the difficulty of connection. For an electro-magnetic wave at frequencies suitable for propagation on a copper twisted pair, a signal originating at some point along the bus will travel both towards the loopback point and back towards the start point If all the stations transmit at the correct time regardless of the reverse wave then at the loopback point (and at all points on the receiving side) the signals must be correctly sequenced and densely packed without interference. The reverse wave will dissipate in the termination and the design of drivers capable of achieving this type of operation are shown in section 6.1.

Data in slots on the return path ▲ r A

T T+10 T+20 T+30 T+40 Time Slot Timing _ Indications __TL_ ____ n ______n ______n ______

Figure 4.3 Synchronet Data Transmission in Slots

The next part of the Synchronet approach is the slot allocation mechanism and the detailed timing of the slots and data frames. A major cycle consists of three parts: the synchronisation "pulse", the maintenance slots and the communication (data) slots. The most sophisticated operating mode will be described, but it will be apparent that there are various simplifications which could be made but each would have an associated small loss of efficiency.

Data is transmitted onto the bus with enchronous encoding [Cri87] an example of which is given in section 6.5. The major cycle synchronisation can thus be established by transmitting more than two bit-cell times in the "one" state. This establishes a deliberate encoding violation which is easily detected by a simple monostable circuit in the receivers. Following the detection of the violation, 5 1 stations synchronise to the transition marking the end of the synchronisation pulse (i.e. back to the zero state). All stations reset, resynchronise and expect the few maintenance slots (2 to 4), some of which have sloppy timing. Sloppy timing means that those first slots are wide enough to permit a data frame and the propagation delay down the length of the bus. The reason for the extra width of some maintenance slots is to allow bus reconfiguration and retiming to be dynamically adjusted at the establishment of a virtual channel. An alternative would be to have the timing set by parameters in each station which, as these would have to err on the side of caution, would not be optimal. Following the maintenance slots the stations expect the communication slots (say 128 to 512) for carrying user data frames. These are only wide enough to carry data frames with minimal guard times between them.

4.2 Synchronet Dynamic Bus Timing

The bus timing for most dense packing can be determined dynamically. First the delay of the whole bus must be established to determine the relative delay of the synchronisation transition. The start/end station which originates the synchronisation does this by timing the round-trip delay of the trailing edge of the pulses which it is monitoring anyway. It informs other stations of this time using the maintenance slots. When a station wishes to establish a virtual channel it first determines its own relative timing by transmitting in a sloppy, maintenance slot and measuring its own round-trip delay. The station now knows the total bus delay (Tn(j(max)) and the delay from its transmitter to its receiver (Tn(j). From these it can determine the delay from the origination of the synchronisation transition to when it passes the station's transmitter {{Tn(i(maxyTn(i)/2}. This establishes the station's position on the bus (which aids in fault detection and location as described in chapter 8) and its relative timing to ensure all data communication frames are densely packed.

4.3 Synchronet Connection Unit

A simple connection box provides the interface between the common cables and various user interfaces such as CCT1T V24 or X21. The simplest operating mechanism to ensure that there are no contention problems is to give each box a unique, fixed-wired talker slot address. The unit may only transmit during the corresponding slot. A single box may have two or more adjacent slots where higher data rates are required. This would guarantee that all stations on a network would have a minimum allocation of bandwidth. A more flexible scheme which would allow stations to take advantage of other stations unused allocations is to have variable talker-slot numbers and this is described with the Sonet experiment in chapter 5.

A simple mechanism was devised to cany out the network switching function. Communication links are allocated and switched using the variable listener slot address, the value for each box being loaded into it for the duration of a link by a dynamic/orcer arrangement. A small allocater computer is connected to the forcer hardware and to a lazy scanner which samples each talker slot when 52 required. The allocater also has some ordinary connection ports through which users may interact and issue requests for links to any computer, service or other terminal in the system, or to access network parameters. Manual override hardware may be easily included in the forcer and scanner interfaces so that in the event of a complete loss of computer allocation facilities, and all backups, changes could still be made.

The start unit has its protocol and timing driver activated to provide pulses to synchronise all other connected units at the start of a cycle, and subsequently to define the width of the talker slots. It is set to provide as many slot indications as there are connected units, plus those for the link allocation mechanism and maintenance. The width of these pulses is adjustable to take account of skew, and the number is adjustable to permit additional connections. All other units disable their P/T function. The end unit also monitors the protocol and timing signals to ensure cable integrity is maintained

4.4 Synchronet Frame Formats

Assume that each connected unit is assigned a fixed-wired talker slot. Its transmission hardware synchronises and then counts timing pulses until it reaches its assigned slot It then transmits a data frame in the format shown in figure 4.4. This long format can be reduced to ten bits as described in section 4.10 and theoretically could be reduced to nine bits were error checking for maintenance purposes sacrificed. If a connection box is powered on and its hardware checks are valid then it transmits an empty control frame. The absence of a D/C bit indicates that the given slot is inoperative, and so a unit assigned that slot can not be linked to. Users and their connection units indicate their requests to establish or disconnect links by sending control frames with the C/D bit set or cleared. The allocation mechanism keeps track of alterations to C/D bits and forces the required actions. The salient parts of the 'scan/action' table are shown in figure 4.5.

D/C DV-C/D D 0-D 7 (data bits)

1 bit 1 bit 8 bits 1 bit

C C/D C 0-C 7 (C/I, F, M, etc.)

D/C Data or Control in frame DV-C/D Data parity in frame or Connect/Disconnect/forward flow control DO - D7 Eight user data bits [or control frame] D8 Latched state for break/sync (reverse flow control) Figure 4.4. Synchronet Frame Format (Long)

X-21 synchronous interfaces provide their control and status indication mechanisms by using a single line in each direction (Control, Indicate) to change the meaning of the eight bit field from data to supervisory information. The I bit provides transfer of this line in each direction. For other interfaces, such as V-24, the I bit is unused and is available for alternate definition such as flow control or break conditions etc. 53 For a V24 interface the ten bits following the data bit could be regarded as an isochronous ten-bit data frame with start bit, eight data bits and a stop bit. It is easier, however, for the various interfaces to consider them as separately defined fields. D0-D7 always represent a character or eight data bits which can be transferred by a slot The DV bit is set to indicate the data field is valid, i.e. a character is being transferred in this slot This acts as the forward flow control for all synchronous and asynchronous interfaces. For synchronous interfaces D8 can also provide the necessary reverse flow control over the network. It also has a useful function for asynchronous interfaces. It determines the state in which the interface data transmit line will latch if idling between characters. This permits both continuous marking or spacing conditions to be transferred and so allows the use of the "break" key found on many terminals. The V24 interface has a plethora of individual control lines and these can easily be mapped into control frames for transfer over a Synchronet. Many of the possible protocol schemes and control lines are discussed further with the Sonet experiment

4.5 Link Management

Link management involves three phases: link establishment, data transfer while a link exists, and disconnection following completion of the necessary transfers. Assuming the frame format shown in figure 4.4, all active connection units must send a frame start bit (D/C) in its allocated slot If it does not it is assumed to be faulty and is ignored. The connection box logic is such that no other control or data bits will be transmitted if the 'D/C' bit is missing. A connection unit can idle sending empty data frames, but for maintenance purposes it is more practical to idle sending control information. All requests to initiate link management operations are signalled by changes to the Connect/Disconnect (C/D) bit in a control frame. It is at this point that the advantage of separation of the link management and data transfer roles of a network can be appreciated. Any existing link is maintained with no effort (or bits required) and only changes need to be signalled.

Synchronet manages its links, as well as maintenance and other functions, using three units each of which may have dynamic standby spares. These are the lazy scanner, the allocater and the forcer. The lazy scanner can monitor any, and all, slots so that other devices may have access to their content without altering it. The allocater maintains tables which describe the state of all known users, connection points, and links. It uses this information to permit the establishment of new links, the disconnection of expired links and to monitor the performance of the network. T he fo rcer uses the maintenance slots to change the listener addresses of devices to allow them to interconnect.

4.6 Lazy Scanning Mechanism

An important feature of Synchronet is lazy scanning. This method was devised to reduce the overhead of addressing, and link maintenance functions are carried out at a cycle rate much reduced from the transfer rate. A lazy scanner (and any dynamic or static standbys) is connected to the receiver data channel (with protocol and timing) and continuously monitors any chosen slots. The 5 4 fundamental lazy scanning interface is provided with a valid slot address Z. After a worst case of a cycle, the scanner stores the complete frame from that slot, placing the frame data and control on its interface and indicating its validity. Every time the slot passes again the data on the interface is updated to reflect the current values.

This mechanism can be used for a number of functions: 1. It is used to detect changes in C/D bits to request the establishment and clearing of links. 2. It is used to check on fo rc e r (see section 4.7) performing the correct function by checking maintenance slots during a. fo rce. 3. It is used to check that a. fo rce is correctly performed at a connection unit by monitoring the talker slot of the unit being forced for the acknowledgment. 4. It is used to confirm the integrity of the unidirectional bus cable(s). 5. It is used to detect failed, disconnected or un-powered connection boxes by the absence of their talker frame bits.

The scanner is entirely passive and so can not corrupt the data on the cables. Obviously a scanner which would monitor multiple slots could be easily produced. Dynamic standby units could be arranged so as to check whether any failure is due to the scanner or one of the faults noted above.

To return to the establishment of a link between two devices. Any unit which wishes to signal a request for a link to be allocated, or its availability to be allocated to a link later, causes a transition (0—>1) on its C/D bit in a control frame and holds it set This action could obviously be under the connected device's control, could be automatic on Terminal Ready', or be caused by a press button. A micro-computer, responsible for allocating links, uses the scanner to perform lazy scans across all the talker slots looking for C/D bits which do not match the entries in its tables. When the transition of the C/D bit of the requesting device is found, the allocater uses the forcer to force one of the allocater's communication ports to listen to the device’s talker slot and then to force the device to listen to one of the allocater’s talker slots.

A normal interaction can then take place, protected by a timeout to prevent allocater-port congestion. The user, or device, interrogates the allocater and instructs it to set up the required link. Once sufficient information has been entered, the allocater uses the forcer to free off its own port prior to forcing the establishment of the required links. For a simple terminal to computer link this would entail forcing the computer port to listen to the terminal's talker slot and the terminal to listen to the computer's talker slot thus establishing the connection.

Any established link remains, even in the event of failures of the allocater and other equipment, until either party to a link signals its desire to disconnect. It does this by causing a transition (1—>0) on its C/D bit in a control frame, and holding it cleared. Thus a device can clear a link or be forced off. The allocater can have special 'hold' features for certain links to give the permanent virtual circuits mentioned. When the allocater detects the C/D bit in disagreement with its tables it forces both sides of the link to listen to the unallocated slot 255 (X’FF). The C/D circuitry in connection boxes 5 5 includes a monostable circuit to ensure that alterations in C/D to disconnect links are held for at least the duration of a lazy scan. A summary of the actions for particular connect/disconnect transitions is shown in figure 4.5. Unallocated devices always listen to a 'null' slot, usually with a slot address of 0FFFtf£y, which can carry Ceefax-like status information.

Old C/D New Action

0 0 None (Idle device) 0 1 Queue for Force to Allocater 0 1 * Incoming Only: just update tables 1 0 Force Clear-connection & update 1 1 None (Connected device) 1 1 + Timeout Terminate interaction & Force Clear

Figure 4.5. Partial Scan - Action Table

4.7 Connection Forcing Mechanism

The primary function of the forcer is to change the listener addresses held in the various connection units. It connects to the talker data channel (with protocol and timing). In its simplest form it is assigned talker slots zero and one and, when strobed, it stores two eight bit addresses X and Y. It then forces talker X to load listener address Y by placing data X in slot zero and data Y in slot one, following the next synchronisation period. It continues to place the addresses in slots zero and one until the new addresses are loaded. To perform no action the interface is given two zero addresses (to force itself to itself). Each connection unit monitors slot zero. If it finds its own talker address present then it transfers the data field from slot one to its listener address register. It acknowledges receipt of the force by returning the new listener address as non-valid data in its talker slot.

The forcer may have a subsidiary role which permits additional integrity checking of the bus cables. If the forcer is placed as the first device on the cable and the lazy scanner placed last on the listener side, complete integrity checks may be performed.

4.8 A Uniform Synchronet

A Synchronet can be arranged to operate in a completely uniform manner without requiring any of the special devices. Certain sacrifices must be made, for instance there can be no logical naming as that would require a special device as was shown in section 1.6. As each station has a unique, fixed talker slot then it can put whatever data it likes in that slot, whenever it likes. This can be used to give a full connection and disconnection scheme. A station can, itself, provide a simple command structure for devices (or users) to use to communicate requests. A sample is shown in figure 4.6. A device wishing to use its station to connect to another simply informs it of the target device address. 56 The station then places this address in the data field in its talker slot, with a request to connect bit set. Unconnected devices monitor the network continually looking for slots which wish to communicate with them. This is a lazy process as many cycles may be taken, if needed, to scan the entire range of slots.

The station requested may accede to the connection request by placing the requesting talker's address in its data field with the connect bit set. The requesting device, which is still continually sending its connection request, monitors the talker slot of the device it is requesting and so sees this connection request immediately. A three phase handshake commits both listener and talker to the connection and resolves any call-establishment collisions. The accepting station will be monitoring the requesting stations talker slot and vice versa so a very simple protocol makes a synchronised connection at the start of a major cycle. If a dual bus were available, similar to that used for the Sonet experiment then a convenient way of doubling the available data rates exists. After a connection were established one of the devices, say the lower numbered unit, would transmit on the first bus channel in its own and the other device's slots. The other device connection unit would use the second bus channel, again using both its own and the other slot for transmission.

D (disconnect) S

Q

Figure 4.6. Sample Commands for Uniform Synchronet

Disconnection is simply handled as either end may send a control frame to force the other end off. Again a phased handshake ensures that both ends are committed to the disconnection. Of course, with this mechanism there is no flexibility as to the allocation of slots and so all slots must be wide enough to cope with the maximum offered data rate. As most RS232c connected devices can support 9.6kbps or 19.2kbps as a maximum, and most users will always chose the fastest speed they can, this presents little loss or difficulty. Note that with this arrangement collision resolution on connection is automatic and receiver controlled, both of which are the desirable characteristics.

4.9 Implementation of a Synchronet

A proposed implementation of an individual Synchronet connection unit using current technology is shown in fig 4.7. Only four integrated circuits are required with only one being a custom device fabricated from a standard gate array process. Single, five volt, power supply is used by selecting an RS232 interface driver which includes the necessary charge pumps to generate the ±10 volts needed for this interface. 57

Figure 4.7 Synchronet Station Hardware

The transceiver circuit (labelled NTU) is directly connected to the unidirectional bus by an indirect Insulation Displacement Connector. This is to ensure that no reflections are caused by impedance mismatching. Stubs of 1-2 cm are easily arranged and quite acceptable as the reflection frequency will be well into the GigaHertz range. For convenience the connection unit could be located at a remote point, but as there are so few components it would seem sensible to have a single small printed circuit card containing all of the components. As all of the circuits, with the exception of the network driver itself, are very low power devices power can be supplied by a low voltage connection from the connected device or from the network cable. The component cost of each connection unit is only approximately £35, thus meeting the requirement for a cost which is small by comparison with the cost of the devices it interconnects. Further implementation details are provided in chapter 6. As the software is broadly similar to that of the Sonet experiment it is not described separately.

4.10 Additional Features of Synchronet

The tables held by the currently active allocater describe the state of all known users, connection points, and links. Many possible conditions may be detected and can be acted upon. Examples are:

A given user may be associated with a connection point and hence have a link made to him. The status of all computers and their ports is monitored so their allocation may be controlled. Third party link instructions, permanent virtual circuits and pre-named closed user group broadcasts can be arranged. Automatic reservations, link establishment at certain times of day, etc. are all possible. 58 Synchronets can be interconnected in a very simple manner using a connection box on each. As buffering is automatic the additional connection hardware is trivial and multiple data links can be provided. A user or device on one network can then request allocation of a connection to a link to another net. For this connection the user or device would be forced to the allocater of the second net for onward routing. Obviously many Synchronets could be connected by using a common 'automatic' net to link between them all, having no facilities other than links to other networks. Provision of extra processing capability in bridges in a non-uniform Synchronet would allow for automatic searching for the required logical name in allocators prior to establishing the necessary physical links. This would make all user connection requests transparent to the intermediate Synchronet.

4.11 Summary of the Proposed System

The proposed Synchronet network uses a shielded or unshielded, dual, twisted-pair cable. The cable is laid to pass every possible connection point. At one end the two pairs are connected to form the loopback. At the other each pair is correctly terminated and the start side has the small bias circuit connected. The start pair also has the first station which also generates the major cycle protocol signal, at the start of each major cycle, by a two cell phase-encoding violating pulse. The end pair connects to the monitor circuit which continuously checks for the major cycle protocol to ensure to give instant warning of any loss of cable integrity. This forms the entire, non-station overhead of the network, and the end circuits are easily made into a single unit.

Each connection unit, as described above, taps onto the cable with its driver connected to the start-loopback pair and its receiver connected to the loopback-end pair. Dynamic bus timing may be used to achieve the most dense data-ffame packing. Short frames (ten bits each) are used as a compromise. This allows for inclusion of a maintenance error check. Control information is carried in separate frames and is only used for connection/disconnection, carrying break conditions and flow control indications under normal circumstances. The same physical hardware may be used with either a scanner/allocater/forcer unit and dynamic standby spares or, if the simplified service is acceptable, with no special stations and operated in a uniform manner. Additional hardware may be included to aid in reconfiguration following a cable break and this is described in chapter eight 59 5.0 THE SONET EXPERIMENT AND HISTORY

The demonstrations of operation of the dense-packed, unidirectional bus proved very successful [Cri79], but the physical data transfer mechanism is only a small, albeit a very important, part of a network. Thus, following the same principles which had led to the Synchronet physical transfer mechanisms, higher levels of protocol and operation were developed for evaluation. To this end the full Simple Open Network was designed and implemented as a test network. It also immediately found a second application in a service environment as commercial networks were not then available to provide the service levels required by the Department of Computing at equivalent cost It should be noted that the copy of the experimental Sonet, used to provide service, was produced outside the College at full commercial cost and it is with this cost that all comparisons were made.

The proposal to use Sonet as a service network in the Department of Computing was made in mid 1982 and the network was installed and commenced live running in January 1983. The network was expanded to include two Sonet 'strings' linked by gateway units. This also allowed experimentation on bridges and gateways. The network was extended to a peak of 120 nodes and was providing a full service at the time of writing in January 1987. Considerations of service provision, availability and the operation of fault detection and recovery are thus taken over this four year period. A node consisting of a network connection unit and its associated network transceiver unit tapped onto the Sonet cable are shown in plate 5.1 with the construction exposed in plate 5.2.

To minimise the complexity of the experiment, one of the design aims of Synchronet, listed in section 4.0, was removed. Sonet was designed only to support the RS232c option D interface, but as more than 97% of the devices in the Department of Computing had this as their communications interface this was considered a minor restriction. As Sonet was to be simple to construct, it was implemented using only available, standard components, so none of the custom integration which would have been needed for Synchronet was included. This gave an additional and significant cost reduction, but imposed some reduction in total bandwidth. Again this was considered acceptable to achieve the demonstration needed. Also, as it was not deemed necessary to repeat the demonstration of unidirectional bus operation, already carried out for Synchronet, and so a simple bus was used which could be driven with available integrated circuits. This reduced the efficiency of transfers by about 7% due to the gaps between frames for bus delays. Obviously the loss of efficiency would have been larger had the simple bus been run at full Synchronet data rates. The drivers and receivers for this are described in sections 6.2 and 6.3.

Sonet in this simple form requires no additional hardware in the devices it connects except for the RS232c interface with any appropriate driver software. LANs which adopt this approach are sometimes somewhat confusingly called 'zero-slot LANs', as no slot from the expansion slots within a computer is needed for a network card. The absence of additional hardware within the connected devices keeps the cost low and simplifies installation. The disadvantage is the limited speed of operation of the traditional (aged) RS232c standard serial interface and its associated processing load on the computer. 60 5.1 Sonet Topology and Data Flows

The simple bus is time divided into slots with a synchronisation period followed by two maintenance slots followed by sixty-four user data slots. A very simple method was devised to double these numbers. An additional twisted pair was included in the bus cable and an extra driver and receiver added to each station. The two pairs were colour coded red (port channel) and green (starboard channel). The drivers and receivers were arranged so that if a station operated its starboard channel driver then its receiver connected to the port channel would be selected and vice versa. When a virtual channel was established between two stations then an adequate number of slots per cycle would be allocated on one of the pairs for one direction of communication. The reverse direction of communication was accomodated by the equivalent slot or slots on the other channel. Each major cycle included four maintenance slots and one hundred and twenty eight user data slots and the total signalling rate on the slow-speed Sonet is one megabaud. These are arbitrary choices but were made to balance the requirement for user communication with that for the establishment of channels and maintenance functions. To simplify the timing system, and reduce the cost of the hardware for slot synchronisation in the stations, the simplest synchronisation distribution method was chosen. An additional twisted pair, the protocol and timing channel (white, PIT) was included with a receiver at each station. This is not the most efficient method of separating slots but again, as the lowest Synchronet level was not part of the Sonet experiment, the lower cost solution was chosen.

i

l i a a a a a n i h im laaaiiaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaiE

NTU Network Bridge Unit \\ ■“! ntu NTU - a t

(iitiiiiiiitiiiiiiin iim iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiitiiiiiiiiM iiii

iiiiiiiiiin iiiiiiiii aaal haaaaaiaaaaaaaaaiaaataaaiaiiaiaaaaaaaaaaaaaaaaaaiaaaaaaaai^Baaaaaai

NTU NTU NTU

NCU

M l ■ connection units a

Figure 5.1 Overall Design of Sonet 6 1 Plate 5.1 Sonet NCU and NTU and Main Cable

Plate 5.2 Sonet Network Connection Unit Construction 62 The synchronisation pulses, a long pulse (40|isec) to mark the start of a major cycle and sixty-seven short pulses (10|isec) to delineate the slots were generated by the read only store driven circuit, shown in figure 6.10, in the currently active Network Server Unit (NSU). Of course as these markers are carried on a separate channel, data frames can travel on.the port and starboard channels during their driven periods and so their pulse width is set to make detection easy and is subject to no other constraint.

As has been argued earlier, a name server is essential in a network if logical station identifiers are to be used to best effect. This function, among many others, is provided by the currently active NSU and is monitored by any other dynamic-standby NSUs. The full layout of a Sonet is shown in figure 5.1 and includes bridging to a second Sonet string. The maintenance slots, the first two slots on each channel after the synchronisation period are used for all communication between the NCUs and the active NSU. The NSU guarantees to transmit in these slots on the starboard channel and listens for a response to its previous cycle's request on the p o rt channel.

The first starboard slot contains a command from the NSU, and the second starboard slot contains the identification number (ID) of the NCU to which the command is sent The NCU which was commanded on the previous cycle responds in this cycle with a 16 bit response placed in the first two p o rt frames. The protocol used for this communication is described in section 5.5 and the data flows and protocol described above are shown in figure 5.2.

PIT Sync 40us n 10us ti­ ll- II-

Figure 5.2 Sonet Data Flows

The Sonet frame contains no addressing, as has been described earlier, and has the minimum of control and error checking information as shown in fig. 5.3.

DV D 0-D 7 (data bits) P/C MB 1 bit 8 bits 1 bit 1 bit 3/4 bit

DV Data valid in frame P/C Frame parity/Control D 0-D 7 eight user data bits FC Frame control Figure 5.3 Sonet Frame Format 63 5.2 Sonet Servers and Central Facilities

Some functions are always needed in a network and are provided by centralised servers. In Sonet these functions are:

• Logical connection requests • Logical service requests • Status and help information • Initialisation of node parameters • Global error detection and recovery • Network management and maintenance

The use of logical names with aliases and generic names applying to numbers of nodes presupposes a more central server to supply the (dynamic) logical to physical mapping. The user of generic logical naming would introduce a large network load if it were to be fully distributed as each node would have to be tested to see if it has the generic name and would be able to accept the traffic. The Sonet servers provide queueing on busy generic names and this would make a distributed server solution completely impractical as a separate queue would have to be maintained in every node bearing the generic name or its alias. This queue would have to contain an entry for every queued device. When a node became available to satisfy the request all other nodes would have to have the corresponding entry removed from their queues. In a centralised name server such as the NSU, however, only one queue and minimal network traffic are required to provide these services.

The status of the entire network or of any individual node should be available for any user in the network. It can be fully distributed but only if the status of nodes which are currently powered-down is not required. Often the most useful information is why a service or connection is unavailable and a note to this effect has to be held centrally. Sonet minimises the network load by the arrangement of information communicated as part of the protocol between NSUs and NCUs. The NSU then maintains a central pool of status information, rather than informing all nodes of every change, and holds notes for nodes which are not available.

If nodes are not continually powered and have no non-volatile store then they will require their default initialisation parameters to be held elsewhere. They could be held in another node but this would require a provision for all parameters of every other node to be held in each node. A centralised server obviously minimises this problem, and the Sonet NSU holds a copy of all current and default parameters for each node as shown in appendix E.

A secure terminal is desirable for operation of the network managers control and maintenance functions. Hardware to assist in global error detection and recovery from failures can also be sited with it, at a central location, probably near main machine operators. Sonet has been designed with a centralised server for these functions. Distributed dynamic standby spares ensure high reliability and availability. In fact the only contention which can ever occur on a Sonet is when the currently active network server unit fails. Then any dynamic standby spare server units compete to decide which one will assume the role. 64 5.3 Interfaces to the Network Control Units

A CALL button is located on the NCU to permit a user to select interaction with the local NCU, temporarily leaving any established channel. The Ring Indicate control line performs a similar function although the protocol for handling RI is slighdy different if the NCU is connected to Data Communication Equipment (DCE). A red/green light emitting diode indicates if the NCU is operable or is failed. If it alternately flashes red and green the NCU can not find any protocol operating on the network. A removable link (or switch) module is fitted internally to each NCU to define the identification number.

Fail -10V Port Receive Stbd

Figure 5.4 NTU to NCU Connection

The interconnection from NTUs to NCUs is made by 15-way D-connectors. The plug on the NCU is shown in figure 5.4, the NTU having a female connector, giving a straight-through lead, or direct rack-mounted connection. The hardware interface to devices is an EIA RS232c option D twentyfive pin DTE (male) connector. Voltages supplied are nominally: -lOv Marking, +10v Spacing. The full range of voltages specified by the standard is accepted. The pin allocations according to the standard are shown in figure 5.5. If a non-standard or signal-deficient device is connected then the missing signals must be 'strapped' to give the correct voltages to the pins. Data Terminal Ready (DTR) is asserted when the NCU has power applied and self-test has been satisfactory, unless one of the user Protocol options for auto-answering or flow control translation are selected.

DTR RI Figure 5.5 Sonet RS232c opt. D Interface

A full 256 character set is transferred transparently over a virtual channel allocated by Sonet if both ends have specified a word format of eight bits. The word format attributes can be set to give a variety of parity and stop-bit options and either seven or eight bit character lengths. The available characteristics for this M ode are listed in figure 5.6. If seven-bit characters are specified at either end then a 128 character set is transferred transparently and parity is added if requested so that 65 devices may interconnect with different specifications (e.g. one odd, one even parity). The only limitations to character set transparency occur when an escape character is requested by the user, or if the logic level reverse-channel flow control protocol is used.

The interface supports bit data rates between 50 and 19,200 bps for asynchronous transfers with a variety of character length, parity and stop bit combinations. The available speeds which are set independently for transmission and reception are: 50,75,110,134.5,150, 300 and 600 bps, 1200,1800, 2000, 2400, 3600,4800,7200,9600 and 19200 bps.

Mode no. Data Bits Parity Stop Bits Mode no Data Bits Parity Stop Bits 7E2 7 Even 2 702 7 Odd 2 7E1 7 Even 1 701 7 Odd 1 8N2 8 None 2 8N1 8 None 1 8E1 8 Even 1 801 8 Odd 1

Figure 5.6 Modes for NCU/Device Communication

5.4 Flow Control and Break Conditions

Flow control is not defined for the RS232c interface and so users will commonly elect to use transparent data transmission over Sonet and perform full duplex end-to-end flow control of their own, using any desired protocol. The only effect of a single Sonet string will be up to four character times extra delay than an equivalent length of wire.

SONET may provide assistance with reverse channel flow control if required. A logic level input at one NCU may have its changes translated into XON and XOFF characters, and these characters may be converted into changes of a logic level output from another NCU. Flow control indication operates in the reverse direction to data transfer and can operate in both directions simultaneously on a full duplex channel. If flow control attributes are used then the character set is reduced to 126 or 254 by the removal of the even parity XON (x'l T) and XOFF (x'93') characters.

If the Flow Control Input Translation (FCIT) attribute, from the set of User Protocols is set on then the V24 interface signal line Ring Indicate pin 22 is mapped into characters. If RI goes from off to on, an XON character is generated and sent across the network, and if RI goes from on to off, an XOFF character is transmitted.

If the Flow Control Output Translation (FCOT) attribute, from the set of User Protocols is set on then characters are mapped into the V24 interface signal line Data Terminal Ready pin 20. If an XON character is received then DTR (pin 20) is set on, and if an XOFF character is received then DTR is set off. The network handles the initial conditions for flow control if FCIT and FCOT are set at opposite ends of a connection, and DTR is set to true if FCIT is not used. 66 Note that Sonet does not take any responsibility for stopping or starting transmission but only transfers the flow control indications so that the source can control its flow. The CCl lT V24 standard contains no specification for reverse channel flow control and hence this use of Data Terminal Ready and Ring Indicate is an extension. This can clash with the use of these pins for providing auto-answering for a modem and so the user is not able to specify illegal combinations of User Protocols. If no User Protocol requiring the Ring Indicate input is specified then it is made available as an input for a remote CALL signal or button.

The flow control translation attributes, auto-answering and control signal indication of connection and disconnection were combined into an attribute which takes a value from 0-7. The permissible combinations of flow control input translation, flow control output translation and ring indicate/data terminal ready control are summarised briefly in figure 5.7.

Protocol State of DTR User Protocol Description Code Default Connect Call RI

0 On Yes Yes Yes No user protocol, DTR is asserted true if the NCU is operable, RI acts as a second CALL button.

1 On Yes Yes Yes PULSE Drop DTR on DISCO. DTR drops for one sec. on disconnection. RI acts as a duplicate CALL button.

2 Off No No No Flow Control Output Translation. Xon & Xoff map into raising & lowering DTR. Initial value is sent to NCU.

3 Off Yes Yes Yes DROP DTR on DISCO. DTR signal is dropped & left down on disconnection. RI acts as a second CALL button. Initial FCOT is sent to NCU but not used.

4 On Yes Yes No Flow Control Input Translation (FCIT).Changes in the RI signal are mapped into Xon and Xoff characters.) RI not available as a second Call button.

5 On Yes Yes No PULSE Drop DTR on DISCO and FCIT. Functions 1 & 4 above are selected but RI not available as a second Call button.

6 Off No No No FCOT and FCIT. Functions 2 and 4 above are selected. Initial FCOT value is sent to NCU and used, RI is not available as a second Call button.

7 Off Yes Yes No DROP DTR on DISCO and FCIT. Functions 3 and 4 above are selected. Initial FCOT value is sent to NCU but not used, RI not available as a second Call button.

Note: CALL, Ring Indicate and Connection assert DTR true unless a protocol with FCIT or FCOT is used. FCIT and FCOT preclude the use of Ring indicate and DTR respectively.

Figure 5.7 Protocols for NCU/Device Communication

Many computers and terminals make use of the break condition on the transmission path between them, for example to abort listing. The break condition indicated by continuous spacing on the data line is transferred transparently over Sonet. If the data signal to Sonet is broken, by pressing the 67 break key on a terminal for instance, the far-end device to which this device is connected receives the same condition. The duration of a break must exceed two character times for it to be reliably transferred, and the duration at the far end may vary slightly from that input, by up to two character times, but this is of no consequence as the duration of any break condition is undefined.

Communication between NCUs and NSUs does not use the break , but if a break condition is detected the current command input buffer is unreliable as spurious character patterns may be generated at the start or end of the condition, and so it is cleared.

As the protocols for NCU to device communication are so intertwined with the interface signals a summary of the use of the EIA RS232c signals made and expected by Sonet follows.

Signal Pin no. Protocol Comments

RTS 4 Any Always asserted true if NCU operable. NSUs have the ability to control any NCUs RTS.Site specific protocols could use this.

CTS 5 Any This signal must be true if an NCU is to communicate. If dropped when a connection is active, a disconnection is forced.

DSR 6 Any This signal is not used by Sonet devices

DCD 8 Any This signal must be asserted true if an NCU is to communicate or connect If it is dropped when a connection is active a disconnection is forced.

DTR 20 0 & 4 DTR is asserted true if NCU is operable. 1 & 5 DTR as above, except on disconnection when DTR goes false for 1 sec. 2 & 6 DTR follows initial state of FCOT on connection, then true on Xon, false on Xoff. 3 DTR is false until CALL, RI, or Connection,when it is set true. It reverts to false on disconnection. 7 DTR as 3 above except not set true on RI.

RI 22 0,1 & 3 RI acts as a duplicate CALL button, and if set true sets DTR true. 2 RI acts as a duplicate CALL button, but does not alter the state of DTR. 4,5,6,7 RI is mapped into an Xon character when it goes true, and an Xoff character when it goes false, and has no other effect

Figure 5.8 Sonet NCU Use of the RS232c Interface Signals

5.5 Establishment and Management of Virtual Channels

A complete summary of how a virtual channel is established in Sonet is made here. As the server was already to perform logical to physical name translation, the allocation and establishment of the physical virtual channel can be done at the same time. The currently active server carries out a dual-level lazy scan of physical device numbers requesting the status of each. As can be seen from the NSU/NCU protocol shown in figure 7.3 and the NCU store arrangement shown in appendix D, the server sends out a request for a response with word-pair <8> specified. For security reasons, only named devices may connect, but for maintenance purposes all IDs are scanned during a slow 68 scan, and their responses are logged in the NSU table. Devices which have responded during a slow scan, a n d have had their default initialisation parameters down-loaded into them, and are named, have their IDs scanned during a fa s t scan. The ratio of slow to fast scans is adjustable but a ratio of one to eight seems about right

If, in response to the NSU fast scan an NCU returns status bytes which indicate a command is ready for processing then the NSU reads the number of bytes specified in the status response. Assuming that the command is Connect, and it parses correctly, then the logical name of the channel destination is searched for in the servers' tables. (There is a 64 byte entry for each NCU in the NSU's tables which contains the logical names and aliases as well as all default and current parameters). If the logical name is found then it is checked to see if it is already busy, and if not whether its software switches and interface control signals are such as to permit connection to it If this fails then the name is checked to see if it is generic in which case further entries of this name are sought If all nodes so named are busy or unable to accept the connection and ih tw a it option was specified then this call is placed in the NSU queue. This queue is checked on each disconnection to see if a channel can then be established. In any case, if the destination can accept the connection then the server checks its slot-allocation table, starting at the last slot, and searches backwards for an adequate group of slots to meet the lower bandwidth requirement of the two devices.

If a channel is required (in the slow experiment Sonet) at 2400 bps or lower then only a single slot per major cycle is needed. If the channel is to be established at 9600 bps then four slots are needed. These would be regularly spaced throughout the major cycle. These slots are then marked in the slot-allocation table and the devices are informed of the base slot num ber, and the number of slots to be used. The requesting NCU is given the starboard channel to transmit on, and the NCU receiving the channel uses the equivalent slots on the port channel. If flow-control translation protocols have been set then the initial conditions for these are also notified to the two NCUs. They are then released, usually after a message informing them of the connection, to make free use of the virtual channel.

The connected NCUs are still part of the fast scan cycle and respond with their status whenever the server asks them. Their status not only contains the fact that they are connected and using a virtual channel, but also the current states of the RS232c control signals. Thus to disconnect the channel either device may place a command in its buffer (and mark this in its status), or drop one of the RS232c control lines, or even be unplugged or powered off. In the latter two cases there are a number of retries, to ensure that the absence of response is not due to noise or a transient fault, before a disconnection is forced.

As can be imagined the NCU and NSU are complex combinations of hardware and software with stringent timing constraints. The full block schematic of the NCU is shown in figure 5.9. Details of the individual circuits are covered in chapter six and the software for both the connection and server units is described in chapter seven. Figure 5.9 Sonet NCU Block Schematic Block NCU Sonet 5.9 Figure

CT\ VO 70 6.0 SYNCHRONET AND SONET HARDWARE DESIGN

All networks must have a connection cost which is low by comparison with the cost of the devices to be connected. For user oriented networks the user's device may be a terminal costing £500 or less, or a microcomputer or workstation costing from £500 to perhaps £5,000 at prices current at the time of writing. Connection costs in excess of a couple of hundred pounds will not be justifiable. This constrains the choice of the media and hence the driver design. The novel driver design for the Synchronet is described below with its associated receiver and the drivers used in the Sonet experiment. It is noted that the resolution of the overlap problem, thus permitting dense packing of data, by the circuit presented here is a significant contributory factor in this work. The design of the salient parts of the controller and server units is also described in this section where such designs represent original work.

6.1 Media and Driver design

The Synchronet design using the looped-back bus is well suited to fibre optic media but currently is only cost-effective if implemented with cheaper, twisted-pair copper cable. The experimental Synchronet and Sonet layouts and the service Sonet were implemented with twisted-pair cables. For Sonet this was Belden 8777 or equivalent cables (Belden 9503, INMAC 1751 etc.). This has individually insulated twisted pairs screened by a wrapped, metal-coated, mylar film. Continuity of this film is maintained by an uninsulated, multi-strand drain wire. It has a characteristic impedance of 55 ohms and a capacitance of less than 180pf/m. Experiments were run on Belden 9730 (INMAC 1731) style cable, which has a group of twisted pairs with a single overall screen and reduces the capacitance to under 150pf/m, and on a variety of other twisted pair configurations including UTP. These experiments were not intended to determine the optimum cable but were primarily intended to demonstrate that the driver designs could cope with a wide range of cables. The result was, however, that the Belden 8777 cable had lower error rates and less noise susceptibility, and also that the drivers performed well on all the tested cables.

For the unidirectional bus of Synchronet, implemented using copper twisted pair cable, the drivers must be capable of driving a signal onto the cable in the presence of an existing signal. If data frames are to be densely packed then one will start immediately following another. Because of the time taken to propagate from one end of the bus to the other, a signal starting near the loop-back point will still be running back towards the near-end termination during the first bits of the next frame if that frame were to originate near the transmitting end. This is shown in figure 6.1. It could be overcome by slot allocation so that slots only occur in the same sequence as the devices occur on the unidirectional bus. This is a very contrived and awkward solution, or rather an avoidance of the problem. It would require a complete reallocation of the slot sequence for each new virtual circuit established in a dynamic allocation regime. It would require a complete reallocation of the slot sequence for each new device attached to the bus if static allocation were used. The problem can only be fully overcome by the design of a driver circuit with the properties given below. 7 1 Data guaranteed in correct slots on the return path

Slot Timing ______Indications [ |______| | ______| | ______| | ______M

Overlap requiring HD (see below)

Figure 6.1 Overlapping of Transmissions in a Synchronet

The requirements for the Synchronet driver are thus: • It must be capable of driving in the presence of another signal, • It must match the cable's characteristic impedance when driving, • It should present minimal load when undriven, and • It should be capable of insertion or removal from a live network.

The driver circuit which resulted from these requirements is shown in figure 6.2 and was named the HD or infinite impedance driver in honour of Adams [Ada79]. As can be seen the test version was a unipolar, unbalanced driver which uses a pulse transformer to give the isolation and impedance characteristics required. This driver operates at rates in excess of 2.5 M baud, limited only by the pulse transformer.

Figure 6.2 Synchronet Driver Design 72 This design does require that a small bias voltage be applied to the line to ensure that the undriven (rest) state is seen as a zero state in the presence of noise. This bias is provided at the ’near-end' (the transmitting end) termination in the experimental version. A fully balanced driver would be more complex, needing twice the number of components, but simply follows the original in design. Slightly higher data rates would then reasonably be expected.

6.2 The Sonet Line-driver

To fit in with the overall specification of the Sonet network a standard integrated circuit driver had to be chosen. This is directly coupled to the cable by an insulation displacement connector (IDC). No standard integrated circuit driver was available at the time to meet the requirements fully, and so the National Semiconductor DS8831/2 was chosen as the nearest fit. This circuit, shown in figure 6.3a, had to be overdriven as it was not designed to source the necessary current for the bussed connection to the low impedance, twisted-pair cable. This relies on some reflections to achieve the necessary drive and so maximum data rates are reduced [see for example Cri87]. Since that time, however, ICs have become available for use with the RS485 standard interface. This has a specification which nearly meets that of Sonet In any case for a fully commercial network, rather than an experimental service demonstration a special purpose driver could have been justified and produced [Rao86].

The DS8831IC includes an overload protection diode not found in the DS8832. Due to supply shortages the 8831 had to be used in some places and it was then necessary to include a separately regulated supply to ensure that no reverse-bias path produced a low impedance (short) across the main cable when the IC is not supplied with power. The additional components for this high- protection circuit are shown in figure 6.3b, but it should be noted that no failures occurred to warrant the higher protection provided.

r- — — - +10 v j 7805 0 C —L. X x*- DS 8831 XT | 470nf 25uF 1N4001 1N4001 b) No Power Protection Circuit

Figure 6.3 Sonet Driver and High-protection Additional Circuits

Each driver pair in the DS8832 provides a fully balanced pair of outputs which may both be disabled into a high impedance (non-loading) state. Experiments were also performed with a parallel driven pair of drivers to reduce the reliance on reflections to provide the drive current and to achieve graceful degradation in the event of any open-circuit driver failure. It also would reduce the current 73 load of the drivers and thus render them less susceptible to failure. Only one driver failed in the measured, four-year, service period and, protected in this way, the reduced driven current was detectable and the circuit was isolated and replaced without any loss of service.

6.3 The Synchronet and Sonet Receiver Designs

The Sonet receiver circuit was adapted from the original Synchronet design, and was given a 12KQ input impedance to permit two hundred and fifty-six receivers to coexist on a string of 55Q cable one kilometre long as shown in figure 6.4. At the heart of each receiver is a comparator (LM311) to permit reliable, differential detection of heavily attenuated, balanced signals. The input circuit provides the designed load impedance, removes any DC bias (under normal operating conditions there should not be any) and restricts the voltage swing to the comparator to prevent input saturation. The receivers were tested with a full complement of transceiver cards distributed across the full length of cable. They were tested with additional resistive and capacitive loads, and with a local crowded area containing thirty-two receivers within five metres of cable. No configuration was found to give any detectable increase in overall error rate for any station except for the addition of very high capacitive loads.

6.4 Sonet Timing and Data-transfer Circuits

Four main synchronisation mechanisms operate in the Sonet system. Major message transfer between units is handled by buffers and semaphores located in each NCU. The semaphores are set and cleared locally by the NCUs and are remotely set and cleared by the transmission of NSU commands. The commencement of each major cycle (of maintenance and communication slots) is detected in the three-channel Sonet by a wide pulse (40 ps) on the Protocol and Timing channel. This (TL') is actually detected after 15 ± 2ps in the version constructed, so the pulse duration could be halved. The commencement of each major cycle would be detected by a phase-encoding violation of more than two bit-cell times (8ps) in a two-channel Sonet. Each communication slot is marked in the three-channel Sonet by a narrow pulse (10 ps) on the Protocol and Timing channel (TP'). This would be handled by a local, crystal-controlled oscillator and counter in a two-channel system. The Sonet protocol/timing channel circuit is shown in figure 6.5. 74

Figure 6.5 Sonet Protocol and Timing Circuits

Data is transferred over a Sonet by a modified, phase-encoding system. Data to be output is transferred by the relevant microprocessor into its network Asynchronous Communications Interface Adapter (6850). The ACIA is clocked on a bit cycle basis from the master bit time circuit. This, like the phase-encoding circuit, is a model of simplicity as shown in figure 6.6. Both circuits were designed by minimal, finite-state machine techniques modified to suit commercially available chips. This embedded clock (phase-encoded) system is such that there will always be a clock transition at the start of each bit-cell. The state of the channel during the latter half of the bit-cell determines the bit (zero or one) to be transferred. Any additional transitions required to achieve this are inserted in the centre of the bit-cell time. This mechanism restricts the range of transmitted pulse frequencies to (f <-> 2f) which reduces the required bandwidth and generated noise. The detection process to recover the embedded clock and read the data uses a simple monostable circuit shown in figure 6.7.

Figure 6.6 Phase Encoding and Master Bit Timing Circuits 75 It is necessary to ensure that the correct number of clock edges are generated regardless of the state of the data which emerges from the 6850 ACIA in non-return to zero (NRZ) format. This requirement is combined with a frame-limitation circuit which ensures that the line drivers are only enabled during the correct transmission time.

While this part of the hardware and its associated software was undergoing prototype test an undocumented design fault in the 6850 was discovered. The frame-limitation circuit was set to give the designed eleven bit-cell frame and the phase encoder was set to insert eleven clock edges at the start of the cells, and intermediate edges in the middle of cells as determined by the data patterns. Using the 6850 in its so-called synchronous mode with one clock edge per data bit to be read in, the circuit would only operate if run continuously. If, as was the design requirement, the data was transmitted in frame-length bursts the 6850 failed to indicate that frames had been received after they had been clocked into its internal shift register.

After further tests it transpired that the 6850 requires an extra clock transition after the data frame has been read into it, to cause the transfer of the data from the shift register to the read-data holding register. Thus an eleven bit frame requires twelve clock edges. As the 6850 is an industry standard integrated circuit it was most surprising to find such a frustrating, undocumented fault When run continuously, the first edge of a second frame was performing double duty as the last (extra) edge of the previous frame and the first edge of this frame, which is why the circuit worked at all.

To overcome this, the data-clock and frame limitation circuits were altered to permit twelve clock edges and eleven and a half bit-cell times respectively. The convenience of the master clock division circuit design was appreciated at this point However, after a clock edge, the drivers would be in the 'one' state and disabling them would result in a long 'discharge' period for the lines to reach their rest state. Thus as well as the extra clock transition at the start of the bit-cell, an extra intermediate, half-cell transition is needed to drive the lines into their rest state before disabling the drivers. Thus the final circuit takes eleven data bits, twelve clock edges to start bit-cells, and such intermediate edges as are necessary to give the correct states of data, and the correct end rest state. 76

The whole frame is encapsulated by a frame limitation of eleven-and-three-quarter bit-cell times during which the line drivers are enabled. The frame limitation and timing circuit is shown in figure 6.8. It should, however, be noted that, had the requirement for partial bit-times been known earlier, a higher modulus counter would have been used to give a simpler circuit A complete set of waveforms, in figure 6.9, shows the entire transmission and reception process. Obviously only one pair, port or starboard, of drivers are enabled for a given transmission, the other pair remaining disabled throughout.

Data Start 0 0 1 1 1 F F 0 1 0 1 1

LH LTU

ENB 1 _

Channel

RxC JLJT_#L_fl_fl_fl_fl_fl_ Data S001 10 101 FF Figure 6.9 Transmission and Reception Waveforms

6.5 Sonet Network Server Unit's Circuits

The Sonet server is a standard NCU/NTU pair with various additions. Firstly the microprocessor (6802) and 4K byte EPROM are removed. A forty-way header takes the 6802 connections to a piggy-back board where the entire NCU/NTU appears as store mapped peripherals to the main 6809 77 processor. The NCU takes up the bottom half of the store map. On a standard NCU the 4K byte EPROM occupies the entire top half of the store map. On the 6809-based NSU the top half of the store is populated by a 16K byte EPROM to hold the server code, 10K bytes of battery-supported RAM to hold the node tables and local server variables, and a programmable interface adapter chip. The PIA provides monitoring connections and control of the ROM-based protocol generator circuit shown in figure 6.10. This circuit connects to the main cable by a network transceiver unit which is modified by moving the starboard channel driver to drive the protocol channel and disconnecting the port channel driver. The three receiver circuits are left for monitoring and self-testing. A card with these links connected is called a network protocol unit (NPU). There are a large range of optional extras which may be included on the NSU card, and those listed below have been designed and tested. For example a clock/calendar chip, also connected to the battery support circuit shown in figure 6.11, gives the NSU continuous access to the correct time. Simple four bit counters can be added to the NPU monitor inputs from the port, starboard and protocol channels to allow more comprehensive, automatic checking of signals and their timing. Extra visual and audible warning devices may be added to alert operators to unexpected conditions.

Figure 6.10 Rom Circuit for Timing Generation

Protocol generation is handled by a ROM-based pattern generator circuit. A sixteen-bit counter with twelve outputs connected to the address lines of a 4Kbyte EPROM is fed by the 1MHz clock signal (E) output by the 6809 microprocessor. Only four bits from each byte of the 2532 EPROM are actually used. One provides the 0/1 data signal to the protocol driver chip in the NPU. A second provides the Enable driver signal to the same chip. The third output provides a pattern to trigger a logic analyser to monitor parts of the cycle of particular interest for test purposes. The whole circuit is controlled by the fourth output which, when true, resets the counter and so restarts the pattern. The pattern used for the Sonet experiment is defined by the contents of the EPROM which are shown 78 in appendix B. A monitoring circuit is included so that a failed NSU can be automatically disabled. The 6809 processor must p rod the monitoring circuit and, if it fails to do so for more than three successive major cycles, a disabling interrupt is generated.

Figure 6.11 Battery Backup for RAMs

The battery support circuit shown in figure 6.11 both supplies power for, and connects the power valid signal to the enable input of, the HCT138 address decoder. This decoder provides chip select signals to the RAMs, EPROM and PIA. This guarantees that in the event of a power failure not only does the trickle charged battery take over supplying power but also the RAMs are disabled and their contents protected. 79 7.0 THE SONET SYSTEM: SOFTWARE

The Sonet software is split into two main functional units for simplicity of operation and ease of update. All parts expected to need change or update are located in the Network Server Units (NSUs), whilst those parts of software in the NCUs are regarded as/Irmware. This choice has been justified by the update releases of software for the devices. Only one release of the NCU EPROM has been needed since the initial release in 1982. During the same four-year period eleven releases of the NSU EPROM were made to correct faults and incorporate modifications to the User's requirements. Because of the division of function, many of these releases were made without the users being aware of the change. The NSU software is split into 28 modules, with all utility routines being located in five of them. Two files are included into these modules to provide definitions of the NCU constants, NCU stored messages and all NSU constants used. The main structure of the NSU code is shown in figure 7.1. Iam indebted to my colleague Mike Reeve who produced the initial versions of a large number of these modules, though later releases were produced by me and so any errors are mine. The NSU software is fairly traditional in design and was initially written in PASCAL before being translated into Motorola 6809 assembler for optimisation. It is held in a 16K byte EPROM with 10K bytes of battery supported RAM as shown in chapter six. The NCU software is split into four modules, held in a 4K byte EPROM and using only the 128 bytes of RAM internal to the 6802. I coded it directly in Motorola 6802 assembler because it is all so time critical in operation. Its overall functional structure is shown in figure 7.2.

7.1 Network Server Unit Software

On power-up the NSU software performs a primary self test function similar to that of the NCUs which is described in detail in appendix C. The NSU then checks to see if another NSU is active by listening to and checking the validity of the protocol and the maintenance slots. If another NSU is active then the startup NSU becomes an active standby unit and monitors all activity of the active NSU. It does this so as to maintain its own tables in as near a consistent form as it can. The standby NSU will respond to all commands input from the network Manager's console but will not take any action to transmit on the network. Thus it can be set up with a new combination of names and parameters, while not active, and then be made to take over as an active server. It can be used as a monitoring device without a network Manager's console having to load the active server.

Detailed consideration has been given to adding a protocol to permit a standby server the ability to inform the active server of its presence and request copies of its current tables. Though this is not difficult it would place a load on the active server as the tables are quite extensive as shown in appendix E. This protocol has not been installed or tested yet

If the server, after its power-up checks, believes that there is no active server then it attempts to become the active server. This is the only possible place where contention could occur in a Sonet An example would be if a power failure caused an entire network to restart when two or more servers would power-up at the same time. 80 In the experimental Sonet there are eleven commands which may be input to a network manager's console on an NSU. These are: connect, disconnect, set, query, attach, detach, test, por, clear, restart and shutdown, with the minimum acceptable abbreviation shown bold.

NSU Power up s e l f t e s t MESSAGES (8M) | ~ n MAIN LOOP (8 0 )T r r ^ FIRQ (30) Handler

Process NCU Command Process NSU Command n ------1 PROMPT (6B) _ EXECUTE •PROMPT (7B) NOT IN YET (6Z DISCO (61) -----ATTACH (70) CONNECT (60)------DETACH (71) SET (6 2 )------NSU SET (72) QUERY (63)------■NSU QUERY (73) POR (64)------— SHUTDOWN (74) ------RESTART (75) PARSE DISCO (61p) ------TEST (76) Note: SLOT MAP routines (20) ------CLEAR (78) TOKEN routines (40) NODE TABLE routines (50) Figure 7.1 NSU Software Structure

T est forces a node to restart its self-test mode to repeat all primary self test operations, including reloading all defaults. P o r forces a node to execute a power-on-reset load of all default settings held by an NSU as though the NCU had just been powered on, but without setting it to a failed mode or performing any tests. C lear clears the error log of a node. Appendix D shows details of error logging, but errors which can be logged include network errors, rejected virtual channel, protocol failure and user interface errors such as lost echo character, parity and framing errors. R estart clears the specified node(s) tables of names and current settings and can optionally restart the network with no connections. Shutdow n closes the NSU as an operating manager so that a graceful power down can be made, for example to move the NSU or for a more general, but expected cessation of power, or to permit another NSU which has a different set of parameters to take over as the active NSU.

The connect command establishes a connection between two devices immediately or as a permanent virtual circuit (PVC) which is established whenever both devices are powered on and ready to connect Permanent-virtual-circuit connected devices can disconnect themselves in the normal way and can return to the PVC by executing a default start using their reset command. The disconnect command breaks a permanent virtual circuit connection. It can also be used with a protection mechanism to force off an ordinary connection. The form of these commands is shown below: CO N N ECT [TO] [ (PERManent] DISCONNECT 81 The Sonet NCUs have identifiers (physical numbers) defined by link modules or switches installed in each. These numbers bear no relationship to the purpose for which an NCU is used and so logical names are used. These will commonly be derived from a computer's name, the location of a printer or the name of a person or room. A generic name may be given to a collection of NCUs and a group of generic names may also be allocated a generic name. For instance, the 9600bps ports on an IBM system might be given the generic name IB M 9600 and those at 2400 bps, IB M 2400. Furthermore, both might be allocated the generic name IBM. A request to connect to IB M 9600 would scan the 9600bps ports until a free one was found. A request for IBM would scan ports in both sub-groups.

To provide the logical names to a server, so that it can perform the logical/physical name translation, two commands are used. A ttach associates a name or names with a node or nodes, installing the device into the polling mechanism so that it can make connection requests. D etach removes a name from a node or service and from the polling mechanism. A TTACH I ] DETACH () I < G eneric Name> ()

The network manager may modify any default or current setting of the characteristics of NCUs using the set command. This is particularly useful for initial settings and for subsequently changing the settings of devices, such as printers, which do not have the ability to perform their own changes. Certain security features can only be changed from the network manager's console. The change option may be set to permit a user (NCU) to change its own default settings, otherwise only the network manager can change them. The command format is shown below: S E T Time DD/MM/YY HH:MM:SS | Note I["("(DEFault|CURrent)] 1 Speeds 1 RX 1 TX 1 MODE 1 PROTOcol (control of dtr RTS etc) 1 C onnect (ON | OFF) 1 Echo (ON I OFF) 1 H elp (ON | OFF) 1 Prom pt (ON | OFF) 1 W arning (ON | OFF) 1 CHange (ON | OFF) 1 ESCape (ON [ »='»] |OFF| n=") 1 PASSword (ON [] | OFF | ) 1 TRAnsfer (ON [n="] | OFF | n=") 1 LOG (ON | OFF) (only allowed on NSU station IDs)

The manager may query the status of the entire network, generic subsets of it or individual nodes of the network. Individual characteristics or the complete group of settings may be queried. In addition the N ote left by any node may be viewed. A maintenance query is included which permits the NSU to view any location of the store of any NCU. Details of the maintenance form of query are shown in appendices D and E, but the normal query command has the form shown overleaf: 82 Q U ER Y ( | cU nique Name> | < G eneric Name>) | Settings ["(" (Current | Default)] | ["(" (Current | Default)] | | Status | Note | NETwork | Time

Typical of the wide range of responses available from the commands above are:

M92E Unknown command M97E Invalid node

M71I Node self testing M72S Node connected or not responding

M73I Node defaults loaded M74I Node error flags cleared

Mill connected to M15E Already connected to

M17S Called node unavailable M19S Network full

7.2 Network Communication Unit Software

The NCU code includes some interesting (or curious) design choices. One major example is that subroutine calls on most microprocessors (the 6802 is no exception) are very slow due to the saving of registers which may be required by the routine. Descriptions of much faster mechanisms for subroutine calls, which would have overcome this problem, are to be found in [Cri87]. As these are not commonly available the consequence was that no subroutines are used in the NCU code. All similar code groups were re-produced by macro-expansion. This trade-off, using more store to gain speed, is one of many used in the NCU which enables it to achieve such outstanding performance from a simple 6802. As it turned out there were never more than eight parallel threads (i.e. a subroutine could only have been called with eight different sets of parameters) so the luxury of subroutines was not missed!

The NCU should not be considered an example of how to write stylish code but it is considered highly satisfactory in view of the stringent time requirements. The 'threads' in the Communicate section have to be timed to within five microseconds throughout, and even less in places.

A simple user interface is provided for both users and devices. There are five commands: connect, disconnect, set, query and reset (power on reset), their abbreviations being shown bold, which may be input to NCUs following receipt of a system prompt of the form:

S91R SONET (xxx) ? where xxx is a manager supplied name for the network.

An additional command memory remains from earlier versions of the NSU. All responses to commands commence with Linefeed and CarriageReturn followed by a coded part "Sxyz" to identify the response to a computer and end with Carriage Return. So that intelligent devices and machines may uniquely read lines and then decode the first three characters of each response, the final response of any group ends with B ell and CarriageReturn. Simple editing of command entry is 83 provided by Backspace, R ubout or D e le te . Commands are terminated by CarriageReturn. The User escape character, C all button or the Ringlndicate signal cause the current input to be ignored and the NCU to reprompt for input. The first character of the coded part is S for NCUs M for NSUs and B for network bridge stations (NBUs). The 'xy' are decimal digits which uniquely identify each response and 'z' is the message type.

For example x= l denotes a response from the connect command; 2 a response from the disconnect command; 3 a response from the set command; 4 a response from the query command; 7 a response due to the help option being selected; 8 a message due to a rem ote action by the NSU and 9 denotes a general response from Sonet.

Figure 7.2 NCU Software Structure

The C onnect command establishes a connection to another device or restores a connection which was interrupted via the CALL button, Ring Indicate line or escape sequence. It has the form: 84 CONNECT [[]["(" (Wait | Nowait)] ]

The W ait option requests that if the specified NCU is busy or unavailable for any other reason, the command does not finish until the device becomes free and the connection can be made. The wait can be terminated by exiting to the command interpreter via one of the three mechanisms above. A command such as P hone could be used instead of connect to give a minimum bandwidth channel for single finger typists to chat through high speed terminals!

The disconnect command breaks the current connection and releases any allocated channel space. The S et and Q uery commands are similar to those for NSUs and permit users or connected devices to change any parameters of their interface and to view their own status and parameters, or those of other nodes to which they are connected. Early releases of the NSU included a command to permit observation of any location in any node's store for maintenance purposes: MEMORY

The reset command is included to permit a node to restart its own power-on self-test procedures and to reload its default parameter settings.

Typical of the wide range of responses available from the commands above are:

Sill Connected to S12W Connected to by

S14I Waiting to connect to S18S Called node busy

S92E Unknown command S16E Invalid Password

S94E Invalid operand S71I Try Connect/DISCO/Query/Set

The characteristics of the users hardware interface are managed by a set of software controlled settings. These are down-loaded to an NCU from an active NSU on powerup. The characteristics of the software interface are also managed by a set of software controlled switches, some of which are down-loaded on powerup the rest being held in the NSUs.

SPEEDS Controls the transmit and receive speeds of the RS232C port.

MODE Controls the number of bits, stop bits and parity for user's characters.

ESCAPE Controls whether transparent or non-transparent communication is required and if non-transparent sets the escape character.The command interpreter is entered whenever the NCU receives the specified code.

PASSWORD Controls whether connection requests must specify the appropriate password prior to acceptance.

CO N N ECT Controls whether a device can be connected to by command. If connect is off the device cannot be connected to (only from). 85

PROMPT Controls whether the NCU powers up in command mode and prompts for user input or in idle mode waiting for the user to request prompting for input If it is not set then the NCU goes into idle mode after completing its self test and waits for CALL, RI or user ESCAPE (if set) and then output a prompt for user input

ECHO Causes any characters received from the RS232c port to be echoed back. As a transparent full duplex path is provided after connection establishment, ECHO has no subsequent effect, echo being the responsibility of the end devices.

WARNING Controls whether the device is to receive an indication when it is the object of successful Connect or Disconnect commands.

HELP Controls whether, following an error message due to faulty syntax the user is given information to show the correct syntax for the command just used.

7.3 Communication Between NSU and NCUs

As the Sonet software is split into the two sections located in the NCUs and the NSUs, reliable communication between the two halves is essential. Communication from the currently active NSU is arranged by the NSU transmitting commands to modify specific parameters of the chosen NCU or by loading the NCU's output buffer with messages. A semaphore prevents the NCU using its output buffer whilst it is being loaded by the NSU. The semaphore is transmitted to the NCU to pass over control of the buffer. To communicate in the reverse direction the NSU can read the contents of any location in the alterable store of an NCU. A semaphore prevents the NSU from acting on the contents of the NCU input buffer whilst it is being filled. The NCU sets the semaphore when the buffer is complete which causes the hand-over to the NSU. The NSU has a continuous monitoring function to aid reliability and maintainability. As it can instruct the NCU to respond with the contents of any alterable location, it can observe the set state of the semaphore and read the contents of the NCU input buffer, and hence act on its commands. The NSU can also check on the settings of the parameters it alters and ensure safe use of the output buffer. Thus to transfer a buffer from an NCU to an NSU the NCU sets its semaphore indicating that the input buffer is ready for the NSU. The NSU in its regular scan of the NCUs' status fields takes control of the semaphore, transfers the buffer and then transmits a clear semaphore signal to the NCU.

To transfer a buffer from NSU to NCU, the NSU checks the NCU output semaphore for exclusive access, transfers the new buffer contents as a mixed sequence of coded and unencoded data and transmits a coded 'set semaphore' to lock out NSU access. The NCU uses the buffer contents, outputting any messages to the device connected to it, and on completion clears the semaphore directly. The NSU in its regular scan monitors the semaphore to ensure it does not access the buffer until it is released by the NCU. 86 A comprehensive yet simple protocol, shown in figure 7.3, permits the exchange of all necessary data and control between NSU and NCU. The Server sends two frames as described previously, the second of which contains the seven-bit node identifier (ID). The eighth bit of data in that frame determines whether the Command frame contains clear text, a coded message, flag settings to be held by the NCU, or a command to configure the NCU, slot number and establish or disconnect a virtual channel. Finally the NSU can demand a response from the NCU with a specified word-pair from the NCU store. With the exception of the demanded response, the NCU addressed by the NSU confirms receipt of the frames by echoing the frames back in the following maintenance cycle on the NCU (port) channel. If a command is rejected for any reason then the NCU echoes it back with the eighth bit of data in the identifier (ID) frame inverted. A device failing to echoback, after retries, is deemed to have failed or be powered off and is ignored until it successfully completes the power-on parameter download procedure.

i l l CO- C l m Hi (I ID0-ID6-C8 ill!

CO Cl C2 C3 C4 C 5 C6 C7 C8 CO C8

Clear Text 0 0 Configure Virtual Channel 0 0 1 Coded Message 0 1 0 Configure Slot Number 0 1 1

NSU flags 1 1 0 Configure NCU 1 0 1 RESPOND but 000000 = DISCO~| 1 1 1

Command RTS/DTR 0 0 Config VC ESC ECHO Protocol 0 0 Conlig NCU ESCAPE char low 0 1 Transmit SPEED 0 1 ESCAPE char high 1 0 Receive SPEED 1 0

SLOTS Port/Stbd FCOT|1 1 Protocol MODE 1 1

Figure 7.3 NCU/NSU Communication Protocol

The currently active NSU may send m essages to any NCU to be output to its connected device. From the main protocol definitions it can be seen that plain text is sent as 7-bit ASCII characters with the eighth bit zero. Transmission of regularly used messages and words is made faster by storing them in the NCU's ROM and sending just a code. Coded messages are sent as six bit offsets into the coded message table held in each NCU read only store. This combination allows all commonly used sequences of characters to be given to a connected device by only a single protocol transfer, but also permits any message sequence to be sent. A very efficient tail-compression algorithm was devised to allow messages which contained common endings to be stored in less space. As many messages also contain the same standard prefix this is also compressed. A code fragment which handles this is shown in figure 7.4. 87

* Previously semaphore has been set to take control of the Output Buffer * w i t h Output_Count characters in it and Index set to zero

Next_Buffer_Char DEC Output Count ★ while not end of buffer BMI Finished_Output LDX Index INX STX Index LDA Output_Buffer,X ★ get char from buffer BMI Coded_Message

* give normal text character to user (checking user interface etc) BRA Next_Buffer_Char * Coded_Message ANDA #03FH * max. 64 messages TAB ASLA ABA * each table entry 3 bytes LDB Message Table Hi * 16 bit table address ADDA Message Table Lo * added by 8 bit machine ADCB #0 STB Temporary_Address STA Temporary_Address L° LDX Temporary Address LDB 0,X * get message length STB Message Length LDX 1,X * Address of message text

Next_Message Char DEC Message_Length * while not end of message BMI Next Buffer Char Tail_Expansion LDA 0,X * get char from message BPL Normal Character LDX 0,X * link address to tail BRA Tail_Expansion Normal Character INX

* give coded, expanded text character to user (checking user interface etc) BRA Next_Message_Char Finished_Output

Figure 7.4 Coded Message Expansion Code

7.4 Self Testing Facilities in Sonet

Sonet NCU's perform a series of self tests prior to enabling themselves onto the network cable, to attempt to ensure that 'rogue' devices do not corrupt the network. Subsequently when enabled further tests are performed interleaving with regular operation. Failure of any of these then switches the NCU to the failed state, disabling its transmit drivers. To make fault determination simpler, syndrome patterns are output onto the least significant 3 bits of the PIA 'A' outputs and thence to a test socket. The primary self tests are conducted with the NCU disabled and the front panel LED showing red. The original primary self test mechanism is shown in greater detail in appendix C. The simplified syndromes are associated with the following groups of tests. 88 <7> Preliminary tests: CPU operations, read only store test code checks, PIA addressability. <6> Checksum of whole read only store. <5> Read, write and pattern change tests on read/write store. <4> Network communication chip (ACIA) is failed if DCD=0 or CTS=1 or the transmit buffer is indicated as clear. All are prevented when the NCU is in its disabled state. <3> Enable protection and frame limiting circuit output, ie ENBL is low. <2> The seven bit NCU identifier number is read in with its parity which must be odd. The incorrect ID is displayed on the baud rate outputs.

This 'fault' can be corrected and the front panel call button pressed to restart the tests. From this point it is assumed that the NCU is basically sound and the network protocol is tested and timed. The syndrome for this failure is given with the front panel LED flashing alternately red and green at about 0.5 Hz.

<1> Protocol failure: the protocol synchronisation signal has failed to rise (or is stuck up) or is repeating with incorrect timing. This commonly happens when the NCU is not connected to the network! <0> Once the protocol has been detected correctly the NCU is set to request its default parameters to be loaded with a RESet command and the NCU is enabled with its front panel LED showing green.

Once the NCU is operational then it runs, monitoring various conditions. If a serious failure occurs then the NCU disables itself, the front panel LED reverts to red and the syndrome shows the cause.

<3> Enable protection or frame limiting circuitry has failed. <6> A spurious interrupt or software interrupt has occurred. Software interrupt traps (SWIs) are scattered between sections of code to trap any excursion from the correct code sequences.

When any network server unit is powered on it performs all the same basic tests as an NCU up to and including the syndrome two tests. The NSU has more read/write store and this may contain tables which are being retained by the battery support circuitry. Thus only a basic part is tested initially and the other RAM chips are tested with their contents stored and replaced. A command is also available to allow for testing at preset times. Once the NSU is deemed to be sound then it listens for network traffic and protocol to see if there is already another active NSU. If this is the case then this NSU simply monitors the network and checks the active one. If no protocol is detected (or if it subsequently fails) then the NSU attempts to become the active NSU. The sequence of actions is as below.

Whilst listening to the data channels for at least one full cycle without reception, the putative NSU checks its own synchronisation protocol generator by timing the signals from it on the internal feedback path. It has still not enabled transmission onto the network. If these checks are satisfactory the NSU transmits its own identifying number (ID) onto the data channels, without any 89 synchronisation transmissions, checking that it both receives its own transmissions and does not receive any others. Sending its ID commences with a T start character on the starboard channel and the 'bits' of the ID (seven bits, least significant bit first) are sent by T being a character sent on the starboard channel and 'O' being a character sent on the port channel. As the NSU has receivers on its normal NTU connection on the opposite channel to its transmitters and extra receivers on its extra NTU card on each channel then there is a positive check that data characters are transmitted onto and received off the network cable. This confirms correct operation of the receivers and so the protocol ensures that if another NSU is also trying to become the active NSU that its ID, being different, will ensure that a character is received by the prim ary receiver, on the opposite channel to one sent by this NSU.

If this occurs the NSU's back off for (127-ID) cycles and retry so that the highest numbered ID will restart this sequence first and complete satisfactorily before the next It then becomes the active NSU by enabling its own protocol synchronisation signal onto the network and transmitting the two frame NSU command and ID in its slots.

The active NSU guarantees to transmit the protocol synchronisation signals and some data in every pair of NSU frames (even if only <1 am ServerxNSU ID> for SONET or for Synchronet) and these are checked by all NCU's and standby spare NSU’s.

Network manager units are able to control the vast majority of functions of the individual connection units and, in particular, can query and clear the NCU's log areas and command individual NCU's to perform all their self test functions. Naturally extensive test procedures, and special purpose test hardware rigs, are needed to check out all units before they are connected to a live network. Many were produced for the Sonet network and, as an example, the procedure adopted for testing a network transceiver unit is shown in appendix F.

7.5 Network Protocol Generation

The protocol signals in the experimental Sonet are carried on the third (white) twisted-pair. The length and pattern of each major cycle is produced by a ROM pattern generator located in the currently active server unit The master clock (E = lfis) is used to step an address register which steps through the patterns stored in a 4K byte programmable read only store chip. The circuit is shown in figure 6.10 and three columns of the store are used to provide the driver signal, the driver enable signal and a restart signal, to reset the address to zero and repeat the pattern. This permits any pattern of protocol signals up to 4096}is long to be generated and the pattern used for Sonet is shown in appendix B. It consists of 64 slots each composed of periods of 10|is driven high, 5|is driven low and 44ps disabled. The maintenance slots, which have sloppy timing, take 255|is for synchronisation, NSU communication and resetting all NCUs. 90 8.0 EVALUATION OF SYNCHRONET AND SONET

A number of performance indicators and metrics can be used to determine the success of the proposals made here and the Synchronet and Sonet experiments. The same efficiency analysis as was applied to international standard networks in chapter two has been used to evaluate both Synchronet and Sonet. In addition it was quite simple to measure the efficiency of Sonet in service because of its deterministic behaviour, and this confirmed the analysis. The ease with which Sonet could be installed, both into a difficult building and into the systems it linked has been confirmed.

A particular feature of Sonet was the attention paid to fault monitoring, detection and correction. The methods used and the results of their use are reported here. They show that quite simple techniques can yield significant improvements in availability. The excellent efficiency and scalable performance which can be achieved by the unidirectional bus is further enhanced by the ease with which faults can be located. The ability of the network to self-repair or reconfigure to an operational system, albeit with reduced performance is another noteworthy feature.

8.1 Efficiency of Synchronet

A Synchronet transfers data from a station when that station's slot(s) time arrives. This means that in the best case the slot time occurs just as the station wishes to transfer, and in the worst case all stations have fair turns and a maximum delay of one major cycle. For the best worst case the protocol efficiency can be easily computed, but depends on the number of maintenance and communications slots and to give a similar comparison with other networks, 256 communications slots have been assumed. Such a major cycle consists of a synchronisation, two maintenance slots with extra round-trip delays and 256 densely packed slots. 256.F Tip ------2iT"d------....Synchronet (a) (258.F + — 2l21 + 4) n v n For a given Synchronet the protocol efficiency is constant. As demonstrated in 1979 with 2.5Mbps data rate, 2km length and vn = 0.65C the protocol efficiency was 97.3%. With the same parameters as the token-passing ring i.e. lKm length operating at 4Mbps and vn = 0.7C, it would be 97.7%. The formula and figures quoted above assume that the dynamic slot-timing mechanism described in section 4.2 is being employed. If the bus length and station position were fixed, or if the second channel slot-timing distribution method were used, then stations could have their relative timing fixed and sloppy maintenance slots would not be required. Under those circumstances the protocol efficiency would improve to over 99% as: 256.F Tip = (258.Fn + 4) .... Synchronet (b)

As a Synchronet can densely pack its data slots then all other frame overheads are included in the protocol efficiency and so the frame efficiency is just the ratio of user data to overall frame length. In the prototype Synchronet the frame was as shown in figure 8.1. 91 D/C DV/I D 0-D 7 (data bits) HI 1 bit 1 bit 8 bits 1 bit

D/C Data or Control in frame DV/I Data parity in frame or Connect/Disconnect/forward flow control DO - D7 eight user data bits or control frame D8 Latched state for break/sync (reverse flow control) _n_0_n_n_n__«_ Synch. ' • H H I I I 1 \- ll— data slots

Figure 8.1 Synchronet Frame Format and Timing

It is possible to reduce the frame length by one bit by having all flow control functions carried in separate frames when data is not available for transfer, thus giving a potential frame efficiency of 80% or more. If data were always available for transfer then this approach might not improve the overall efficiency. Thus keeping to the standard approach the frame efficiency is constant at: U . 8 T |f = = — = 72.7% ....Synchronet 1 n Thus the demonstrated Synchronet performed with a constant overall efficiency of T| T = 70.76% and a Synchronet is always more efficient than either Ethernet or a Token-passing ring for packets containing a full line of data or less. An Ethernet is not more efficient until all packets contain more than half a kilobyte. Synchronet is always more efficient than a Cambridge ring.

By using the reduced, ten-bit frame, dynamic slot timing and associated mechanisms as described, a Synchronet can be made to perform with a constant overall efficiency of Tj T = 79.2%.

8.2 Efficiency of the Experimental Sonet

The analysis which follows is of the Sonet network as was used for the service experiments. This network was deliberately run with additional safety factors built in for experimental purposes. It also operated at reduced rates to permit implementation with standard components and thus to reduce the cost. The factors described below lower the efficiency from that which would have been possible with a application specific integrated circuit (ASIC).

A bidirectional bus was used as described in chapter five. This added round-trip delays to every slot and not just to the maintenance slots. This is reflected in the reduction in protocol efficiency of about 7%. Due to the design fault discovered in the M6850 Asynchronous Communications Interface 92 Adapter chip used in the experimental hardware, each slot had to be increased by 3/4 of a cell time (about 6%). This is reflected in the reduced frame efficiency as was discussed in section 6.4. A separate twisted pair was used to carry the major cycle synchronisation. Additional pulses were carried for slot synchronisation to simplify the experiment but, combined with the processor instructions used to detect them, required additional 'slop' in each slot. This produced a reduction in efficiency of about 8%

DV D 0-D 7 (data bits) P/C

1 bit 8 bits 1 bit 1 bit

DV Data valid in frame P/C Frame parity/Control DO - D7 eight user data bits FC Frame control * See text

Sync n_ n_ n_ n_ /A

Figure 8.2 Sonet Frame Format and Timing

A Sonet transfers data from a station when that station's slot(s) time arrives. This means that in the best case the slot time occurs just as the station wishes to transfer, and in the worst case all stations have fair turns and a maximum delay of one major cycle. For the best, worst case the protocol efficiency is similarly computed. In the experimental Sonet each pair carried a major cycle of a synchronisation, two maintenance slots, a reset slot and 64 communication slots. 64.F n Sonet T1P R .d 68 (F + -2-JL) n V n The service Sonet frame frame layout and timing are shown in figure 8.2. Each frame was phase encoded and transmitted over the network at a rate fixed by a crystal-controlled clock. A frame limitation circuit guaranteed that a frame could not exceed 11.75 cell times. The notional frame efficiency was thus: 8 68 % Sonet Tlf 11.75

As the instruction loop used to detect the minor cycle synchronisation had a spread of detection times, which could cause frames to be sent with different starting times, each slot was made wider than required for the frame by 1.5 cell times. Thus the measured frame efficiency was Tj f = 60%. The total efficiency for the service Sonet was constant at Tj T = 51%. 93 100% H

Figure 8.3 Summary of Efficiencies with Synchronet and Sonet

Thus even this low-cost, service Sonet was always more efficient than Ethernet or Token-passing ring for packets of a half line of data or less, and Ethernet is not more efficient until all packets contain more than 256 bytes. Sonet is always more efficient than a Cambridge ring. Adding these efficiencies to those derived in chapter two produces the graph of figure 8.3, which demonstrates that the network efficiency problem can be gready alleviated for character-oriented devices by adopting the solutions introduced here.

Perhaps as instructive are the patterns of delay versus offered load and actual data delivered (throughput) against offered load for character-by-character transfer shown in fig 8.4. Synchronet exhibits uniform delay and throughput characteristics as might be expected and outperforms the main standard networks in areas which are significant for character-oriented devices.

8.3 Cabling for Sonet

The standard ducting in the Huxley building at Imperial College is too small for many network cables, (e.g. Ethernet and fibre-optic) and particularly is too small to accommodate bends around 94 comers. To overcome this problem the Sonet cable layout was chosen to be both cheap and very simple. A split connection unit solution was adopted with small transceiver units (NTUs) tapped direcdy onto the main Sonet bus cable. This was done by using insulation displacement techniques so that no cable breaks were necessary. This main cable was routed in the passageways and a short drop cable connected each transceiver through the wall to its main network connection unit

Along with the difficulty of the layout of signal cables there is the problem of mains power supply. The Huxley building has an inadequate number of mains socket outlets to support a large network. The approach adopted for the Sonet experiment was to daisy-chain power from the wall socket to the network connection unit and thence to the connected device, giving incremental installation.

The difference in complexity of wiring for different topologies was well demonstrated by the experiments described in this thesis. Twisted-pair screened cable was chosen to provide a fairly rugged solution that would be simple to install. It took one person three days to put the one hundred and fifty taps onto the initial 800m cable run for the Department of Computing's Sonet. These cable runs were then installed to cover all DoC offices on four floors of the Huxley building by two people in three days. A subsequent star network covering the same area and with only twice this original number of Sonet connection points took two people a few m onths to install. Upgrades requiring a new location to be added took one fifth of the time on the Sonet compared with its star replacement.

8.4 Interconnection with Synchronet and Sonet

For the Sonet network both bridge and gateway units have been demonstrated [Chu84]. In addition a repeater station was tested for the Sychronet system to show how to extend its range beyond a few kilometers. For the Sonet experiment, the simplest gateway uses a standard network connection station to provide an RS232c port which is linked to an equivalent port on the other network. The disadvantage of this approach is that the gateway is not intelligent and so requires user actions to operate it. It can be made more automatic by using one of the range of RS232c protocol and control options provided by the Sonet stations, and explained in section 5.4. However, this relies on the other network's interface being able to act in a similar "automatic protocol" manner.

8.5 Security Aspects of the Experimental Sonet

Sonet, like all copper-media-based networks is open to eavesdropping by tapping onto the communication medium. The only protection available with such a network is to include encryption in the higher level protocols. A network can give no absolute guarantee of the source of a message and so similar factors apply to authentication.. To lessen the risk of unauthorised access to the network, Sonet only accepts connection requests from devices which have been nam ed in the currently active name server. Thus, though connecting a suitable spying device to the network will permit eavesdropping it does not permit unauthorised access as a valid user. A secure terminal is 95 required for the name server to prevent an intruder gaining access by validating an invalid name. Deliberate or accidental disruption to Sonet can be curtailed by the monitoring and self-repair circuits in the network discussed below. Because of the unidirectional and selt-timing aspects of the Synchronet proposal any new device connecting to the network can have its position detected by test algorithms run in the other stations. This gives a measure of added protection against unauthorised access though an eavesdropping device can not be detected.

8.6 Fault Monitoring and Finding in Synchronet

A monitor unit is included with the line termination at the end of the loop (back at the starting position). This monitor for the P/T function is a simple monostable arrangement to check that all transmitted pulses are returned and to detect stuck up or down conditions of more than 25 |isec. It can be used to activate back-up or self test operations. A back-up P/T driver can be switched in to replace a failed unit and provided that it is switched onto the line before the failed unit is switched out, and that it is in an extended synchronisation state when switched in, there will be no loss of data, except possibly that caused by throughput loss.

Figure 8.5 Partial Repair of Loopback Bus

There are two possible repair mechanisms for a Synchronet unidirectional bus. Should the entire cable break then the system is partitioned into two segments, and like any other bus can not heal the break without a duplicated cable. The two parts can be arranged to operate separately as shown in figure 8.5, but providing only partial coverage.

repair at Start/End break

Figure 8.6 Repair of Loopback Bus into Simple Bus 96 The Synchronet unidirectional bus has a significant advantage over a simple bus, though, as if only one of its channels is broken it can reorganise itself from the unidirectional mode into a bidirectional simple bus. This can continue to operate with full coverage, but at somewhat reduced efficiency. Additional time must then be allowed for full bus-length propagation between frames as was discussed in chapters two and three. A repair of this kind is shown in figure 8.6.

8.7 Fault Monitoring and Finding in Sonet

Any complex, multi-node system such as Sonet requires considerable attention to aspects of reliability and availability. A wide range of special facilities were added to achieve high availability amongst which were:

• Power on self test of Network Connection, Server and other units • Remotely commanded self test of NCU's • Continuous self test and monitoring of NSU's • Dynamic standby sparing of NSU's with automatic takeover protocol • Logging of errors in transfers between NCU's • Logging of errors and retries in transfers between NCU's and NSU's • Logging of perceived errors in third party transfers.

The chance of a cable break should be much lower than that of many of the semiconductor components in a network. Sadly, experience did not confirm this theory. Though deliberate faults have been inserted to experiment with detection and recovery, during the four-year service period three unplanned failures occurred. As these were a better test of the error recovery mechanisms they are described. Despite their serious nature, none of the failures caused a catastrophic failure of the network and performance continued quite normally for many users.

The first failure was caused by the attentions of telephone maintenance personnel straining the Sonet cable whilst installing and changing their own cabling. The complete cable on level five of the Huxley building was replaced as a consequence. Strain reliefs had been broken, four connectors had been ripped off and the cable had one of its twisted pairs broken in three places. Perhaps surprisingly, the only result of this damage was that for 75% of users the Sonet continued in full operation, with only a marginal increase in error rates. The damage was not immediately apparent as the broken tap connectors were not in use and the cable breaks happened to occur beyond the last connected device on the string. The damage was detected because the break removed the correct termination from the cable. This caused a non-uniform error pattern to be observed, similar to that shown in figure 8.7. This provided a guide as to the location of the fault. This pattern might reasonably be expected by considering the effect of reflections from the unterminated cable-break corrupting the data. 97 Errors

The second fault was believed to be caused in the same way but only resulted in a single cable break. As some devices were located on the far side of this break, the continuous monitoring by NSUs immediately detected the fault and suggested its likely location. The fault was fixed immediately and so insufficient error logging occurred to build up the non-uniform pattern of figure 8.7. The third fault was caused by one driver of a duplex pair failing in a low impedance state. The non-uniform pattern of errors this produced is shown in figure 8.8.

Errors

Figure 8.8 Error Distribution Caused by Low Impedance Driver 98 9.0 APPLICATIONS TO EXISTING NETWORKS

It would be naive to consider this work in isolation, as so many network installations, particularly international standard ones, now exist. Though the use of Synchronet and Sonet as stand-alone networks has been demonstrated it is unlikely that any system would be installed using only character-oriented devices in the future. It is most probable that a mixture of high data-rate block transfer devices; low-cost, medium data-rate block transfer devices and character-oriented devices will be used. As an example, in the test site there has been a steady migration from 97% character- oriented devices when this work started to the current position of about 100 Ethernet-connected, high-speed devices, about 100 Appletalk-connected, medium-speed devices and about 600 Gandalf- connected, character-oriented devices. The Ethernet has already been split with repeaters and bridges to allow control of its load to maintain acceptable performance. The Appletalks are simply bridged onto the Ethernet Extensive additions have had to be made to the Gandalf star switch to permit connectivity to the Ethernet to be established.

To avoid the problem of extensive star wiring and then an expensive connection to the main network the use of multiplexers has been investigated. The problem is that all the devices to be connected are distributed, almost uniformly, throughout the whole test site. It is obvious from the description of the Synchronet that it forms a distributed multiplexer and can be operated in a very similar manner to a synchronous time division multiplexer. This was foreseen in the original proposal [Cri79] when the term distributed synchronous time division multiplexer (DSTDM) was used. Implemented as shown in chapter four, such use provides the most convenient and cost-effective way of connecting character-oriented devices to the major networks.

The study of allocation as a network access mechanism has also led to the additional view that unstable behaviour can be controlled in block-oriented networks. Though this does not form part of the main thesis it is included here as an interesting by-product Other proposals have been made to solve the same problem [e.g Ono78] but they differ in the manner of controlling the mode exchange.

With the CSMA/CD algorithm, as offered load is increased beyond a critical point, the users' data throughput decreases until the network stalls. This is called the Hammersmith Broadway phenom enon and it occurs for the same reason as the traffic jam in West London. Too many decisions are having to be made by too many units so that all available time is taken in deciding who should transmit (or move) leaving no time for actual data transfer (or moving vehicles). Once the possibility of deterministic worst-case operation is seen it is an easy matter to arrange for distributed, deterministic switching between two modes. One method is outlined below, although obviously other deterministic worst-case operating schemes, and other switching schemes are possible.

A switch between modes should be made if a simple deterministic algorithm would perform better than CSMA/CD under the offered load. If the 'Jam' signal occurs for more than 20% of the time then the Ethernet's performance is already noticeably degraded. Beyond 50% critical instability sets in and the performance for editing users is completely unacceptable. A switch to deterministic 99 operation is achieved by all stations, which have observed the high jamming percentage, terminating their jam signals by a predetermined code pattern. Any station seeing the tail jam signal at the end of its own jam signal, or if it sees the tail jam signal when it was not transmitting, holds off from attempting any transmission. Following this a confirmation that all stations have agreed to switch to deterministic operation is signalled by a blank period on the bus. The minimum time for this blank period is twice the minimum packet time [of 57.6 psec] so that it is seen by all stations and they all have a normal chance to transmit which they forgo. The deterministic algorithm could equally well employ token passing [IEEE83d].

If more than two stations were in collision at the time a switch was indicated by the tail jam then it is possible that stations might not see the indication. During the blank time all stations have a chance to try to transmit Any that tries will be immediately jammed off by the nearest station which has switched over. Once the full length blank period has passed all stations operate the deterministic cycle. The switch is shown in figure 9.1. A marginal improvement in reliability is achieved if the final two stations causing the collision which triggers the switch do not actually indicate themselves but instead a third party station starts the jam (with tail jam) once the culprit jamming signal is seen. This is in fact quite likely behaviour anyway if the monitoring and counting of collisions is handled at the end of each collision.

Switching back to random operation can be accomplished as the performance of a deterministic pre-allocation system is easy to monitor. Once its loading falls below that which would have been achieved by CSMA/CD then a switch back is made by any station (or number of stations) jamming in a reserved "deallocation" slot.

OVERLOAD NORMAL LOAD

lillllllll Random Jam Tail Jam Blank | Deterministic §j I*Jam | | Random m m m

Figure 9.1 Switching between CSMA/CD and Pre-Allocation

The percentage of jam signals required to cause a switch-over can be different for each station, with a bias related to the address of the station. This would ensure that the lowest (or highest) active station would attempt to force a switch-over when needed, but before other stations. This would avoid any possibility of the clash described above and would make the blank period shorter.

A measure of hysteresis between the load values detecting the onset of the overload and cessation of the overload condition is required to ensure that 'hunting' between two modes does not occur. It is highly unlikely that this proposal will become anything other than an interesting side note, but the principle of multi-modal networks deserves further study. 100 10.0 CONCLUSIONS

This thesis has provided a number of new insights into network design for character-oriented local area networks. It has demonstrated how existing networks, designed primarily for large block and file transfer between machines, are unsuitable for the connection of user-oriented devices operating on a character-by-character basis. At best such networks are costly solutions to the problem. At worst they perform with such high (and inconsistent) delay that they seriously incommode their users. Some of the most serious problems are those on which the least effort has previously been expended. An analysis of the problems revealed a wide range of areas where small improvements in efficiency could be made. The adoption of all these operational and protocol factors leads to very high efficiency networks for the required devices.

The unidirectional, loopback bus was devised as a solution to the problems described, and has been shown to have excellent properties. The ability of this network structure to provide efficient performance and to scale with improvements in technology has been demonstrated. It is shown how twisted-pair media can be employed to advantege with this structure. Extension to fibre-optic cable is natural and forms the basis for further study. If the unidirectional, loopback bus is used with a deterministic, dense frame-packing protocol it can be very cost effective. This is particularly true for low technology implementations where the effect of the low efficiencies of other approaches can not be masked by an abundance of bandwidth. Examples of such protocols have been theoretically and experimentally demonstrated here. The unidirectional bus can be optimised by employing dynamic bus timing to gain the greatest time-density of frames.

Efficiency for character-oriented devices in any network is greatly improved by the use of virtual channel operation supported by frames of minimum size. For high efficiency file or block transfer the frame size should be as large as can be supported. A balance has to be struck as frame size is proportional to efficiency but inversely proportional to reliability and delay for such transfers. Arguments in favour of frames of intermediate sizes are shown not to be well founded. The use of implicitly addressed virtual channels with selective receivers divorces the switching function on the network from its communication function. This permits lazy scanning and lazy switching to be used, again increasing efficiency and performance while reducing cost The Synchronet and Sonet experiments have demonstrated the soundness of the theoretical derivation of these features. To permit experimentation many novel circuits were devised and demonstrated, including the phase encoding and decoding circuits and the infinite impedance driver.

A full scale experimental network was produced to show also the validity of the higher levels of the solution. It proved highly satisfactory when operated in a full service environment for a four year period of continuous measurement and monitoring. This solution to the problem of character- oriented networks yields superior performance to all current PAD (mesh), ring or bus based systems and can only be matched in efficiency by centralised exchange-style switches with their far higher cabling requirements. The thesis has also demonstrated how the proposed structures provide distributed synchronous time division multipling to act as feed ers to existing standard networks. 101

The advantages and disadvantages of particular cabling strategies have been discussed and the importance of regarding the communications media as an asset to be protected is emphasised. Linking systems to networks with application level protocols is compared with the current international standard approach which is shown to be a heavy, and avoidable, overhead. New protocols will have to be devised, which do not reduce the point-to-point bandwidth, if very high speed networks are to yield performance improvements.

The difficulty of using existing standard serial interfaces to connect devices to networks is highlighted. A wide range of choice of user interface protocols had to be included in Sonet to cope with the inadequacies of the existing standards. Obviously networks themselves have failed to provide a universal connection mechanism so it must be hoped that the connection to them could provide one.

This thesis has also shown insights into the problems of fault detection, location and tolerance for networks. Techniques which can be employed on simple medium speed busses are introduced. These rely on statistical estimation, from triple level fault logs in the various nodes, to estimate the location of a fault. A unidirectional loopback bus adds just sufficient structure to make fault detection and location significantly easier than for a simple bus. It also adds just sufficient redundancy to permit simple faults to be repaired so that the network can continue operation, albeit with reduced performance.

The demonstration of a fibre-optic loopback bus with minimum protocol overheads, and the development of a truly universal serial interface for all point-to-point connections represent the future direction for this work.

Overall the thesis presents an efficient solution to the problem of interconnecting character-oriented devices using local area networks which is convenient, reliable, cost effective and applicable over a wide range of technologies. 102 11.0 CITED REFERENCES AND BIBLIOGRAPHY

[Abr70] The Aloha system, N Abramson, AFIPS Conf. proc Vol. 37 (1970 FJCC). 1970. [Abr78] Computer Networks, eds M Abrams, R P Blanc & IW Cotton, IEEE publication. 1978. [Abr82] Local area network media selection for ring topologies, P Abramson and F E Noel, Proc. COMPCON (fall), Washington D.C. September 1982. [Ada79] Hitch Hikers Guide to the Galaxy, D N Adams, Pan books ltd, London. chl0,p68. 1979. [Ahu79] Routing and Flow Control in Systems Network Architecture, V Ahuja, IBM Systems Journal vol 18 no 2 pp 298-314,1979. [Bac83] Statistical Multiplexers gain sophistication and status, L Bachmann, Mini-Micro Systems, March 1983. [Bar64] On Distributed Communication Networks, P Baran, IEEE trans. Comms. vol 12 pp 1-9. March 1964. [Bar83] CN-III: An Integrated Local Area Network, R Barnett and R C Beckwith, ICCC publication, Imperial College, London. 1983. [BrH83] Classification Categories and Historical Development of Circuit Switching Topologies, G Broomwell and J Heath, ACM Computing Surveys, June 1983. [Bre76] Diagnosis and reliable design of digital systems, M A Breuer and A.D. Friedman, Prentice Hall. January 1976. [Bro78] CCDNET for the DEC PDP9 & 15, D Brown, Dissertation, Dept of Computing and Control, Imperial College, London. June 1978. [CCI80] CC1TT recommendation X21 Interface between DTE and DCE for synchronous operation on public data networks, CCITTVol VIE, Fac VEI.2. 1972 (amended 1980). [Cam86] Wiring up the Workplace, R Camrass and K Smith, IBC publishers. 1986. [Cat83] Kermit Users' Guide, 3rd ed, W Catchings and F DaCruz, Columbia University, New York, 10027. 1983. [Cha86] Local Network Survey, D Chadwick, Systems Int. pp21-28. July 1986. [Chu84] A channel multiplexed bridge unit for SONET system, C M Chiu, Msc Dissertation, Dept of Computing, Imperial College, London. September 1984. [CIa83] Lmodem: A Small Remote-Communication Program, D D Clark, Byte. November 1983. [Cle81] Ultra Low Cost network for personal computers, K Clements & D Daugherty, Byte. October 1981. [Coo85] Notes from mid-revolution, searching for the perfect PBX, E Coover, Data Communications, Aug 1985. [Cra80] Practical considerations in Ethernet local network design, R C Crane and E A Taft, Hawaii International Conference on System Sciences. January 1980. 103 [Cri70] The IBM7094-PDP9/15 Fast Data Link, MDCripps, publication 72/47, Dept of Computing & Control, Imperial College. July 1970 (revised Marchl972). [Cri72] Multifunction Peripheral Units for the PDP9/15, M D Cripps, publication 72/33, Dept of Computing & Control, Imperial College. October 1972. [Cri79] SYNCHRONET - A reliable distributed connection and switching system, M D Cripps and E F Davies, pubn 79/30 CCD, Imperial College. January 1979 (rev. January 1980). [Cri82] SONET: Introduction and User's Guide, MDCripps, Mouse Enterprises Ltd. Carshalton, Surrey. July 1982. [Cri84a] SONET: A Simple Open Network, M D Cripps, M J Reeve and C Sowden, proc 8th annual Microprocessor Symposium, University of Liverpool. March 1984. [Cri84b] SONET: An Efficient Byte Transfer LAN, M D Cripps, M J Reeve and C Sowden, proc European Computer Communications Conf. Networks 84, London. July 1984. [Cri85] SONET: A Simple Open Network, M D Cripps, M J Reeve and C Sowden, Journal of Microcomputer Applications, vol 8 pp 63-74. August 1985. [Cri87] Computer Interfacing: Connection to the real world, M D Cripps, Edward Arnold (publishers) Ltd. ISBN 07131 3588 3. 1987. [Csg83] The NET Command and Program, G Pugh, Computing Support Group 4.45/21, Imperial College, London. 1983. [DaC84] Kermit Protocol Manual 5th ed, FDaCruz, Columbia University, New York, 10027. April 1984. [Dat81] Networks and Architectures (WangNet), Datapro reports on Data Communications, Datapro Research Corp. Delran, New Jersey, USA. October 1981. [Dav78] CCDNET for the IBM 370, E Davies, Dissertation, Dept of Computing and Control, Imperial College, London. June 1978. [Dav84] Security for Computer Networks, D W Davies and W L Price, John Wiley & sons, Chichester. 1984. [Dig80] The Ethernet, a local area network specification, Digital Equipmentjntel and Xerox corporations, Xerox Corporation, Stamford, Conn. October 1980. [Dix83] A token ring network for local data communications, R C Dixon, N C Strole and J D Markov, IBM Systems journal vol 22 no 1. 1983. [EIA69] EIA standard RS232C, Electronic Industries Association, EIA publications, Washington, DC. October 1969 (reasserted June 1981). [Egg86] PABX Reliability, Throughput and Cost, T Egginton, Systems Int. pp34-39. July 1986. [Eid82] Local area network transmits at multiple speeds, D Eidsmore, Computer Design, February 1982. [Eos85] Cable Management Program, EOSYS report (C. Cole), EOSYS ltd, Famham Common, UK. 1985. [Far69] An experimental distributed switching system to handle bursty computer traffic, W D Farmer and E E Newhall, Proc. ACM symposium Optimisation of Data Communication Systems, 1-33. October 1969. 104 [FiR86] CCITT X21 and Data Communication Testing, D Fish and W Risley, Telecommunications, March 1986. [Fie85] Self-clocking Networks, A J Field and M D Cripps, Proc IEEE International Conference on Parallel Processing, Chicago. August 1985. [FoI82] Data Communication Standards 2nd ed, ed. H C Folts, McGraw Hill. 1982. [Fra83] Towards a Universal Data Transport System, A G Fraser, IEEE journal Selected Areas in Communications vol 1 no 5 pp 803-16,1983. [Gan85a] Gandalf PACX 2000 system overview, 3080G revA0, Gandalf Data Ltd. Ontario, Canada. August 1985. [Gan85b] Gandalf PACX 2000 system operation, 3043G revA2, Gandalf Data Ltd. Ontario, Canada. June 1986. [Gif86] ISDN User Network Interfaces, W Gifford, IEEE journal Selected Areas in Communications, May 1986. [Gra84] Multilink - A ring for connecting individual outstations, J Grant, proc European Computer Communications Conference Networks 84, London. July 1984. [Hal87] FAMNET: an integrated voice and data network, F Halsall, D Sirovica and S M Harrison, Proc. IEE. Vol 134 part E no 1, January 1987. [Ham75] Development of a transmission error model & an error control model, J L Hammond, J E Brown and S S Liu, Tech rept RADC-TR-75-138, Rome Air Dev. Centre. 1975. [Har84] A Schmidt trigger design technique for logic noise spike elimination, B L Hart, UEEE vol 21 pp 353-6,1984. [Hay85] Network monitoring station for Sonet, P Hayden, Dissertation, Dept of Computing, Imperial College, London. June 1985. [Hem79] System X - The Complete Approach to Telecommunications, W A C Hemmings, Systems Technology No 32. September 1979. [HoM82] Choosing between Broadband and Baseband Local Networks, G T Hopkins and N B Meisner, Mini-Micro Systems, June 1982. [Hop86] Local Area Network Design, A Hopper, S Temple and R Williamson, Addison-Wesley publishers ltd. 1986. [IBM63] IBM7090/7094IBSYS Operating System Verl3 System Monitor, C28-6248-4, International Business Machines Corp. 1963 (revised April 1965). [IBM82] Office Systems Research, IBM Systems Journal Vol 21 No 3. 1982. [IEEE83a] IEEE standard 802.1 Architecture and Internetworking, Institution of Electrical and Electronic Engineers. March 1983 (revised as ANS1/IEEE standard 1985). [IEEE83b] IEEE standard 802.2 Logical link control (connectionless, transaction and connection oriented). March 1983 (revised as ANSI/IEEE standard 1985). [IEEE83c] IEEE standard 802.3 CSMA/CD, Access method and Physical Layer Specification. March 1983 (revised as ANSI/IEEE standard 1985). [IEEE83d] IEEE standard 802.4 Token Passing Bus Access method and Physical Layer Specification. March 1983 (revised as ANSI/IEEE standard 1985). [IEEE83e] IEEE standard 802.5 Token Passing Ring, Access method and Physical Layer Specification. March 1983 (revised as ANSI/IEEE standard 1985). 105 [IS079a] ISO standard 3309 Data communication High Level Data Link Control procedures, Frame structure 2nd ed. International Standards Organisation. 1979. [IS079b] ISO standard 4335 Elements of procedures for ISO3309, International Standards Organisation. 1979. [IS086a] Connection OrientedTransport Protocol and Transport Service Definition, IS08072, International Standards Organisation, 1986. [IS086b] Basic Connection Oriented Session Protocol and Service Definition, IS08327 and 8329, International Standards Organisation, 1986. [Jos86] High Performance Networks, a focus on the Fiber Distributed Data Interface, S Joshi, IEEE Micro, June 1986. [JuN83] Digital Private Branch Exchanges, S Junker and W Noller, IEEE Communications magazine, May 1983. [Kas79] Survey of Digital PBX design, J M Hasson, IEEE trans. Communications, July 1979. [KuR78] Packet and Circuit Switching: Cost/Performance Boundaries, K Kiimmerle and H Rudin, Computer Networks vol 2 pp 3-17,1978. [Lar82] Cambridge Ring 82, Protocol specifications, ed J Larmouth, Joint Network Team, Science & Engineering Research Council. 1982. [Law82] A study of file transfer protocols, K J Lawrence, Dissertation, Dept of Computing, Imperial College, London.. June 1982. [Lee83] The principles and performance of Hubnet, E S Lee and P I Boulton, Computer systems research group, Univ. of Toronto, Canada. 1983, also in IEEE SAC-1(5) pp 711-20, Nov 1983. [MED79] A survey of Terminal Protocols, F Magnee, A Endrizzi and J Day, Computer Networks, Nov 1979. [Mal81] The Microcomputer Connection to Local Networks, J Malone, Data Communications, Dec 1981. [Mea80] Introduction to VLSI Systems, ch7, C Mead & L Conway, Addison Wesley. 1980. [Met76] Ethernet: Distributed packet switching for local computer networks, R M Metcalf & D R Boggs, Comm ACM Vol 19, No 7, July 1976. [NCC82] NCC staff (ed G Bleazard), Handbook of Data Communications 2nd ed. NCC Publications. 1982. [Ono78] A Random Access Packet mmunication System with Priority Function - Priority Ethernet, M Onoe, Y Yasuda and M Ishizuka, National convention of the Information Processing Society of Japan, 3A-1, Aug 1978. [PDP68] PDP-9 User's Manual, F95, Digital Equipment Corp, Boston, Mass, 1968. [Pf082] Comparing the CBX to the Local Network, G Pfister and B O'Brien, Data Communications, July 1982. [Pie72] Network for block switching of data, J R Pierce, Bell System Technical Journal, Vol 51 No 6. July 1972. [Ph!85] The Use of Local Area Networks in Secure Office Automation, P Phillips, Computer Newsletter, Nov 1985. 106 [PoZ78] A Tutorial on Protocols, L Pouzin and H Zimmermann, proc IEEE, Nov 1978. [Qua86] Notable Computer Networks, J S Quarterman and J C Hoskins, CACM Vol29, No 10, pp932-971. October 1986. [Rao86] Fabrication of a Bipolar Local Area Network Interface Chip, G Rao and C Lewis, Texas Instruments Technical Journal, vol.3, no.6. November 1986. [RoS76] Circuit and Packet Switching - a Cost and Performance Tradeoff Study, R D Rosner and B Springer, Computer Networks vol 1 pp 7-26,1976. [Ros65] PUFFT The Purdue University Fast FORTRAN Translator, S Rosen, R A Spurgeon and J K Donnelly, CACM v8 noil. November 1965. [RuM79] On Routing and Flow Control, H Rudin and H Muller, proc Int Symp. Flow Control in Computer Networks pp 241-6, Versailles France, 1979. [Sau79] CCDNET for a microprocessor - the M6800, K Saunders, Dissertation, Dept of Computing and Control, Imperial College, London. June 1979. [Sch81] The new breed - Switching Multiplexers, T H Scholl, Data Communications, June 1981. [ScW74] The National Physical Laboratory data communications network, R A Scantlebury and P T Wilkinson, 2nd Int Conf Computer Communication (ICCC74) 223-8, Aug 1974. [Sha82] Cambridge Ring 82, Interface specifications, eds W P Sharpe & A R Cash, Joint Network Team, Science & Engineering Research Council. 1982. [Sha85] NET and KERMIT communication for the Corvus, S Shah, Dissertation, Dept of Computing, Imperial College, London. June 1985. [Sho79] Measured performance of an Ethernet local network, J F Schoch and J A Hupp, Local Area Communications Network Symposium, Boston. May 1979. [Sou78] CCDNET for the Texas Instruments 960A, C Southall, Dissertation, Dept of Computing and Control, Imperial College, London. June 1978. [Sta86] Here is one way to get a close estimate of a Data Link's Efficiency, W Stallings, Data Communications, Oct 1986. [Tow88] Mixing Messages with Multiplexers, K Townsend, Informatics, Vol9 no6, Jun 1988. [Wil75] Communication using a digital ring, M V Wilkes, proc PACNET conference, Japan. August 1975 (revised December 1976). [Xer81] Internet Transport Protocols, XSIS 028112, Xerox OPD, December 1981. [Zim80] OSI Reference Model - The ISO model of architecture for open systems interconnection, H Zimmermann, IEEE trans. Comms. COM-28 4. April 1980. 107

APPENDIX

APPENDIX A. Symbols and Abbreviations

CSMA/CD Carrier Sense Multiple Access with Collision Detection d Distance between directly connected target stations dn distance on network between target stations DSA Direct Store Access O Luminous flux ( in lumens) Fd Frame length of user data frame in bits Fn Frame length on network Ilf Frame efficiency of network Tip Protocol efficiency of network Tit Throughput efficiency (of user communication to network) riT Throughput efficiency of network ISDN Integrated Services Digital Network LAN Local Area Network M Number of minipackets on a ring N Number of stations (or nodes passed through) on network O n Overhead on network OSI Open Systems Interconnection PABX Private automated branch exchange PAD Packet Assembler and Dissassembler PSTN Public Switched Telephone Network Ptr Probability of a transmission being successful PTT Postes, Telephonique et Telegraphique (common carrier) 0 Secondary beam luminous flux R d Data rate of user interface Rn Data rate on network RS232C Interface standard of the Electrical Industries Association of America T i0 Delay due to input/output at end stations T nd Delay due to transmission on network Fndm ax Maximum delay due to transmission on network Fpr Delay due to protocol overheads T td Time to transmit user data u d Number of user data bits in user data frame V Velocity of signal propagation vn Velocity of signal propagation on network V24 Committee Consultatif Internationale Telephonique et Telegraphique Standard W avg Mean wasted transmission time X2I, X25 Committee Consultatif Internationale Telephonique et Telegraphique Standards APPENDIX B. Protocol Generation ROM Code.

"6802" **31/8/82 PROTOCOL ROM FOR SONET (M.D.Cripps 1982) * CDET (CLEAR =1, DATA =1, ENAB =0, TEST =1) REPT 10 BIN 00101111 [lOflS LOW DATA end of final slot] REPT 40 BIN 01011111 [40JIS PROTOCOL NMI sync] REPT 5 BIN 00001111 [5|lS LOW DRIVE END LONG] REPT 35 BIN 00101111 [35|1S LOW REST END LONG] REPT 10 BIN 01011111 [CMD/R1 10|1S HIGH SHORT] REPT 5 BIN 00001111 [5flS LOW DRIVE] REPT 45 BIN 00101111 [45|1S LOW SLOT] REPT 10 BIN 01011111 [ID/R2 lOp-S HIGH SHORT] REPT 5 BIN 00001111 [5|lS LOW DRIVE] REPT 45 BIN 00101111 [45|lS LOW SLOT] REPT 10 BIN 01011111 [RESET/RESET 10|lS HIGH SHORT] REPT 5 BIN 00001111 [5|lS LOW DRIVE] REPT 40 BIN 00101111 [40J1S LOW SLOT (for RTI timing)]

REPT 10 BIN 01011111 [USER 00 10|1S HIGH SHORT] REPT 5 BIN 00001111 [5M-S LOW DRIVE] REPT 50 BIN 00101111 [50JJ.S LOW SLOT (for RTI timing)] ★ SLOTDEF MACRO [Defines standard slot =49p.s] REPT 10 BIN 01011111 [10|ls HIGH DRIVEN SHORT] REPT 5 BIN 00001111 [5\iS LOW DRIVEN RECOVERY] REPT 44 BIN 00101111 [44|1S LOW DISABLED] MEND * REPT 61 [standard slots 01-62 ] SLOTDEF [10 up/5 low/44 off] * REPT 10 BIN 01011111 [USER 63 10(1S HIGH SHORT] REPT 5 BIN 00001111 [ 5 J1S LOW DRIVE] REPT 34 [ 34+1 |1S LOW SLOT] BIN 00101111 [ LOW SLOT] BIN 10101111 [ (1) PLUS CLEAR 10 J1S LOW AT START] * * COPYRIGHT NOTICE IN ROM HEX C3,A0,CD,CF,D5,D3,C5,A0,C5,CE,D4,A0,Bl,B9,B8,B2 ASC "COPYRIGHT M.D.CRIPPS 1982." END 109 APPENDIX C. NCU Primary Self Test.

The primary self test routines must be passed before any NCU may operate on the network. Primary self test commences with the syndrome pattern F/F visible as outputs from the B-half of the PIA. The primary self test routines check the basic operation of the processor, the checksum for the test code in read only store, and the ability to address the PIA before setting genuine syndromes. Syndromes are eight bit patterns and are interpreted as shown below.

i i Test 1 1 Cycle Test being run OK Failure Syndrome i_____ i _____ I_____ 1_____

Figure A.1 Primary Self Test Syndromes.

Cycle Indicates that a test is cycling. If not set then the test is performed once only. Test OK =1 with syndrome =0 indicates correct termination of a test. The syndrome is set to any other pattern to indicate the reason for the test failing, with Test OK=0. The reason for this is to ensure at least one output is set aslamp a check.

The tests carried out during primary self test and the reasons for failure are described below with the appropriate syndromes.

F/F Internal 6802 checks: Check setting and clearing of carry and branch. Check loading of A and B registers, comparison of registers and branching on results. Check the direct addition of registers and subtraction from registers and branching on conditions {+,-,=} Check loading and comparison of index register and branch.

The test code part of the read only store is then checksum tested to ensure it is not corrupt and that the ROS is addressable. This checksum has eight, alternate odd and even parity checks to ensure the test is safe. The PIA addressability is then tested by first reading then writing and re-reading the control registers and then stepping through the B direction register checking writing and reading of each bit. As all the B-half pins are outputs this is a safe test, but a similar test can not be performed on the A half. ?/? Following the setup of the correct direction register patterns and control register contents the B outputs themselves are checked by setting and reading each one. This steps the syndrome through various patterns but the program should not halt with any of these showing. 7/0 Once this is complete the syndromes are set to 7/0 to indicate completion of the internal tests.

The external primary tests then commence and syndrome patterns indicate their progress, though it should be borne in mind that the PIA outputs or indicators could fail and so the syndrome shown may not be that which is being output. Using a logic analyser, emulator or similar would show the unique addresses. Syndrome patterns on the left margin will be displayed during a test and continuously if the test described fails.

6/1 The rest of the read only store is checksum tested in a similar fashion to the test code, again with both odd and even checks. 6/2 The internal 128 byte read/write store in the 6802 processor chip is tested by writing a test pattern and checking it to ensure it is correct Alternate bits are tested for 0 & 1. 6/3 The content of the store is inverted so all bits should change. The new content is checked. 6/0 If the store checks correctly the syndrome 6/0 is output. The other code 6 syndromes are used for checks on the additional RAM stores in the NMUs. 110

5/1 The state of the enable (ENBL) line is then checked to ensure that the NCU could not output to the network if the FAIL condition was removed. If this line is in the wrong condition the NCU could become a jibbering idiot box and corrupt others. 5/2 The addressability of die network communications chip is tested by reading its status which should contain a true DCD condition, as this is wired in, and a false CTS as this is connected to the FAIL signal. 5/3 The network communications chip is then set up and further tested to ensure it is unable to transmit (TDRE false) as it is held so by FAIL. If this register is not addressable after setup this test will discover the condition. D/? The NCU ID is read in. This involves cycling through patterns on the syndrome lower bits so the additional D is output in case failure occurs, to avoid confusion. 5/4 The ID is set on the least significant 7 switches (reading lsb from the left) and has the most significant bit (rightmost) set to give ODD parity. If the parity is incorrect the NCU is set to port reading, and otherwise in safe states and the test fails. As the ID can be set by the user, the CALL button is used to indicate the switches have been set and the ID should be re-read. This causes the previous tests to be rerun and the ID to be read. Thus setting an illegal ID and keeping CALL set cycles the tests.

The ID numbers run from 0-127 and are set on eight switches inside the NCU by the indicator light By convention zero is reserved for test units, NMU boxes have their numbers decending from 127.

[switches at edge of printed circuit board] Switch number 1234567 8 on Function Parity 64 32 16 8 4 2 1 [odd] [ID number set in binary] off

5/0 If a valid ID is read in syndrome 5/0 confirms that the first group of external tests have been passed. If secondary self test is selected the rest or the primary tests are omitted.

5/1 The reception ofProtocol signals from the net is checked. If protocol is not detected and the enable signal is correct, the program flashes the LED alternately green and red at half second intervals. Failure of the ENBL test is indicated by syndrome 5/1. 4/1 The protocol channel should not stay up more than 40jis. When protocol on the net drops the PIAB1 input goes up and sets the flag bit on by negative edge activation. 4/2 The short pulses on the protocol channel last 10|is. The protocol should not remain low for more than 80 j is . 4/3 The reception of the NMI generation protocol from the network is tested. The program waits for the long code to appear allowing 3800jis. 4/4 The NMI generation protocol must appear again within the maximum allowed time. If the long protocol fails to go low this test is failed. 4/0 If the protocol conforms to specification then the NCU is content that it is connected to a network. If the protocol fails then the requisite syndrome is displayed with the flashing LED. To permit the user to re-enter the primary self-tests from this point the CALL button is used and if pressed returns to powerup. Following the successful primary self test sequences the NCU initialises all the primary defaults. The ID is set as read from the switches and an Illegal slot number of -4 is set. The POR command is validated for the NSU to have the default settings loaded from the NSU. The NCU can not operate until the defaults are set. 5/5 If, during the operation of an NCU the ENBL check fails, the NCU disables itself from transmitting on the network. 7/1 If, during the operation of an NCU an illegal interrupt or a software interrupt occurs due to either hardware or software failure, the NCU disables itself from transmitting on the net i l l Appendix D NCU Store Arrangement

This appendix describes the arrangement of each NCU store. The main store map is shown in figure D. 1. The RAM locations contain much useful information which may be observed via various NSU commands including MEMory.

USER NETWORK Interrupt RAM (128) PIA (4) ACIA(2) ACIA (2) ROM (4K BYTES) Vectors (8)

0 0 0 0 2 2 4 4 FF FF 0 0 0 0 0 0 0 0 0 F F F 0 7 8 8 0 0 0 0 0 F FF 0 F 0 3 0 1 0 1 0 7 8 F

Figure D.l NCU Store Map.

Each NCU maintains its own log of errors it observes in its own transactions and those it perceives on transactions it believes other NCU's are carrying out (it may be wrong!). This logged information is structured into a three level error log and is held in the first two words of the NCU store. The primary error log contains observed communication errors [A] over the network to the user, and [B] from the user to the network. The presence of a log error does not mean that there has necessarily been an error as for instance a condition gives a framing error which is logged even though the condition may have been deliberately caused.

<1> Primary Error Log Logs an NCU virtual channel since the last connect or Clear. [A] Net error log, received over net (8 bits as:) <—><—><—><—> [B] User error log, caused by user and states of interface signals (8 bits as:)

<2> Secondary and Tertiary Error Logs Error log of idle NCU to NSU link since last Clear command. [A] Secondary level net error log (8 bits as:) [B] Tertiary level net error log (8 bits as:) <—xParity errxOverrun errxFrame e rr x —x —xN o NCU ID in fram exNo CMD>

<3> Slot Number and Virtual Channel Characteristics Connected devices are allocated a slot number, a number of slots and a hand to define their channels. Communicating devices have the same slot number. Other codes are used to indicate the manner of their last disconnection.

[A] Slot number or disconnection code (8 bits as:) <—x~ x s i x bit SLOT Number> if connected, or a code: disco by NSU command disco by User lowering CTS or DCD signals initial powerup [B] Virtual channel characteristics (current or last used) < C P x C P x ~ x F C O T init statexPort/Stbd Tx><2 bit SLOTS > if< CP >=10 Connection is pending the end of output, else=not CP. and 00=2400,01=4800,10=9600,11=19200. 112 <4> User Data Speeds and Escape Character Pattern The current transmit and receive speeds, and the 8 bit pattern of the Escape character are held. A flag indicates whether escape by character is currently enabled or not

[A] User data speeds for user RS232 (two 4 bit values with defaults of X'7'as:)

code user data rate code user data rate code user data ratecode user data rate 0 50 bps 1 75 bps 2 110 bps 3 134.5 bps 4 150 bps 5 300 bps 6 600 bps 7 1200 bps 8 1800 bps 9 2000 bps A 2400 bps B 3600 bps C 4800 bps D 7200 bps E 9600 bps F 19200 bps [B] ESCAPE character pattern if used (8 bit value as:) <8 bit escape code for usei> validated in NCU flags.

<5> NSU and NCU Flags (l=on, 0=off unless shown otherwise) Six flag values may be stored for the exclusive use of NSUs. The flag parameters for each NCU are set into it from the active NSU's tables on power-up or reset. [A] NSU flags < 1 x 1 x 6 bits of NSU flags> <0><0>< no NSU flags set > [B] NCU flags (eight bits as:) <-->

<6> ID and User Acia Settings The user ACIA setting, copied regularly into store to permit monitoring, includes the state of Request To Send, and the mode parameters (see table) which are loaded like other NCU parameters. The NCU ID is read from the switches or header mounted in the NCU. [A] ID [B] User ACIA (eight bits as:) <0>

Mode Code Data Bits Parity Stop Bits ModeCode Data Bits Parity Stop Bits 7E2 0 7 Even 2 702 1 1 Odd 2 7E1 2 7 Even 1 701 3 1 Odd 1 8N2 4 8 None 2 8N1 5 8 None 1 8E1 6 8 Even 1 801 7 8 Odd 1

<7> Net Acia Settings The net ACIA setting includes its MODE parameter and the state of the DTR interface signal. The mode changes for out of band control functions. [B] is a dummy constant. [A] Net ACIA (eight bits as:) <0xD TR 0=onx0xM ode: 801=7 or 8E1=6 xstate, 00=normal> [B] In_temp cconstant zero byte>

<8> NCU Input Buffer Count and Status The count is the number of characters from the user. A semaphore indicates if the buffer is assigned to the NCU or to the currently active NSU. Flags show the status of the NCU. [A] Input count and assignment of buffer (eight bits as:) <0 NCU has bufferx7 bit chararacter count> <1 NSU has buffer> [B] NCU Status (eight bits as:) DCDxRejectVC><—> 113 <9> User Input Buffer (<9> and following numbers) The maximum length of the input buffer is 36 characters, stored in byte pairs. Internal variables follow until: <36> User Input Buffer, Count and Contents [A] Output count and assignment of buffer (eight bits as:) <0 NSU has buffer><7 bit chararacter count> <1 NCU has buffer> [B] Output buffer to user, which contains clear text (having the top bit =0) and coded messages which are expanded from the NCU Rom (top bit =1). The rest of the store is taken by internal variables and the stack. <0> and greater than <63> are illegal operands for the MEMory command as they address beyond the range of the NCU store. These details refer to release NCU: 1.0: 29FE. 114 Appendix E NSU Store Arrangement

Querying a without specifying a causes the node table for the given node, held in the NSU receiving the query, to be printed out. Each node table contains 64 bytes of data and in release NSU 2.3: 2210 has the form shown below.

[0] Node Flags and default drop and mode {NCUFlags2)(8 bits as:) < Current Node Flags >< Default NCUF2 > <3 bit Mode> [1] -[6] Node Password, six characters password. [7]-[12] Transfer Destination, six character name. [13] Node Name 1, Name Flags (8 bits as:) [14] -[19] Node Name 1, Name Text, six character name. [20]-[25] Node Name 1, Transfer Destination, six character name. [26]-[38] Node Name 2, and associated flags arranged as 13-19 above. [39]-[51] Node Name 3, and associated flags arranged as 13-19 above. [52] NCU Flags 1 Default speeds (two four-bit codes as:) [53] NCU Flags3 Default and Current flags, 8 bits as: < Default x Current > [54] NCU Flags4, Default Escape pattern if set, 8 bits.

[55] USER Flags, Default and Current flags, 8 bits as: < Default x Current > [56] VC Flagsl, 8 bits as: <6 bits, Base Slot as NCU><2 bits, Slot Code as NCU> [57] VC Flags2, 8 bits as: [58]-[63] Name of Node Connected toOR Queued on, 6 characters. 115 Appendix F Sample Test Procedure for a NTU

A sample test procedure is shown here which demonstrates how Network Transceiver Units (driver/ receiver cards) can be certified as reliable before insertion into a live network. This is typical of the large range of such procedures designed for the Sonet network. Five partially failed 6802 processor and RAM chips were found, prior to installation, by the NCU checks as were 3 failed ACIAs, six other integrated circuits and a switch pack with inverted pin connections!

Using a small test network an NSU is set up with a manager terminal and one known good NCU/NTU pair. This NCU is named in the NSU and has its Connect flag set on. The NCU must have DCD and CTS strapped true if no terminal is available to connect to it. A second known good NCU is required with a terminal, ready for the NTU which is to undergo testing. The NSU should be set with the correct parameters for the testing pair (eg P on, W on, Speeds correct for the terminal etc). Should any failure be apparent then it will be indicated by the syndrome patterns (e.g appendix C) or the external indicators.

If the NTU/NCU is not rack mounted, i.e. the normal test setup then:

1) Set an incorrect identification number (ID) on the NCU (eg zero) 2) Connect power to the NCU 3) Connect the NTU to the network 4) Connect the NTU to the NCU 5) Set the correct ID on the NCU 6) Reset the NCU (by placing a short across C3 with a miniature probe) 7) The NCU indicator light should go from red to a firm green and a prompt should appear on the terminal 8) Type 'TEST' followed by the ID set for the NCU on the NSU console to initiate a remote test on the NTU/NCU combination. 9) Type a few commands on the NCU checking the responses match those in the User Guide 10) Finally CALL the good NTU/NCU. If a terminal exists on it as well then a duplex check can be made.

If the device is rack mounted then the procedure is varied by setting the correct ID on the NCU and then performing 3) 4) 2) 7) 8) 9) 10) but if 7) does test correctly then 6) 7)....