Error Behaviour in Optical Networks Laura Bryony James Corpus Christi College This dissertation is submitted for the degree of Doctor of Philosophy 30th September 2005 Department of Engineering, University of Cambridge This dissertation is my own work and contains nothing which is the outcome of work done in collaboration with others, except as speci¯ed in the text and Acknow- ledgements. Abstract Optical ¯bre communications are now widely used in many applications, including local area computer networks. I postulate that many future optical LANs will be required to operate with limited optical power budgets for a variety of reasons, including increased system complexity and link speed, low cost components and minimal increases in transmit power. Some developers will wish to run links with reduced power budget margins, and the received data in these systems will be more susceptible to errors than has been the case previously. The errors observed in optical systems are investigated using the particular case of Gigabit Ethernet on ¯bre as an example. Gigabit Ethernet is one of three popular optical local area interconnects which use 8B/10B line coding, along with Fibre Channel and In¯niband, and is widely deployed. This line encoding is also used by packet switched optical LANs currently under development. A probabilistic analysis follows the e®ects of a single channel error in a frame, through the line coding scheme and the MAC layer frame error detection mechanisms. Empirical data is used to enhance this original analysis, making it directly relevant to deployed systems. Experiments using Gigabit Ethernet on ¯bre with reduced power levels at the receiver to simulate the e®ect of limited power margins are described. It is found that channel bit error rate and packet loss rate have only a weakly deterministic relationship, due to interactions between a number of non-uniform error characteristics at various network sub-layers. Some data payloads su®er from high bit error rates and low packet loss rates, compared to others with lower bit error rates and yet higher packet losses. Experiments using real Internet tra±c contribute to the development of a novel model linking packet loss, the payload damage rate, and channel bit error rate. The observed error behaviours at various points in the physical and data link layers are detailed. These include data-dependent channel errors; this error hot- spotting is in contrast to the failure modes observed in a copper-based system. It is also found that both multiple channel errors within a single code-group, and multiple error instances within a frame, occur more frequently than might be expected. The overall e®ects of these error characteristics on the ability of cyclic redundancy checks (CRCs) to detect errors, and on the performance of higher layers in the network, is considered. This dissertation contributes to the discussion of layer interactions, which may lead to un- foreseen performance issues at higher levels of the network stack, and extends it by considering the physical and data link layers for a common form of optical link. The increased risk of errors in future optical networks, and my ¯ndings for 8B/10B encoded optical links, demonstrate the need for a cross-layer understanding of error characteristics in such systems. The development of these new networks should take error performance into account in light of the particular requirements of the application in question. Acknowledgements I am grateful for the support, encouragement and ideas provided by my supervisor Ian White, particularly whilst I was writing up. Many thanks also to Andrew Moore for untiring assistance, insightful advice and sage wisdom, and for eating my food so I didn't have to. This work would not have been possible without many helpful discussions with Derek McAuley, Madeleine Glick, Richard Penty and numerous others in Intel Research, the Com- puter Lab and the Photonics Systems Group who spared the time to contribute their thoughts | thanks to you all. In less technical ways, I received a great deal of support during my studies from mentors and former colleagues, who in various ways advised and encouraged me; you have my eternal gratitude. Markus Fromherz, Ursula Martin and both the ex-ORL/ex-AT&T and Menlo Studio crowds deserve special mentions. Many thanks to Adrian Stephens, Olly Johnson, Richard Gibbens, Dick Plumb, Ian Wassell and David Hunter, for clarifying things I didn't understand and providing feedback on my work. I would like to acknowledge James Bulpin, for the development of the non-uniform error testbed of Section 5.4.1. Many thanks also to Adrian Wonfor, who kept me up to date with the intricacies of running Microsoft Windows Server, ensured that I had a working computer and ¯le storage, and sometimes even helped me with my work. I am extremely grateful to Guy Roberts, Michael Dales, Wenxin Tang, Tao Lin, Enrique Rodriguez de la Colina and Amyas Phillips for repeatedly having the patience to listen to my ramblings and still suggest ideas afterwards, for uncomplainingly proofreading, and for many a productive (or otherwise) tea break. Jeremy Sosabowski also helped me maintain a balance between work and tea, and the PhD experience would not have been the same without him. The UK Engineering and Physical Sciences Research Council and Marconi Corporation supported my work ¯nancially through an Industrial CASE studentship. Finally, most of all, I would like to thank my parents, and Paul, for valuable proofreading and invaluable support. Publications Parts of the following work have previously appeared in the following publications. London Communications Symposium (LCS) 2003 Wavelength Striped Semi-synchronous Optical Local Area Networks L B James, G F Roberts, M Glick, D McAuley, K A Williams, R V Penty and I H White Passive and Active Measurement Workshop (PAM) 2004 Structured Errors in Optical Gigabit Ethernet L B James, A W Moore, M Glick London Communications Symposium (LCS) 2004 Beyond Gigabit Ethernet: Physical Layer Issues in Future Optical Networks L B James, A W Moore, R Plumb, M Glick, A Wonfor, I H White, D McAuley and R V Penty Optical Fiber Communications Conference (OFC) 2005 Packet error rate and bit error rate non deterministic relationship in optical network applica- tions L B James, A W Moore, A Wonfor, R Plumb, I H White and R V Penty Poster presented at INFOCOM 2005 A Graphical Exploration of non-Uniform Errors L B James, A W Moore, M Glick and A Wonfor IEEE Communications Magazine, August 2005 Chasing errors through the network stack: A testbed for investigating errors in real tra±c on optical networks A W Moore, L B James, M Glick, A Wonfor, I H White, D McAuley and R V Penty IEEE Journal of Lightwave Technology To appear October 2005 Optical Network Packet Error-Rate due to Physical Layer Coding A W Moore, L B James, M Glick, A Wonfor, R Plumb and I H White For my parents Contents 1 Introduction 1 1.1 Optical Networking . 1 1.2 Computer Networking . 3 1.3 Motivations . 11 1.4 Outline . 15 2 Development of a New Prototype Optical Network 17 2.1 A Review of Some Optical Networking Technologies . 17 2.2 The SWIFT Network Prototype . 25 3 Context 39 3.1 Coding and Errors . 39 3.2 Error Detection Methods . 43 3.3 System Design: Layering . 46 4 Error Behaviour of Gigabit Ethernet Using 8B/10B Coding 49 4.1 Outline of 8B/10B Block Coding Scheme . 50 4.2 Analysis of the E®ects of a Line Error in Gigabit Ethernet . 55 5 Observed Error Characteristics in Gigabit Ethernet 75 5.1 The E®ects of Optical Attenuation . 76 5.2 Network Performance Implications of Error Hot-spotting ............. 99 5.3 Whitening to Achieve Uniformity of Error . 100 5.4 A Comparison With Copper Physical Layer Networks . 103 5.5 Summary of Experimental Observations of Errors in Gigabit Ethernet . 107 6 Measuring Errors Through the Network Stack 109 6.1 Error Hot-spotting and Real Network Tra±c . 110 6.2 Transmission Experiments With Real Network Tra±c . 116 6.3 Connecting Packet Loss Rate to Bit Error Rate . 122 6.4 Explaining Relative Bit Error Rates and Packet Loss Rates for Di®erent Payloads131 7 Error Non-uniformities and Layer Abstraction 137 7.1 Summary of Work . 137 7.2 Layer Abstraction and Error Behaviour . 141 7.3 Designing Future Optical Networks . 146 7.4 Overall Conclusions . 152 Bibliography 165 List of Figures 1.1 Layering system as it applies to a Gigabit Ethernet network with example ap- plication . 7 1.2 The Gigabit Ethernet Reference Model . 10 1.3 An illustration of bit-rate and transmission power trade-o® . 13 2.1 Overall architecture of the SWIFT demonstrator . 27 2.2 Overall network topology . 29 2.3 2 £ 2 optical add-drop switch structure . 30 2.4 3 £ 3 cross-bar switch architecture using discrete SOAs . 30 2.5 An illustration of the concepts of control channel architecture and broadband data switching . 31 2.6 Slot and packet timing at switch or receiver . 33 4.1 Valid 8B/10B code-groups represented on the full 10-bit codespace, showing the current code-group (Ci) and the preceding one (Ci¡1)............... 52 4.2 Frame validity check process at the PCS and MAC layers . 60 5.1 Main test environment for analysis of errors in gigabit ¯bre links . 77 5.2 Flowchart of real-time ¯bre link test software, tcp¯redi® ............. 78 5.3 Contrasting packet-error and bit-error rates versus received power . 82 5.4 Error positions for frames of 46 octets in length containing uniform data . 83 5.5 Error positions for frames of 1492 octets in length containing uniform data .