The Oslo Børs cash equities and fixed income markets migration to Millennium Exchange

OSLMIT 602 Network Guide

Issue 1.0 23 February 2012

Important note This document has been produced by Oslo Børs to assist customers of the Oslo Børs markets throughout the migration to Millennium Exchange. Part of the documentation is based on documentation from Millennium IT and the London Stock Exchange Group. Where the document title includes a number (i.e. OSLMIT 201), the number corresponds to similar documentation from The London Stock Exchange and/or from Borsa Italiana. For more details, please see references in the appendix.

If you have any general queries relating to this document, please email: [email protected]

Further copies of this document can also be downloaded from the Oslo Børs website at the following link: http://www.oslobors.no/ob_eng/Oslo-Boers/Trade/Delta/Millennium- Exchange/Technical-documentation

Disclaimer This document has been prepared on the basis of the best information available. Oslo Børs has taken reasonable efforts to ensure that the information in this publication is correct at the time of publication, but shall not be liable for decisions made in reliance on it. Oslo Børs will seek to provide notice to customers of changes being made to this document, but this notice cannot be guaranteed. Therefore, please note that this publication may be updated at any time. The information contained is therefore for guidance only. This document does not form part of the contractual documentation between the Oslo Børs and its customers.

Oslo Børs | OSLMIT 602 Network Guide 2

Change log This document can be updated at any time, and has been through the following iterations:

Issue Date Description

1.0 23 February 2012 Initial document release

Please note that only the latest issue of this document will be available from the Oslo Børs website. Details of the changes made in each issue of the document are described in the appendix.

Oslo Børs | OSLMIT 602 Network Guide 3

Content

Important note ...... 2 Disclaimer ...... 2 Change log ...... 3 Content ...... 4 1. Introduction ...... 5 1.1 Purpose ...... 5 1.2 Readership ...... 5 1.3 Document series ...... 5 2. Technical Configuration ...... 7 2.1 Service Description ...... 7 2.2 Network Connectivity ...... 8 Multicast: ...... 9 Unicast: ...... 9 2.3 Network Recommendations ...... 9 2.4 Customer IP Addressing ...... 9 2.5 Traffic Flow Diagram ...... 10 2.6 Bandwidth and Buffers ...... 11 2.7 Failure Documentation ...... 11 Appendix 1 – Issue updates ...... 13 Issue 1.0 – Released 23 February 2012 ...... 13 Appendix 2 – References ...... 14

Oslo Børs | OSLMIT 602 Network Guide 4

1. Introduction Oslo Børs is migrating its equity and fixed income trading from the TradElect/Infolect platform to Millennium Exchange, a flexible, highly scalable trading platform with ultra- low latency from MillenniumIT, a part of the London Stock Exchange Group. Oslo Børs aims to strengthen its competitive position by migrating to this new technology, and is pleased to offer customers a trading system of the highest standard that offers all the functions one might expect from a state-of-the-art trading platform.

The migration to a new trading platform for the Oslo Børs equities and fixed income markets will require both market participants and their technical suppliers to make a number of changes to their systems. As a part of the migration process, Oslo Børs will publish a series of documents covering different aspects of the technical changes related to the migration to Millennium Exchange.

1.1 Purpose The purpose of this document is to serve as a guide to the network service provided with Millennium Exchange.

1.2 Readership This document is meant to be an advisory guideline, based on experiences made by the LSE during their operation of the Millennium trading system. It is important to note that trading systems are highly dynamic and forecasting is difficult. It is important to plan the infrastructure to be flexible and scalable if rapid changes should occur.

This document is particularly relevant to technical staff within the Oslo Børs members, information vendors and network providers.

1.3 Document series This document is a part of a series of documents providing a holistic view of full trading and information services available from Oslo Børs post the migration to the Millennium Exchange platform.

The current series of documents are outlined below:

General OSLMIT Oslo Børs Market Model Equities OSLMIT Oslo Børs Market Model Fixed Income

Trading OSLMIT201 Guide to New Trading system OSLMIT202 FIX Trading Gateway (5.0 SP2) OSLMIT203 Native Trading Gateway OSLMIT204 Post Trade Gateway (FIX 5.0 SP2) OSLMIT205 Drop Copy Gateway (FIX 5.0 SP2)

Market Data OSLMIT302 FIX/FAST Gateway OSLMIT303 ITCH Gateway OSLMIT306 FIX/FAST News and Indices Gateway OSLMIT401 Reference Data

Oslo Børs | OSLMIT 602 Network Guide 5

Other OSLMIT501 Testing OSLMIT502 Guide to Application Certification OSLMIT504 Guide to Customer Enablement OSLMIT601 Guide to Trading Services Disaster Recovery OSLMIT602 Network Guide OSLMIT603 Business Parameters OSLMIT604 Technical Parameters OSLMIT605 Live Environment Connectivity OSLMIT606 CDS Environment Connectivity

The latest version of this documentation series can be found on: http://www.oslobors.no/ob_eng/Oslo-Boers/Trade/Delta/Millennium-Exchange/Technical- documentation

Oslo Børs | OSLMIT 602 Network Guide 6

2. Technical Configuration There will be need for some network infrastructure changes as a part of Oslo Børs migration to facilitate the new and faster trading system, Millennium Exchange. Parts of the changes will also introduce new functionality to reduce the need for re-requesting of lost messages.

2.1 Service Description The Millennium trading system will require a faster and more reliable infrastructure to facilitate the speed and functionality in the system. The system will use UDP multicast for fast distribution of feed data, and TCP/IP unicast for trading, re-request and other two- way communications.

Due to the fact that a faster system will lose more messages on a short interval than a slower system, the Millennium system introduces a new fail restrictive action for preventing customer loss of multicast feed data. The principal behind this solution is to give all connecting clients access to two identical streams and continuous streams of messages. If a message is lost on one stream, it can be retrieved on the other stream. This will decrease the latency added for customers when re-requesting lost messages, and the trading system will spend less computing power serving re-requests.

Oslo Børs | OSLMIT 602 Network Guide 7

2.2 Network Connectivity

Figure 1. A and B path.

Oslo Børs | OSLMIT 602 Network Guide 8

Multicast: Millennium introduces two identical diverse multicast data streams so that, if implemented correctly, the need for re-requests from the trading applications should be reduced. The two simultaneous message streams (A- and B-stream) will be delivered to each customer through their network provider. The member should make an effort to keep the network paths for the A- and B-streams separate all the way to their trading applications to gain full benefit from this implementation.

Multicast IP addresses for A- and B-stream will be provided later.

Unicast: There will be no changes from TradElect in the way TCP unicast is implemented other than new IP addresses.

2.3 Network Recommendations It is recommended that the A- and B-streams have separate, diverse paths throughout the infrastructure. It is recommended that members prioritize multicast traffic with regards to router buffers. It is recommended that packet losses must be kept at an absolute minimum because the re-request possibilities are limited. It is recommended that members plan ahead of time when creating a new network structure, to give all parties time to create, register and enable the correct connections and addresses. It is recommended to start the dialog with the network providers before starting any system development to ensure that the network layout is taken into account when designing the application. It is recommended that each customer have all their IP addresses within the same IP range. This will make enablement work and fault searching easier.

2.4 Customer IP Addressing Each member will get their own IP addresses from their selected network provider. Each service provider will be responsible for giving each client application a unique IP address that has access to Millennium. The customers must use separate IP addresses for the Production and CDS environments.

Each address that is approved for connection to Millennium must be enabled for application access. To ensure that this processes flows as easy and quickly as possible, we recommend members to order new addresses in reasonable time in advance of actual use.

Oslo Børs | OSLMIT 602 Network Guide 9

2.5 Traffic Flow Diagram

Figure 2. A and B path traffic flow.

Oslo Børs | OSLMIT 602 Network Guide 10

The A-stream will be delivered from the Oslo Børs Primary Data Center (PDC) and the B- stream from the Secondary Data Center (SDC). This ensures diversity and redundancy so that trading applications will have maximum possibility of receiving all multicast data, and reducing the need to re-request missing messages because of the two feeds containing identical data.

In London the PDC is the active site for both the A- and B-stream. The A- and B-stream network paths use separate communications lines in the backbone to Oslo. The SDC is considered to be a “cold failover”. If the PDC goes un-operational, disaster recovery procedures will make SDC operational within 2 hours. In this scenario only B-stream will be delivered.

2.6 Bandwidth and Buffers To ensure transportation of data correctly, both the A- and B-stream need a minimum of 20 Mbps of bandwidth. Oslo Børs estimates that 20 Mbps for each network path is sufficient for all services provided by Oslo Børs, including SOLA traffic. All routers included in the multicast topology must ensure that their buffers are set high enough to handle big intra-second bursts. A large router buffer will prevent the router from dropping packets in high traffic periods. Routers will normally drop packets when experiencing full buffers.

It is extremely important that customers do take utmost care to prevent package loss, due to the daily limit of performed re-request in Millennium.

LSE expects network burst to be around 60 Mbps for Oslo Børs, reference data might burst around 200 Mbps. Bursts will last less than 1 second. The damage occurs at 1-200 milliseconds if buffers are too low.

It is expected that members implement a LAN infrastructure which is not a bottleneck compared to the WAN infrastructure and the London backbone. 1 Gbps throughput on the LAN, routers, firewalls etc. connecting to the network provider is recommended.

2.7 Failure Documentation Both members and network providers should implement methods to document their delivery of multicast. This documentation will be a helpful tool to pinpoint problem areas if continuous packet loss occurs.

Documentation of multicast delivery can be done by implementing the use of pcap/tcpdump capture points with a network tap or port monitor on the network equipment handling the multicast messages. Pcap files are easily shared and easy to compare with files from other locations. Capture points are implemented using either port monitor functionality on a switch or using a network tap device.

Figure 3 shows an example of a Linux pcap server receiving the A-stream on interface eth0 and the B-stream on eth1. The streams are forwarding by the use of a monitor command on a Cisco switch (similar functionality is included on most switches for enterprise use). Network providers who use tcpdump must specify to store only the «public» multicast addresses.

Oslo Børs | OSLMIT 602 Network Guide 11

Figure 3. Interface example.

Oslo Børs | OSLMIT 602 Network Guide 12

Appendix 1 – Issue updates

This appendix describes the details of the changes made in each issue of this document.

Issue 1.0 – Released 23 February 2012 Initial release of the document.

Oslo Børs | OSLMIT 602 Network Guide 13

Appendix 2 – References

References to external resources are given below. Technical information for Millennium Exchange from the London Stock Exchange: http://www.londonstockexchange.com/products-and-services/millennium- exchange/technicalinformation/technicalinformation.htm

Millennium Migration for Borsa Italiana: http://www.borsaitaliana.it/borsaitaliana/intermediari/gestione- mercati/migrazionemillenniumit-mit/millenniumitmigration.en.htm

Oslo Børs | OSLMIT 602 Network Guide 14