A new Janet Access Infrastructure James Blessing Rob Evans Jeremy Sharp History & approach

Jeremy Sharp, Janet Infrastructure Director 14-May-2015 v06 / jas CurrentJANET6 BACKB OJanetNE OVERVIEW –architecture Oct 2015

North East Scotland

South East Scotland Access layer

Northern South West Glasgow Ireland Scotland

Yorkshire & Regional infrastructure Cumbria North West North East Humberside

Manchester National (UK) backbone

Birmingham Nottingham

PSBA West East Bristol Wales Midlands Midlands shared datacentres GÉANT peerings & South West London gateways

London London South EastERN West East 1 global Global Thames KPSN Valley Kent London London NRENs Docklands East 2

Core PoP (Point of Presence) Regional network Transmission site A new Janet Access infrastructure

Overall aim: to reduce complexity of Janet > Access layer access infrastructure, including: > Standardise equipment Regional infrastructure > Overarching management platform > Some estate rationalisation National (UK) backbone > More use of fibre are we where – Including future products – Increase self-provisioning Access layer > …and break the cycle of re-procuring regional network telco contracts every 4-6 years National (UK) backbone what we want we what Steps towards a new Janet Access infrastructure >2017-Q2/3: Procurement for access provisioning strategy >Competitive dialogue >Range of approaches invited >Proposals mainly based around suppliers’ existing network infrastructures >Expensive - lacking flexibility & agility around provisioning & upgrade >Compared against Jisc build and operate approach (as with Janet6 backbone) –Self-build better meets overall aims Access infrastructure – generic approach

Co-Lo x Aggregation Point Router Site 1 A Site Access circuit/fibre 2a Core RNEP Fibre backhaul 1 A Site Co-Lo Co-Lo Co-Lo 2b 1 2 3 Core x Janet core PoP G.8032 Ethernet Ring Janet connected Co-Lo Co-Lo Co-Lo Site 3 6 5 4 organisation G.8032 Core RNEP Sub- 2 B Ring Co-Lo Co-Lo Site 3 Jisc managed Ethernet/optical Router 4b 4a B equipment used to light fibre and terminate circuits Building the new Janet Access infrastructure

>Components: > Regulated Openreach products (EAD, OSA, OSA Filter Connect, DFA, Exchange space) > Other telco fibre & col-lo: DPS framework > Optical/Ethernet switching equipment, support services: New Ciena framework > Other transmission requirements: Renew Janet telco framework (summer 2018)

>New access infrastructure introduced as regional network contractual arrangements expire over next 3-4 years. UK Telecoms Regulation (a beginners guide as to why we can't have nice things)

James Blessing, Deputy Director, Network Architect BT Group

BT Group

BT BT Retail Wholesale& Ventures Openreach

BT Business & Public BT Consumer Sector EE BT Sport BT Wholesale BT Ventures

Sell to End User Sell to Service Providers TLAs* >EAD – Ethernet Access Direct >OSA – Optical Spectrum Access >OSEA – Optical Spectrum Enhanced Access >DFA – Dark Fibre Access

*Not all the three letter acronyms used in this presentation, just some that are important right now >Regulator for Telecom >And other communication things like TV & Radio >Market Reviews >Narrowband, Business Communications, Wholesale Local Markets, Spectrum, Broadcasting, etc >Protect "people" from providers with significant market power (i.e. mostly BT Group, but also Kingston Communications in Hull) >BCMR 2016 >DFA Openreach DFA

> Point to Point Dark Fibre > Three parts > Site -> Local Telephone exchange > Exchange -> Exchange > Remote exchange -> JISC POP (core or RNEP) > Can be used to make backhaul networks > Can't be used for "national" networks (it's an access product) > Regulated, fixed price, nationally available Legal

> BT Group don't agree with BCMR > Wrong market geographical definition > 1G not same as 10G > Pricing is wrong (too cheap) > Take to Competition Appeals Tribunal > CAT Agrees with BT :( > Ofcom propose a 1G DFA option

> Openreach respond with OSA FC

Building using OSA FC

Need a national solution, national DF not (currently) an option

Openreach Telephone Exchanges as POPs

Effectively OSA FC is an optical layer, so can focus on Ethernet

Establish DF framework, use where more cost effective that OSA FC

Move to DFA if/when it becomes available Ethernet Platform

> Needs to support 100G backhaul > 100G Access as part of the upgrade path > Needs to support 1G/10G access > Sub 1G > Build on a 'family' of hardware to reduce complexity

> Options > Traditional Vendor > White/Brite box DF Framework

> Use as comparator against OSA FC >I.e. each new link request a DF alternative > Request "partial" solution for a region > Sit in room and build based on OSA FC, per link and full solution What we’re deploying

Rob Evans, Chief Network Architect Example topology Router Site 1 A Site 2a Core RNEP 1 A Site Co-Lo Co-Lo Co-Lo 2b 1 2 3

G.8032 Ethernet Ring

Co-Lo Co-Lo Co-Lo 6 5 4 G.8032 Core RNEP Sub-Ring 2 B Co-Lo Co-Lo Site 3 Router 4b 4a B Equipment options

> Ciena 5170 > Regional Aggregation or 100G NTE – I.e. in the co-location PoPs and RNEPs > 1 rack unit > 4 x 100GE – Scalable up to a 200G regional aggregation ring > 40 x 10GE – Access, or multiple of 10GE for lower-demand sub-rings > To be released: DWDM converter shelf for coherent 100GE – 100G can’t be used over OSA FC until that’s available Equipment options > Ciena 8700 > 10 slot shown, 4 slot to be deployed > Regional aggregation where we can’t wait for the 5170 coherent shelf Equipment options: NTE

> Ciena 3903 / 3928 > 3903 – 2 network-facing 1G ports, 2 user-facing 1G ports > 3928 – 4 x 1G / 10G SFP ports – 8 x 1G SFP ports Equipment options: Optical > Ciena 6500 > When capacity exceeds 5170/8700 > Same device used on the Janet backbone > 100GE coherent What will this look like? IP service delivery Router Site 1 A Site 2a Core RNEP 1 A Site Co-Lo Co-Lo Co-Lo 2b 1 2 3

G.8032 Ethernet Ring

Co-Lo Co-Lo Co-Lo 6 5 4 G.8032 Core RNEP Sub-Ring 2 B Co-Lo Co-Lo Site 3 Router 4b 4a B PoP to PoP

Site 1 Site 2 Site 3

PoP A PoP B Ethernet ring protection >G.8032/Y.1344 >Less than 50ms protection switch – (Assuming fibre ring < 1,200km!) >Protection at layer 2, so do not need to wait for IP to reconverge on link failure Ethernet Diagnostics – Improved OAM >Check circuit before being brought into service >RFC2544 / Y.1564 – SLA testing/validation – Loss – Delay – Throughput >Performance monitoring >CFM / Y.1731 - Connectivity Fault Management – Delay – Jitter – Loss >Improved visibility of Netpaths through the aggregation infrastructure Presentation >Services delimited by VLANs >Or additional ports on the NTE >Services might be >IP >ExpressRoute >Netpath What can we do in the future? Automation >Ciena MCP >“Manage, Control, Plan” >Formerly “Blue Planet MCP” by Cyan >Point-and-click provisioning around a single region >Netpath provisioning through the core needs to be “stitched” manually >APIs into MCP and the backbone routers could enable automation of this >”Zero touch provisioning” for NTEs Scalability >If using OSA FC, upgrades will be down to engineer time. >New port on the 5170, re-use the NTE if it’s a 3928. >No involvement from Openreach >Upgrades to regional aggregation rings >Insert SFPs, connect to the OSA FC filters. Network Function Virtualisation

>3926m >NTE with option for embedded x86 processor >Virtual machines for firewall, DPI, … ? Jisc in the UK e-Infrastructure context

>HEFCE – gone >UKRI ( .ukri.org ) – Research England (RE) >OfS (.officeforstudents.org.uk ) – independent public body reporting to parliament >Jisc governance….

>Detail not for this meeting.. >But - strategic links to UKRI and future e-Infrastructure discussions as noted by Dave and Pete this morning Jisc Board of Trustees

> Professor David Maguire (chair) - vice-chancellor, University of Greenwich > Professor Liz Barnes - vice-chancellor and chief executive at Staffordshire University - appointed by Universities UK > Susan Bowen - vice-president and general manager, Cogeco Peer 1 > Dr Paul Feldman - chief executive, Jisc > Professor Philip Gummett OBE - retired chief executive, Higher Education Funding Council for Wales - appointed by our funders > Robin Ghurbhurun, chief executive and principal of Richmond upon Thames College - appointed by AoC > Professor Paul Layzell - principal, Royal Holloway, University of London > Rob McWilliam - previously vice-president of consumables, Amazon UK > Professor Mark Smith - vice-chancellor, Lancaster University > Professor Nigel Seaton, principal and vice-chancellor, Abertay University - appointed by GuildHE > Dr Ken Thomson - principal, Forth Valley College > Professor Anne Trefethen - pro vice-chancellor and chief information officer, University of Oxford Jisc Funders and Owners Group

> Advice to Trustees and Jisc management > Professor David Maguire, University of Greenwich (Chair) > David Beards, Scottish Funding Council > Steve Butcher, Office for Students > John dePury, Universities UK (UUK) > David Hughes, Association of Colleges (AoC) > Marian Jebb, Welsh Government > Mike Klym, Department for Education, England > Lynne Miskelly, Department of Economy, Northern Ireland > Gordon Mckenzie, GuildHE > Dr Cliona O'Neill, Higher Education Funding Council for Wales (HEFCW) > David Sweeney, UK Research and Innovation (UKRI)/Research England Janet network connection for Jisc members

>Primary – capacity as needed (still the policy) >Secondary (resilience) – 1Gbit/s bundled >Additional capacity - £10k flat-rate Thinking about connecting at the 100Gbit/s scale

> This is huge capacity – how to structure & present it ? (mix IP, L2…) > RAL is now operational at 100Gbit/s resilient IP > plus the 10Gbit/s paths RAL-CERN etc. > May now be appropriate for some University sites > Imperial…others… > DiRAC – Cambridge, Leicester, Durham – Edinburgh in due course – may be a mix of 100G and 40 G > Help to build research case & Jisc/Janet technical and funding implications – Involve Infrastructure teams and research group representatives > Look at implications – not a trivial upgrade – Funding, new equipment, appropriate LAN architecture Research Engagement…

>GÉANT TF-RED >Task force on Research Engagement Development >I now chair this >Janet/Jisc, Surfnet, DFN, ESNET, Internet2…. >Interested in how this is done within your Universities >Internal outreach – communicating with real users.. >NRENs, EU Projects (EOSC etc…)