SKA1 LOW LINFRA DETAIL DESIGN REPORT Revision: 4.0 Released: 2018-11-20

SKA1 LOW LINFRA DETAIL DESIGN REPORT

Document Number SKA-TEL-SADT-0000260-DDD Document Type DRE Revision 4.0 Author M. Tearle Date 2018-11-20 Status RELEASE

Name Designation Affiliation Signature Owned by:

Infrastructure University of Mark Tearle Mark Tearle Engineer Manchester Mark Tearle (Nov 20, 2018)

Approved by:

AEON SADT Systems Rob Gabrielczyk Engineering Robert Gabrielczyk Engineer Robert Gabrielczyk (Nov 21, 2018) Ltd.

SADT Project University of Mike Pearson MH Pearson Manager Manchester MH Pearson (Nov 21, 2018)

Released by:

SADT University of Keith Grainge Keith Grainge Consortium Lead Manchester Keith Grainge (Nov 21, 2018)

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

DOCUMENT HISTORY

Revision Date Of Issue Engineering Change Comments Number

0.1 2016-07-04 Initial revision copied up to Sharepoint. Document metadata change only.

0.2 2016-07-17 First DRAFT completed for Perth Face To Face Meeting

0.3 2016-08-04 Incorporation of comments post Perth Face To Face Meeting

0.4 2017-03-23 Major revision into DDD template

1.3 2017-03-29 Incorporate Network Layout Document updates and feedback from Jill Hammond

1.4 2017-04-10 Complete Verification, Assumptions/Risks sand Safety sections

1.5 2017-04-17 Complete Maintenance

1.6 2017-04-17 Complete Integration, Training, Implementation sections

1.7 2017-04-18 Complete Inside Plant, RAMS, Deliverables

1.8 2017-05-08 Corrections throughout document

1.9 2017-05-26 Update STFR.FRQ fibre requirements (loss) §4.1.3.1 Update Security §4.1.9 Note UPSC

1.10 2017-06-01 Revise §4.1.4 Inside Plant

1.11 2017-07-17 Fix typos and edits after review Link out to MDAL Flesh out §6 sections Incorporate Fibre Gap

1.12 2017-09-11 SADT17-00162 Amend document title and number

1.13 2017-09-19 SADT17-00160 Incorporate KVM

1.14 2017-10-16 Add clarified Fibre Gap drawing – Figure 42

1.15 2017-10-18 Updates from review 9/10/2017 Fix various formatting issues Update text around Fibre Gap requirements and implementation Update text around trenches Amend OTDR testing

2018-11-20 Page 2 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Revision Date Of Issue Engineering Change Comments Number

1.16 2017-10-20 Updates from review 9/10/2017 Text corrections Environmental management Perceived risks and site visits Text around final review (Outstanding Work) Expand on Recommendations

1.17 2017-10-23 SADT17-00178 SAT.STFR.FRQ Down-select Remove references to UWA system

1.18 2017-10-25 Revise / update LOW fibre requirements (Table 4)

1.19 2017-11-02 ECP-170010 Remove DDBH from LOW design and substitute SADT17-00194 for anticipated LFAA replacement Update LOW ODF arrangements

1.20 2017-11-08 Update LOW ODF arrangements

1.21 2017-11-15 SADT17-00207 Change of optics for SAT.STFR.UTC. Updates to §4.1.3.1 and §4.1.3.3.4

1.22 2017-11-17 Add Cable Management Minor language tidyups

1.23 2017-11-17 SADT17-00013 Adopt use of SPC over SPF

1.24 2017-11-21 Update interoperability and requirements validation

1.25 2017-11-23 SADT17-00195 RPF and Observatory UPS addition/alignment SADT17-00209

1.26 2017-11-23 Reference documents updates Tidy up AIV text – configurations and future work Review fibre requirements Review and update acronyms / definitions

1.27 2017-11-30 SADT17-00220 Add CINs for Special Test Equipment Remove placeholders for proposed appendices

1.28 2017-12-05 SADT17-00177 Add NREN Interconnect Fibre

1.29 2017-12-07 Add document numbers for Reference Drawings Correct equipment location matrix Reference out to RAMS and ILS reports, and associated FMECA/RAMS/ILS text updates

1.30 2017-12-08 Updates based on comments from P. Boven Revise Table 21 Add baseline statement

1.31 2017-12-12 Update internal plant table Clarify AIV arrangements Text clarifications

2018-11-20 Page 3 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Revision Date Of Issue Engineering Change Comments Number

1.32 2017-12-14 Add fibre loss calculation methodology in Appendix

1.33 2017-12-15 SADT17-00174 Add GNSS Fibres and Shelter

1.34 2017-12-15 Tidy up document references

1.35 2018-02-05 Incorporate corrections and feedback from Bruce Wallace Incorporate feedback from Keith Grainge Update Security Risk Assessment document numbers

1.36 2018-02-19 Address CDR Review comments from BHW, SL, and RO (Pass 1)

1.37 2018-02-21 Address CDR Review comments from BHW, SL, and RO (Pass 2)

1.38 2018-02-22 Address CDR Review comments from BHW, SL, and RO (Pass 3)

2.0 2018-02-23 - MH: Signature preparation, major release for SADT CDR delivery to the SKAO. 2.1 2018-05-30 ECP-180006 NWA-403 – Reference for assessment of vendors against safety requirements (changes to §4.9.4.1.3) 2.2 2018-05-30 ECP-180006 Internal OARs (INT428, INT429, INT430, INT431, INT432, INT433, INT434, INT435, INT437) – Mirror corrections to grammar and typos 2.3 2018-06-13 ECP-180006 NWA-51, NWA-52 Confirm document versions and update Applicable and Reference documents INT572 Update Figure 1 2.4 2018-06-14 ECP-180006 NWA-46 §4.3.3.3 §6.2.2 Clarify survey arrangements for cable routes 2.5 2018-06-14 ECP-180006 NWA-53, NWA-196 §4.3.2.2 §4.5.2 Clarify arrangements for crossing under roads and watercourses 2.6 2018-06-15 ECP-180006 NWA-57 §6.2.3 §6.2.4 Expand on areas which require further work to achieve full alignment 2.7 2018-06-19 ECP-180006 NWA-58 §4.3 (including subsections) Add fibre connection to Power Station 2.8 2018-06-22 ECP-180006 NWA-76 INT586 §4.9.1.2 Add information about cabling guidance 2.9 2018-06-28 ECP-180006 INT590 §4.2 Add Maser Room Rack 2.10 2018-07-05 ECP-180006 INT625 Update reference to Detailed Cost Model [RD8] SKA-TEL-SADT-0000701

2018-11-20 Page 4 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Revision Date Of Issue Engineering Change Comments Number

2.11 2018-07-16 MH: Amendments to revision numbers referenced for AD3, AD6, RD8, RD9, RD19, RD21, RD29, RD30 3.0 2018-07-16 MH: Signature preparation, major release for SADT CDR delivery to the SKAO in final July submission. 3.1 2018-09-25 - SADT-CM MH: Removal of document revision numbers from AD1 to AD10, AD12 to AD15 (1.3.1) and RD1 to RD4, RD6, RD8, RD11, RD13 to RD16, RD18 to RD24, RD26 to RD28, RD31, RD32, RD39, RD40 (1.3.2), as directed by SKAO-EPM/CM. 3.2 2018-11-20 201118 SADT-CM: Changes made to FRQ product names to align with SADT CDR baseline objectives, as agreed with SKAO: SAT-STFR.THU to SAT.STFR.FRQ-LOW pg’s 35/36 Tb6 & THU changed to low in 1.18 Document History. Document title and references unchanged as agreed. 4.0 2018-11-20 SADT-CM MH: Signature preparation, major release for SADT CDR delivery to the SKAO. November submission for CDR closeout.

DOCUMENT SOFTWARE

Package Version Filename

Wordprocessor Word 2010 SKA-TEL-SADT-0000260_DDD_LINFRADetailDesignDocument(LOW).docx

ORGANISATION DETAILS Name University of Manchester Registered Address Jodrell Bank Centre for Astrophysics Alan Turing Building The University of Manchester Oxford Road Manchester, UK M13 9PL Fax. +44 (0)161 275 4247 Website www.manchester.ac.uk

2018-11-20 Page 5 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

TABLE OF CONTENTS 1 EXECUTIVE SUMMARY...... 13 2 INTRODUCTORY SECTIONS ...... 14

2.1 PURPOSE ...... 14 2.2 SCOPE ...... 14 2.3 INTENDED AUDIENCE ...... 15 2.4 APPLICABLE AND REFERENCE DOCUMENTS ...... 15 Applicable Documents ...... 15 Reference Documents ...... 16 Applicable Standards ...... 17 Reference Drawings...... 19 3 ACRONYMS AND DEFINITION OF TERMS ...... 20

3.1 ACRONYMS ...... 20 3.2 DEFINITION OF TERMS ...... 23 Cabinet ...... 23 Cable Vault ...... 23 Drop Splice ...... 23 Enclosure ...... 23 Inside Plant ...... 23 KVM Switch ...... 23 Optical Distribution Frame (ODF) ...... 23 Outside Plant ...... 23 Patch Field ...... 23 Pathway ...... 23 Pit ...... 23 Pluggable Optic ...... 24 Power Distribution Unit (PDU) ...... 24 Splice ...... 24 Rack ...... 24 Uninterruptible Power Supply (UPS) ...... 24 4 SOLUTION DESCRIPTION...... 25

4.1 PRODUCTS ...... 28 4.2 EQUIPMENT-LOCATION MATRIX (NODES) ...... 28 4.3 SKA1 LOW OUTSIDE PLANT FIBRE RETICULATION DESIGN ...... 33 User Requirements ...... 33 4.3.1.1 Requirements from the CPF to each RPF ...... 37 4.3.1.2 Requirements from the CPF to the Power Station ...... 38 Overview of the SKA1 LOW External Fibre Reticulation Design...... 39 4.3.2.1 Cable reticulation at the CPF ...... 40 4.3.2.2 Cable backhaul along the spirals...... 45 4.3.2.3 Cable backhaul for the Power Station ...... 51 4.3.2.4 Cable reticulation into each RPF ...... 51 4.3.2.5 Cable reticulation into the Power Station ...... 54 Analysis of the LOW Fibre Reticulation Design ...... 55 4.3.3.1 Site Constraints ...... 55 4.3.3.2 Equipment Constraints ...... 59 4.3.3.3 Cable reticulation...... 61 4.3.3.4 Distance and Optical Constraints ...... 62 4.3.3.5 Technologies and components for the design ...... 65 4.3.3.6 Compliance with User requirements ...... 72 4.4 SKA1 LOW INSIDE PLANT – FIBRE AND OTHER CABLE DISTRIBUTION ...... 73 User requirements ...... 73

2018-11-20 Page 6 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Design of Internal Fibre and Other Cable Distribution at CPF ...... 78 4.4.2.1 Overview ...... 78 4.4.2.2 Inter-room connectivity ...... 78 4.4.2.3 Inter-row connectivity (Cabinet Room only) ...... 80 4.4.2.4 Intra-row connectivity ...... 80 Design of Internal Fibre and Other Cable Distribution at RPF ...... 81 Design of Internal Fibre and Other Cable Distribution at EOC ...... 81 Design of Internal Fibre and Other Cable Distribution at SPC ...... 82 Design of Internal Fibre and Other Cable Distribution at SOC ...... 82 Analysis of SKA1 LOW Internal Fibre and Other Cable Distribution ...... 82 4.4.7.1 Site Constraints ...... 82 4.4.7.2 Equipment Constraints and Selection of technologies for the design ...... 82 4.4.7.3 Cable reticulation...... 82 4.4.7.4 Compliance with User requirements ...... 83 4.5 SKA1 LOW FIBRE FOR CSP-SDP CONNECTIVITY ...... 84 User Requirements ...... 84 Summary of proposed implementation ...... 85 4.6 SKA1 LOW FIBRE AND ENCLOSURES FOR GNSS EQUIPMENT ...... 87 User Requirements ...... 87 Summary of proposed implementation ...... 87 4.6.2.1 Optical fibre for GNSS signals ...... 87 4.6.2.2 GNSS Calibration Shelter ...... 88 4.6.2.3 Optical fibre for reference signals to GNSS Calibration Shelter ...... 88 4.7 ANCILLARY ITEMS ...... 89 User requirements ...... 89 Overview ...... 89 4.7.2.1 Cable Pathways ...... 89 4.7.2.2 Racks and Cabinets ...... 91 4.7.2.3 Cable Management ...... 93 4.7.2.4 Blanking and Ventilation Panels ...... 94 4.7.2.5 Power Distribution ...... 95 4.7.2.6 Uninterruptable Power Supplies ...... 97 4.7.2.7 KVM Switch ...... 99 4.7.2.8 Earthing ...... 99 4.7.2.9 Test Equipment ...... 100 4.7.2.10 Cabling Installation Records ...... 100 Analysis and Compliance with User Requirements...... 101 4.8 SOFTWARE ...... 102 Overview of requirements ...... 102 4.8.1.1 Software for operation of equipment ...... 102 4.8.1.2 Software for Record Keeping ...... 102 Use Cases ...... 102 4.8.2.1 Use Case 1: Use of craft interfaces for initial configuration of PDU ...... 102 4.8.2.2 Use Case 2: Review of OTDR result in case of fault ...... 102 4.9 IMPLEMENTATION...... 104 Overview ...... 104 4.9.1.1 Outside Plant ...... 104 4.9.1.2 Internal Plant ...... 104 4.9.1.3 Ancilliary Items ...... 105 Configurations ...... 105 4.9.2.1 Configuration for AIV Array Assembly AA-1...... 105 4.9.2.2 Final ...... 106 Execution ...... 107 Verification ...... 107 4.9.4.1 Verification for Procurement ...... 107 4.9.4.2 Functional Verification ...... 109 4.10 RELIABILITY, AVAILABILITY AND MAINTENANCE...... 111 RAMS Modelling...... 111

2018-11-20 Page 7 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Logistics and Long Lead Time Items ...... 111 Maintenance Scheduling ...... 111 4.10.3.1 Fibre ...... 111 4.10.3.2 Ancillary ...... 113 Line Replaceable Units and Spares ...... 113 4.11 SECURITY ...... 115 System Boundaries ...... 115 Foreseeable Threats and Mitigations ...... 116 Recommendations and Compliance ...... 118 4.12 SAFETY ...... 124 General ...... 124 Hazardous Items and Scenarios ...... 126 4.13 MAINTENANCE ...... 129 Summary ...... 129 Maintenance Schedule ...... 129 Staffing, Training and Safety Equipment Required ...... 130 4.14 INTEGRATION ...... 134 Component Integration ...... 134 System Integration ...... 135 Wider System of Interest (WSOI) Integration...... 137 Precursor Integration ...... 138 SADT Interdependencies ...... 138 Non-SADT (External) Dependencies ...... 138 4.15 INTEROPERABILITY ...... 140 SADT Interoperability ...... 140 Non-SADT (External) Interoperability ...... 141 5 EVALUATION ...... 142

5.1 FITNESS FOR PURPOSE ...... 142 Functionalities ...... 142 Functional Opportunities ...... 143 Functional Weaknesses ...... 143 5.2 VERIFICATION ...... 144 Design Verification (Pre-CDR) ...... 144 5.2.1.1 Overview ...... 144 5.2.1.2 Calculations Undertaken for Outside Plant Fibre ...... 144 5.2.1.3 Known uncertainties and factors affecting Design Verification ...... 145 Functional Verification (Post-CDR) ...... 145 5.2.2.1 Overview ...... 145 5.2.2.2 Fibre and Pathway ...... 145 5.2.2.3 Racks and Cabinets ...... 146 5.2.2.4 Power Distribution Units, Uninterruptable Power Supplies and KVM Switches ...... 146 Validation of Requirements ...... 146 Recommendations ...... 147 5.3 SYSTEM RELIABILITY, AVAILABILITY, MAINTENANCE AND SAFETY ...... 148 Key Points ...... 148 Recommendations ...... 148 5.4 COSTS ...... 149 Summary of Costing Assumptions ...... 149 Cost Saving Opportunities ...... 149 Recommendations for Procurement...... 150 5.5 ASSUMPTIONS ...... 151 5.6 RISKS ...... 152 6 RECOMMENDATIONS AND DEVELOPMENT ROADMAP ...... 153

6.1 SUMMARY OF DELIVERABLES ...... 153 2018-11-20 Page 8 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

6.2 OUTSTANDING ISSUES/WORK ...... 153 Further design work for GNSS antennas and GNSS calibration shelter ...... 153 Completion of site survey work ...... 153 Alignment of design with work by INAU...... 154 Alignment of design with work by LFAA ...... 154 Alignment of design with work by AIV ...... 154 Arrangements for extension of the existing CSIRO/AARNet cable to the CPF ...... 155 Arrangements for the EOC, SOC and SPC facilities ...... 155 Corrections and harmonisation for System CDR process ...... 155 Transformation of design into specification for procurement ...... 155 6.3 RECOMMENDED ACTIONS ...... 155 7 STATEMENT OF COMPLIANCE...... 156 8 CONCLUSIONS/RECOMMENDATIONS...... 156

8.1 CONCLUSIONS & RECOMMENDATIONS ...... 156 9 APPENDIX A – FIBRE SPLICE DIAGRAMS...... 157

2018-11-20 Page 9 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

LIST OF FIGURES Figure 1: SKA1-LOW key locations, regions and logical cable topology ...... 26 Figure 2: SKA1-Low Array Outer Station Region. RPF, CPF and antenna station locations and cable topology ...... 27 Figure 3: Typical arrangement for each spiral ...... 39 Figure 4: Typical arrangement for Power Station ...... 40 Figure 5: Typical ODF – Commscope NG4 ...... 41 Figure 6: Splicing arrangement for RPF S8 (typical) at the ODF ...... 43 Figure 7: Optical Fibre Entry Box ...... 44 Figure 8: Cable Vault and Conduits under building – photo from ASKAP ...... 44 Figure 9: Photograph of ASKAP splice trays ...... 44 Figure 10: Photograph of ASKAP waveguide splice tray cabinet ...... 44 Figure 11: Individual ODF Tray ...... 45 Figure 12: Photograph of Waveguide...... 45 Figure 13: ODF ...... 45 Figure 14: ODF Submodule ...... 45 Figure 15: Overlay of Power and Fibre routes ...... 48 Figure 16: Cross section of shared trench (typical) ...... 49 Figure 17: Typical pit spacing ...... 50 Figure 18: Indicative 3D of Remote Processing Facility (See [RD7] Figure 35) ...... 51 Figure 19: Remote Processing Facility Splice Arrangement (RPF S6, typical) ...... 53 Figure 20: Remote Processing Facility Site Services Layout (typical) ...... 54 Figure 21: MRO Site Boundaries with RPF Locations plotted ...... 55 Figure 22: SKA1-Low Array Core Region. RPF, CPF and antenna station locations and cable topology ...... 56 Figure 23: SKA1-Low Array Outer Station Region. RPF, CPF and antenna station locations and cable topology ...... 56 Figure 24: Schematic diagram and cumulative fibre distances to each RPF ...... 57 Figure 25: HV Single Line Diagram provided by INAU for PDR ...... 58 Figure 26: Fibre Loss Calculation for RPF N16 (generated by [RD8]) ...... 60 Figure 27: Fibre Loss Input Table to Solution Modelling ...... 64 Figure 28: Typical Loose Tube Fibre Optic Cable Outdoor Rated ...... 65 Figure 29: Typical 1 and 2 RU Fibre Patch Panels installed at MRO ...... 66 Figure 30: Optic Fibre Entry Design – INAU concept ...... 66 Figure 31: Cable vault outside (left of image) ASKAP Control Building ...... 67 Figure 32: Single core splice tray (typical) ...... 67 Figure 33: Splice tray holding ribbon fibre splices ...... 67 Figure 34: Typical Optical Distribution Frame (ODF) ...... 68 Figure 35: Coils of HPDE conduit ...... 68 Figure 36: PVC Communications Conduit ...... 69 Figure 37: Marker posts besides AARNet pit at MRO ...... 69 Figure 38: Typical Australian P5 polyethylene pit (shown with concrete lid) ...... 70 Figure 39: Typical Australian P6 polyethylene pit (shown with concrete lids) ...... 70 Figure 40: Typical gasket fitted in plastic pit ...... 70 Figure 41: Underground fibre optic splice and jointing enclosure ...... 71 Figure 42: Splice enclosure installed in P8 pit (typical) ...... 71 Figure 43: Patch lead labels ...... 71 Figure 44: Outside Plant cable tag ...... 71 Figure 45: Underground Marker Tape...... 72 Figure 46: Photograph of overhead cable tray ...... 78 Figure 47: Typical inter-room cabinet elevation ...... 79 Figure 48: Typical inter row cabinet elevation ...... 80

2018-11-20 Page 10 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Figure 49: Typical intra-row cabinet elevation ...... 81 Figure 50 Extract from [RD14] CSP-SDP Detailed Design Report showing logical connectivity ...... 84 Figure 51: Schematic overview of the fibre extension required for CSP-SDP connectivity (in blue) ...... 85 Figure 52: Geographic overview of “Fibre Gap” connection ...... 86 Figure 53: Metal Cable Tray ...... 90 Figure 54: Metal Cable Basket ...... 90 Figure 55: PVC Cable Trough ...... 90 Figure 56: Example of CPF Cabinet (including fan coil unit) ...... 91 Figure 57: Example of Open Frame Rack with Vertical Cable Manager ...... 92 Figure 58: Existing timescale installation at NPL with open frame racks...... 93 Figure 59: Typical 1RU Horizontal Cable Manager ...... 94 Figure 60: Blanking Panel ...... 94 Figure 61: Vertical PDU ...... 95 Figure 62: Horizontal PDU ...... 95 Figure 63: Typical rack mounted UPS ...... 97 Figure 64: Typical KVM Switch and Console ...... 99 Figure 65: Potential AIV enabling splice configuration at RPF S8 ...... 106 Figure 66: LINFRA Fibre to Pathway integrations ...... 134 Figure 67: LINFRA Fibre to terminations integrations ...... 135 Figure 68: System Integration for LINFRA fibre link ...... 136 Figure 69: Typical Splice Arrangement – Branching (J27 near RPF S7) ...... 157 Figure 70: Typical Splice Arrangement – On Branch (J26 near RPF S6)...... 157 Figure 71: Typical Splice Arrangement – On Trunk (J34 near RPF S15) ...... 158 Figure 72: Typical Splice Arrangement – End of Trunk (J36 near RPF S16) ...... 158

LIST OF TABLES Table 1: LINFRA Equipment Location Matrix ...... 32 Table 2: LINFRA Equipment Location Matrix – Special Test Equipment ...... 32 Table 3: External Fibre interfaces with other SKA1 LOW telescope elements ...... 33 Table 4: External Fibre interfaces with other SADT sub-elements...... 33 Table 5: Summary of external user fibre performance requirements ...... 34 Table 6: Summary of SADT sub-element user fibre performance requirements...... 36 Table 7: Summary of external user fibre core requirements ...... 37 Table 8: Summary of SADT sub-element user fibre core requirements ...... 37 Table 9: Summary of SADT sub-element user fibre core requirements ...... 38 Table 10: Summary of ODF Capacity Requirements ...... 42 Table 11: Chart of cumulative distance and link loss for each RPF on each spiral ...... 63 Table 12: Chart of cumulative distance and link loss for the Power Station...... 64 Table 13: Inside Plant interfaces with other telescope elements ...... 73 Table 14: Inside Plant interfaces with other SADT sub-elements ...... 73 Table 15: Key Connectivity at CPF – from ODF ...... 74 Table 16: Key Connectivity at CPF - Direct ...... 75 Table 17: Key Connectivity at RPF ...... 76 Table 18: Key Connectivity at EOC ...... 76 Table 19: Key Connectivity at SPC ...... 77 Table 20: Key Connectivity at SOC ...... 77 Table 21: Summary of Inside Plant Interfaces ...... 83 Table 22: Definition of States and Modes ...... 96 Table 23: Definition of conditions ...... 96

2018-11-20 Page 11 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Table 24: Definition of system states ...... 96 Table 25: UPS Operational States ...... 98 Table 26: Definition of conditions ...... 98 Table 27: Definition of system states ...... 99 Table 28: ITU-T L.25 Table 1 ...... 112 Table 29: SKAO Revision 10 Security Requirements ...... 115 Table 30: The physical boundaries of the LINFRA equipment and infrastructure ...... 116 Table 31: The logical boundaries of the LINFRA equipment ...... 116 Table 32: Suggested Mitigation ...... 118 Table 33: Security features to be supported and enabled ...... 120 Table 34: Network Access Rules ...... 122 Table 35: Example risks, products and mitigations ...... 128 Table 36: Maintenance models ...... 129 Table 37: Training Requirements ...... 132 Table 38: Wider System of Interest integrations ...... 137 Table 39: IICD documents relevant to integrations ...... 137 Table 40: Internal Interface Control Documents with other SADT sub-elements ...... 140 Table 41: External Interface Control Documents between SADT and other elements ...... 141 Table 42: Design Verifications (Pre-CDR) ...... 144 Table 43: List of non-compliances with requirements ...... 156

2018-11-20 Page 12 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

1 EXECUTIVE SUMMARY

Signal and Data Transport (SADT) provides data, timing and frequency signal transport to the telescope between major subsystems. SADT Local Infrastructure (LINFRA) provides the physical fibre network, racks, power distribution, uninterruptible power supplies (UPS) and KVM to other SADT sub-elements to enable this. These other SADT sub-elements then provide connectivity and signals to other telescope consortia (NSDN, SAT.STFR.FRQ and SAT.STFR.UTC) A key part of SADT.LINFRA is the physical fibre optic cable network which provides the physical media across which all of the following SKA1 SADT telescope networks to operate – Low Frequency Aperture Array (LFAA) data network (for the transmission of science data at the Murchison Radio Observatory (MRO)), Non Science Data Network (NSDN) (for the transport of services for telescope management and auxiliary networks such as the Building Management System), SAT.STFR.UTC and SAT.STFR.FRQ (for timing and frequency synchronisation). These networks are the core of the telescope, without which the telescope is inoperable and unable to produce the scientific results for which it has been developed. The design and successful implementation of the fibre network is paramount to the successful operation of the LOW telescope. A successful design will provide both flexibility to accommodate changing requirements as network technologies develop and additional requirements identified. The challenge is to achieve these results in the most technically acceptable, economic manner to achieve the maximum utilisation and life expectancy of the fibre network installation. The fibre reticulation will consist of single mode [AS1] ITU G.652.D fibre. Inside the Central Processing Facility (CPF), [AS2] ITU G.657.A1 and ITU G.657.A2 cable will be used where required to provide flexibility for reticulating the fibre inside the shielded equipment room where it is not possible to use ITU G.652.D compliant fibre. The purpose of this document is to:  confirm the overall site cable reticulation design and document ancillary infrastructure for the SADT LINFRA (Local Infrastructure) work package for SKA1 LOW  ensure associated co-ordination required with the work undertaken by INAU and LFAA are documented. Fibre connectivity between the CPF and the EOC is required. This is to be documented as an extension to the Central Processing Facility (CPF) from the existing fibre backhaul between the MRO and Geraldton. This backhaul is part of a larger system providing connectivity back to Perth and the NREN, AARNet. This document contains diagrams of the layout of the fibre network radiating out from the CPF, an overview of the design, and splicing arrangements. The fibre reticulation will utilise a consistent cable size along the spirals. The philosophy of this design is explained in this document. Finalisation of the design of the CPF and RPF buildings and confirmation of the exact physical locations of the various manholes and trench requirements (in conjunction with Infrastructure Australia (INAU) and LFAA) will enable further development of this design into specification for procurement.

2018-11-20 Page 13 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

2 INTRODUCTORY SECTIONS

2.1 Purpose This document describes the design for the SKA1 LOW SADT Local Infrastructure. This includes the fibre reticulation networks for the SADT element of the SKA1 LOW telescope, and additional supporting hardware and equipment required for the other SADT sub-elements. The SKA1 LOW LINFRA Detailed Design Report reflects the current baseline (Rev 3) of the SKA Programme (see [AD1]) and [AD2] Level 1 requirements as per Revision 11 at the time of writing the report. The report also reflects ECP-170010 - Cost Control Project: 5.38 Simple DDBH end-to-end connection for SKA1-Low, and ECP-170024 - SKA1 LOW Address Fibre Gap. SADT will align fibre trenching and conduit requirements with the power reticulation wherever practical and seek to share a common trench, observing any segregation required by the relevant Australian standards. Optimisation of site-wide trenching activities will be required in order to reduce cost. The telescope is laid out as a set of three spirals from a common centre point, with a core element. The spiral design was chosen as an engineering compromise to provide the maximum number of baselines and angles to achieve the science aims, whilst remaining affordable to build. The current INAU design is to construct an underground HV power distribution network with radial feeders along each spiral and a step down transformer/substation located at each Remote Processing Facility (RPF).1 The final configuration of the RPF and power network configuration will enable the detailed design of the fibre routes (providing connectivity to the RPF buildings along each spiral). For the purposes of this document, we have assumed complete alignment with the routes and an ideal arrangement for the trench in all locations.2 The SKA1 SADT fibre network is an entirely “green-field” network with minimal interfaces to existing infrastructure (communications or otherwise) on site. The type (grade) of fibre and the number of fibres has been based on the requirements specified by the other SADT sub-elements that will utilise the fibre reticulation network to carry their signals between the RPFs (Remote Processor Facilities) and the CPF (Central Processor Facility). Furthermore, there will be internal fibre distribution between SADT sub-elements inside the CPF. The supporting ancillary hardware and equipment are recommended to be standard Commercial Off The Shelf (COTS) items.

2.2 Scope This report applies only to the fibre reticulation networks and ancillary items including cable pathways, fibre cables, power distribution, UPS, KVM, earthing, and hardware required for the SADT sub-elements of SKA1 LOW telescope. These sub-elements include Science Data (CSP-SDP), Synchronisation and Timing Sub-Elements (Frequency and UTC), and the Non Science Data Network (NSDN). The NSDN network will transport the Network Manager (NMGR) and SAT Local Monitoring and Control (SAT.LMC) signals across the network for SADT. This document provides a detailed description of the SKA1 LOW LINFRA Fibre Network.

1 [RD6] SADT.LINFRA.MDAL-0204 2 [RD6] SADT.LINFRA.MDAL-0040

2018-11-20 Page 14 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

2.3 Intended Audience This design report is to inform the SKAO, SADT Consortium and any other design consortia of the design for the SKA1 LOW SADT Local Infrastructure being supplied by the SADT.LINFRA Work Package. This document is part of the SADT Consortium milestone deliverables and shall be delivered to the SKAO. The information in this document shall form the basis of the design for industry the construction procurement process.

2.4 Applicable and Reference Documents The following documents are applicable to the extent stated herein.

Applicable Documents

In the event of conflict between the contents of the applicable documents and this document, the applicable documents shall take precedence. [AD1] SKA-TEL-SKO-0000002 SKA1 SYSTEM BASELINE DESIGN V2 [AD2] SKA-TEL-SKO-0000008 SKA Phase 1System Requirements Specification [AD3] SKA-TEL-SKO-0000422 SKA1_Low_Configuration_Coordinates [AD4] SKA-TEL-SADT-0000181 SADT Product Breakdown Structure (PBS) SKA1-LOW [AD5] SKA-TEL-SKO-0000202 SKA EMI/EMC Standards, Related Procedures and Guidelines [AD6] SKA-TEL-AIV-4410001-SE-RP-MPL Roll-Out Plan for SKA1_LOW [AD7] SKA-TEL-SADT-0000638 LOW Construction Plan [AD8] SKA-TEL-SADT-00000633_REP SADT RAM Report LOW [AD9] SKA-TEL-SADT-00000635_REP SADT ILS Report LOW [AD10] SKA-TEL-SKO-0000740 SKA Project Safety Management Plan [AD11] Signal and Data Transport Group Detailed RAMS Analysis Report, FNC 49409/45693R [AD12] SKA-OFF.PAQA-SKO-QP-001 SKA Product Assurance & Safety Plan [AD13] SKA-TEL-SADT-0000107_SPE LINFRA Technical Requirements Specification (TRS) [AD14] “The Square Kilometre Array Design Reference Mission”, SKA Science Working Group, SCI- 020.010.020-DRM-002 [AD15] SKA-TEL-SADT-0000520 SADT Element Technical Requirements Specification [AD16] ECP-170010 - Cost Control Project: 5.38 Simple DDBH end-to-end connection for SKA1-Low [AD17] ECP-170024 - SKA1 LOW Address Fibre Gap

2018-11-20 Page 15 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Reference Documents

The following documents are referenced in this document. In the event of conflict between the contents of the referenced documents and this document, this document shall take precedence: [RD1] SKA-TEL-SADT-0000445_ICD LINFRA to NSDN (LOW) Internal Interface Document [RD2] SKA-TEL-SADT-0000439_ICD LINFRA to SAT.STFR.UTC (LOW) Internal Interface Document [RD3] SKA-TEL-SADT-0000440_ICD LINFRA to SAT.STFR.FRQ (LOW) Internal Interface Document (THU) [RD4] SKA-TEL-SADT-0000360_DDD SAT.STFR.FRQ (THU) Detail Design Report (LOW) [RD5] ASKAP Geotechnical reports, Aurecon - (Technical_Specifications_Geo_SKAO_ASKAP.pdf) [RD6] SKA-TEL-SADT-0000066 SADT Master Data Assumptions List (MDAL) [RD7] SKA-TEL-INAU-0000015-BLDS-MROLFAA-RE Buildings Draft Detailed Design Report, Rev 3, 4 August 2017 [RD8] SKA-TEL-SADT-0000701 LINFRA LOW Detailed Cost Model [RD9] 100-000000-026 Interface Control Document SADT to LFAA, Rev 5.0 [RD10] 100-000000-024 SKA1 LOW Telescope Interface Control Document SADT to INAU, Rev 4.0 [RD11] SKA-TEL-SADT-0000438_ICD LINFRA to CSP-SDP (LOW) Internal Interface Document [RD12] The Square Kilometre Array – Networks and Computing, Dr Shaun W Amy, 6 September 2012. Presented at AusNOG 2012. http://www.ausnog.net/sites/default/files/ausnog- 2012/presentations/ausnog-2012-d01p03-shaun-amy-csiro.pdf, Accessed 15 December 2017 [RD13] SKA-TEL-SADT-0000330_DDD SADT Clocks Detailed Design Document [RD14] SKA-TEL-SADT-0000220_DRE CSP-SDP Detail Design Report (LOW) [RD15] SKA-TEL-SADT-0000441_ICD LINFRA to SAT.CLOCKS (LOW) Internal Interface Document [RD16] SKA-TEL-SKO-0000293 SKA1 Power Quality Standard [RD17] Utility Providers Code Of Practice For Western Australia, https://www.1100.com.au/wp- content/uploads/2017/01/2016-UPSC-Code-of-Practice-Rev-1.pdf , Accessed 15 December 2017 [RD18] SKA-TEL-SADT-0000679 Risk Assessment LOW CPF SADT Network Infrastructure [RD19] SKA-TEL-SADT-0000680 Risk Assessment LOW EOC SADT Network Infrastructure [RD20] SKA-TEL-SADT-0000681 Risk Assessment LOW RPF SADT Network Infrastructure [RD21] SKA-TEL-SADT-0000682 Risk Assessment LOW SOC SADT Network Infrastructure [RD22] SKA-TEL-SADT-0000683 Risk Assessment LOW SPC SADT Network Infrastructure [RD23] SKA-TEL-SADT-0000684 Risk Assessment LOW SADT Portable Devices [RD24] SKA-TEL-SADT-0000045 Standards for SKA Networks [RD25] Murchison Radio-astronomy Observatory HSE AND SITE INFORMATION FOR CONTRACTORS, http://www.atnf.csiro.au/observers/visit/MROHSEManualForContractors_v4.pdf, Version 4.0 January 2013 , Accessed 15 December 2017 [RD26] SKA-TEL-SADT-0000423 Operational Cost Model Description [RD27] SKA-TEL-SADT-0000442_ICD LINFRA to NMGR (LOW) Internal Interface Document

2018-11-20 Page 16 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

[RD28] SKA-TEL-SADT-0000443_ICD LINFRA to SAT.LMC (LOW) Internal Interface Document [RD29] 100-000000-023 Interface Control Document SADT to CSP (LOW), Rev 4.0 [RD30] 100-000000-025 Interface Control Document SADT to SDP (LOW), Rev 5.0 [RD31] SKA-TEL-SKO-0000153 SKA1 Interface Control Document TM to SADT [RD32] SKA-TEL-SADT-0000034 SADT Risk Register [RD33] Guidance on the principles of safe design for work, Safe Work Australia, https://www.safeworkaustralia.gov.au/doc/guidance-principles-safe-design-work, Accessed 15 December 2017 [RD34] Model Code of Practice - Safe Design of Structures, Safe Work Australia https://www.safeworkaustralia.gov.au/doc/model-code-practice-safe-design-structures, Accessed 15 December 2017 [RD35] “SKA SADT Proposal”, R.Schilizzi for SADT Consortium, SKA-TEL.SADT-PROP_PROP-001-A, Rev 1, 7th June 2013 [RD36] SKA SADT Concept Generation and Down Selection Report SKA-TEL.SADT-CONCEPT-001, Rev 1, June 2014 [RD37] SKA SADT Horizon Scan Report for Work Package: LINFRA SKA-TEL.SADT-HORIZON-LINFRA-001, Rev 1, April 2014 [RD38] SKA-TEL-SADT-0000122 SADT CDR Statement of Work [RD39] SKA-TEL-SADT-0000539 SADT Network Equipment Configuration [RD40] SKA-TEL-SADT-0000621_TN SKA1 LOW Address Fibre Gap for CSP-SDP [RD41] SKA-TEL-INAU-0000015-BLDS-MRO-LFAA-RE-RE Buildings_Concept_Design_Report, Rev2, 27 March 2015 [RD42] “Network Cabling Design Best Practices: 2017” white paper, CTC Technologies Inc & Cisco, https://www.ctctechnologies.com/wp-content/uploads/2017/04/Network-Cabling-Design-Best- Practices.pdf, Access 22 June 2018

Applicable Standards

The following documents are referenced in this document. In the event of conflict between the contents of the referenced documents and this document, this document shall take precedence. [AS1] ITU-T (International Telecommunication Union) G.652: Characteristics of a single-mode optical fibre and cable [AS2] ITU-T (International Telecommunication Union) G.657: Characteristics of a bending-loss insensitive single-mode optical fibre and cable for the access network [AS3] IEC 60297-3-100:2008 Mechanical structures for electronic equipment - Dimensions of mechanical structures of the 482,6 mm (19 in) series - Part 3-100: Basic dimensions of front panels, subracks, chassis, racks and cabinets [AS4] IEEE Std 802.3-2012, IEEE Standard for Ethernet [AS5] Telcordia GR-253 OC192 LR2 [AS6] Telcordia GR-449-CORE Generic Requirements for Fiber Distributing Frames

2018-11-20 Page 17 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

[AS7] AS/NZS IEC 60825.1:2014 Safety of laser products - Equipment classification and requirements [AS8] AS/NZS IEC 60825.2:2011 Safety of laser products - Safety of optical fibre communication systems (OFCS) [AS9] AS/NZS IEC 60825.14:2011 Safety of laser products - A user's guide [AS10] AS/CA S008:2010 Requirements for Customer Cabling Products [AS11] AS 1049.1-2014 Telecommunication cables - Insulation, sheath and jacket – Materials [AS12] AS/NZS 3000:2007/Amdt 2:2012 Electrical installations (known as the Australian/New Zealand Wiring Rules) [AS13] AS 2067:2016 Substations and high voltage installations exceeding 1 kV a.c [AS14] AS/CA S009:2013 Installation Requirements For Customer Cabling (Wiring Rules) [AS15] AS/NZS 2648.1:1995 Underground marking tape - Non-detectable tape [AS16] AS/NZS 4130:2009 Polyethylene (PE) pipes for pressure applications [AS17] AS/NZS 2053.2:2001 (R2016) Conduits and fittings for electrical installations - Rigid plain conduits and fittings of insulating material [AS18] ETSI EN 300 019-1-3 V2.3.2 Environmental Engineering (EE); Environmental conditions and environmental tests for telecommunications equipment; Part 1-3: Classification of environmental conditions; Stationary use at weather protected locations [AS19] AS/NZS 60320.1:2012 Appliance couplers for household and similar general purposes - General requirements (IEC 60320-1, Ed. 2.1 (2007) MOD) [AS20] AS/NZS 60320.2.2:2004 (R2016) Appliance couplers for household and similar general purposes - - Interconnection couplers for household and similar equipment (IEC 60320-2-2, Ed. 2.0 (1998) MOD) [AS21] IEC 60309-1 Ed. 4.2 Plugs, socket-outlets and couplers for industrial purposes - Part 1: General requirements [AS22] IEC 60309-2 Ed. 4.2 Plugs, socket-outlets and couplers for industrial purposes - Part 2: Dimensional interchangeability requirements for pin and contact-tube accessories [AS23] AS 62040.1.1-2003 (R2013) Uninterruptible power systems (UPS) - General and safety requirements for UPS used in operator access areas [AS24] AS 62040.1.2-2003 (R2013) Uninterruptible power systems (UPS) - General and safety requirements for UPS used in restricted access locations [AS25] AS 62040.2-2008 Uninterruptible power systems (UPS) - Electromagnetic compatibility (EMC) requirements [AS26] AS IEC 62040.3-2012 Uninterruptible power systems (UPS)-Method of specifying the performance and test requirements [AS27] AS/NZS 3085.1:2004 (R2016) Telecommunications installations - Administration of communications cabling systems - Basic requirements [AS28] AS/NZS ISO/IEC 14763.3:2012 Telecommunications installations - Implementation and operation of customer premises cabling-Testing of optical fibre cabling (ISO/IEC 14763-3:2011, MOD) [AS29] AS/NZS ISO/IEC 17020:2013 Conformity assessment - Requirements for the operation of various types of bodies performing inspection

2018-11-20 Page 18 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

[AS30] ITU-T (International Telecommunication Union) Recommendation L.25: Construction, Installation And Protection Of Cables And Other Elements Of Outside Plant Optical Fibre Cable Network Maintenance [AS31] AS/NZS ISO 31000:2009 Risk Management – Principles and guidelines [AS32] AS/NZS 4804:2001 Occupational health and safety management systems - General guidelines on principles, systems and supporting techniques [AS33] AS/NZS 4801:2001 Occupational health and safety management systems - Specification with guidance for use [AS34] AS/NZS ISO 14001:2016 Environmental management systems - Requirements with guidance for use [AS35] AS/NZS 3760:2010 In-service safety inspection and testing of electrical equipment [AS36] AS/NZS 3666.2:2002 Air-handling and water systems of buildings - Microbial control Operation and maintenance [AS37] AS/NZS 3123:2005 (R2016) Approval and test specification - Plugs, socket-outlets and couplers for general industrial application [AS38] AS/NZS 3080:2013 Information technology - Generic cabling for customer premises (ISO/IEC 11801:2011, MOD) [AS39] ETSI EN 300 386 V2.1.1 Telecommunication network equipment; Electro Magnetic Compatibility (EMC) requirements; Harmonised Standard covering the essential requirements of the Directive 2014/30/EU [AS40] AS 60529-2004 Degrees of protection provided by enclosures (IP Code) [AS41] Australian Building Codes Board National Construction Code (NCC) 2016 Vol 1, 2 & 3 [AS42] ISO 45001:2018(en) Occupational health and safety management systems — Requirements with guidance for use

Reference Drawings

The following drawings are referenced in this document. In the event of conflict between the contents of the referenced drawings and this document, this document shall take precedence: [D1] SKA-TEL-SADT-0000650 SKA1 Low Fibre Design Schematic [D2] SKA-TEL-SADT-0000654 SKA1 Low Internal Fibre Splicing [D3] SKA-TEL-SADT-0000652 SKA1 Low Fibre Segregation [D4] SKA-TEL-SADT-0000656 SKA1 Low Pit Spacing [D5] SKA-TEL-SADT-0000649 SKA1 Low External Fibre Splicing [D6] SKA-TEL-SADT-0000657 SKA1 Low RPF Site Layout [D7] SKA-TEL-SADT-0000651 SKA1 Low Fibre Distances [D8] SKA-TEL-SADT-0000653 SKA1 Low Internal Fibre Reticulation [D9] SKA-TEL-SADT-0000655 SKA1 Low Pit Placement

2018-11-20 Page 19 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

3 ACRONYMS AND DEFINITION OF TERMS

3.1 Acronyms AAA ...... Authentication, Authorisation and Accounting AARNet ...... Australian Academic and Research Network AIV ...... Assembly, Integration, and Verification APC ...... Angled Polished Connector APL ...... AARNet Pty Ltd AS ...... Australian Standard ASKAP ...... Australian Square Kilometre Array Pathfinder CD ...... Chromatic Dispersion CDR ...... Critical Design Review COTS ...... Commercial Off-The-Shelf CPF ...... Central Processing Facility CSIRO ...... Commonwealth Scientific and Industrial Research Organisation CSP ...... Central Signal Processing DBYD ...... Dial Before You Dig DDBH ...... Digital Data Back Haul DHCP ...... Dynamic Host Configuration Protocol ECP ...... Engineering Change Proposal EICD ...... External Interface Control Document EMI ...... Electro-magnetic Interference EMC ...... Electromagnetic Compatibility EOC ...... Engineering Operations Centre ETSI ...... European Telecommunications Standards Institute FMECA ...... Failure mode, effects and criticality analysis FRQ ...... Frequency distribution FTP ...... File Transfer Protocol GE ...... Gigabit Ethernet GNSS ...... Global Navigation Satellite System HDPE ...... High Density Polyethylene HSE ...... Health, Safety and Environment HTTP ...... Hyper-Text Transport Protocol HTTPS ...... Secure Hyper-Text Transport Protocol HV ...... High Voltage HVAC ...... Heating, ventilation and air-conditioning ICD ...... Interface Control Document IEC ...... International Electrotechnical Commission IEEE ...... Institute of Electrical and Electronics Engineers IICD ...... Internal Interface Control Document ILS ...... Integrated Logistics IMP ...... Integrated Management Plan INAU ...... Infrastructure Australia INFRA ...... Infrastructure ISO ...... International Standards Organisation IT ...... Instituto de Telecomunicações, ITU ...... International Telecommunications Union JSA ...... Job Safety Analysis KVM ...... “Keyboard, video and mouse” LC ...... Lucent Connector LINFRA ...... Local Infrastructure LFAA ...... Low Frequency Aperture Array LMC ...... Local Monitor and Control LRU ...... Line Replaceable Unit LSPM...... Light Source and Power Meter LV ...... Low Voltage MeerKAT ...... The South African precursor array being built on site in the Karoo MDAL ...... Master Data Assumption List MIL‐STD ...... Military Standard 2018-11-20 Page 20 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

MM ...... Multi-mode MPO ...... Multi-fibre Push-On MRO ...... The Murchison Radio-astronomy Observatory MSDS ...... Material Safety Data Sheet MSF ...... Murchison Radio-astronomy Observatory Support Facility MTP ...... Multi-fibre Termination Push-on NMGR ...... Network Manager NPL ...... National Physical Laboratory NREN ...... National Research Network NSDN ...... Non Science Data Network NTP ...... Network Time Protocol NWA ...... Network Architecture NZS ...... New Zealand Standard ODF ...... Optical Distribution Frame ORL ...... Optical Return Loss OSI ...... Open Systems Interconnection OSNR ...... Optical Signal to Noise Ratio OTDR ...... Optical Time Domain Reflectometer PAT ...... Portable Appliance Testing PBS ...... Product Breakdown Structure PDR ...... Preliminary Design Review PDU ...... Power Distribution Unit PMD ...... Polarisation Mode Dispersion PMP ...... Project Management Professional PPE...... Personal Protective Equipment PPS ...... Pulse per second PSU ...... Power Supply Unit PVC...... Polyvinyl Chloride QSFP ...... Quad Small Form-factor Pluggable RAM ...... Reliability, Availability and Maintainability RAMS ...... Reliability, Availability, Maintainability and Safety RCD ...... Residual Current Device Rev ...... Revision RFI ...... Radio Frequency Interference RFoF ...... Radio Frequency over Fibre RM ...... Requirements Matrix RPF ...... Remote Processing Facility SaDT ...... Signal and Data Transport Synchronisation and Timing SAT ...... Synchronisation and Timing SAT.LMC ...... Synchronisation and Timing Local Monitor and Control SC ...... Standard Connector or Subscriber Connector SDP ...... Science Data Processor SFP+ ...... Small Form-factor Pluggable(the + denotes enhanced that can carry 10 Gbit/s) SKA ...... Square Kilometre Array SM ...... Single Mode SNMP ...... Simple Network Management Protocol SKAO ...... SKA Office SLA ...... Service Level Agreement SOC ...... Science Operations Centre SPC ...... Science Processing Centre SSH ...... Secure Shell SSL ...... Secure Sockets Layer STFR ...... System for Timing and Frequency Reference SWMS ...... Safe Work Method Statement TBC ...... To be confirmed TBD ...... To be determined TFTP ...... Trivial File Transfer Protocol THU...... Tsinghua University TLS ...... Transport Layer Security TM ...... Telescope Manager TRL ...... Technology Readiness Level

2018-11-20 Page 21 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

TRS ...... Technical Requirements Specification ULL ...... Ultra Low Loss UMan...... The University of Manchester UPC ...... Ultra Polished Connector UPS ...... Uninterruptible Power Supply UTC ...... Coordinated Universal Time UWA ...... The University of Western Australia VGA ...... Video Graphics Array VLBI ...... Very Long Baseline Interferometry WA ...... Western Australia

2018-11-20 Page 22 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

3.2 Definition of Terms

Cabinet

A rack enclosed with doors and side panels.

Cable Vault

An underground room, constructed from reinforced concrete or brick, housing communications cables

Drop Splice

A splice off of a backhaul cable used to service an individual Remote Processing Facility

Enclosure

A cabinet for equipment designed to protect the contents from the environment

Inside Plant

Refers to all of the communications cabling and equipment inside a building

KVM Switch

A device to share the keyboard, video and mouse between several computers

Optical Distribution Frame (ODF)

An Optical Distribution Frame is used to provide interconnections between multiple fibre cable terminations and to allow changes by means of fibre patchleads

Outside Plant

Refers to all of the physical cabling and supporting infrastructure (such as conduit, cabinets, tower or poles) located between buildings.

Patch Field

One or more patch panels featuring a number of jacks, usually of the same or similar type, for the use of connecting and routing circuits for monitoring, interconnecting, and testing circuits in a convenient, flexible manner.

Pathway

A means to reticulate a cable or wire from one location to another

Pit

A structure set below ground with lid flush to the surface to enable access for cable installation and termination

2018-11-20 Page 23 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Pluggable Optic

An optical transceiver module that converts an electrical signal - typically on a system board, line card or Network Interface Card (NIC) to an optical signal for transmission over fibre optic infrastructure. A Multi Source Agreement (MSA)3 defines the physical and electrical characteristics of a Pluggable Optic. The optical interfaces meet optical specs such as 100GBASE-LR4, 100GBASE-SR4 defined by the IEEE.

Pluggable optic examples include SFP (Small Form factor Pluggable - used for 1Gbps Ethernet), SFP+ (identical physical specs, but with signal rate defined for 10Gbps - used for 10G Ethernet), QSFP+( 4 electrical lanes defined, each adhering to the SFP+ electrical specification - used for 40G Ethernet), and QSFP28 ( 4 lanes defined like QSFP+, but with the lane signalling rate increased to 28 Gbps - used for 100G Ethernet). Most pluggable optics are "hot pluggable" i.e. they can be inserted or removed whilst the hosting equipment is operational.

Power Distribution Unit (PDU)

A device fitted with multiple power outlets designed to distribute electric power to racks of equipment within a cabinet and/or rack.

Splice

A joint between two optical fibres created using heat.

Rack

A standardised frame or enclosure for mounting multiple items of equipment, compliant with [AS3] IEC 60297. May be an open frame or an enclosed frame

Uninterruptible Power Supply (UPS)

An electrical apparatus to provide emergency power to a load in the event of the input power source failing.

3 See http://www.cfp-msa.org/

2018-11-20 Page 24 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4 SOLUTION DESCRIPTION

The SADT Local Infrastructure design mirrors the physical arrangement of the stations and RPF across the telescope.

Figure 1 shows the arrangement of stations for the telescope. Stations fall into two categories - Core and Outer. Core stations close to the CPF are provided connectivity by the LFAA element. Outer stations are placed on the spiral and in proximity to each Outer station location, a remote processing facility (RPF) will be required. SADT is required to provide connectivity from each RPF back to the central processing facility (CPF). The CPF is located to the south west of the core of the telescope, where the “Core” stations are clustered and the common starting point of each spiral exists, to minimise RFI impacts.

‘Outer Stations’ Region

x12 SPIRAL 2 RPF RPF (North) x12

< SPIRAL 1 80 km (East) RRPPFF

m 0k ASKAP ‘Core’ < 8 Control Region Building ~30km For fibre CPF m k connectivity to site 0 5 3 ~

EOC

SPIRAL 3

m (South) k 0 ~ 8

4 0 < 0 k m x12

RPF RPF ~ 1 km

SOC SPC

2018-11-20 Page 25 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

‘Outer Stations’ Region

x12 SPIRAL 2 RPF RPF (North) x12

< SPIRAL 1 80 km (East) RRPPFF

m 0k ‘Core’ < 8 Region

CPF

km 50 EOC ~ 3

SPIRAL 3

m (South) k 0 ~ 8

4 0 < 0 k m x12

RPF RPF ~ 1 km

SOC SPF

Figure 1: SKA1-LOW key locations, regions and logical cable topology4

Figure 2 shows an overview of the SKA1 LOW telescope configuration showing the centres of each of the remote station locations. These stations are laid out in three radiating spirals (north, east and south).

4 [RD6] SADT.LINFRA.MDAL-0205

2018-11-20 Page 26 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Figure 2: SKA1-Low Array Outer Station Region. RPF, CPF and antenna station locations and cable topology

Shown between these locations is the shortest path between the CPF and each RPF along each spiral for the fibre, ignoring the vagaries of the intervening terrain. Some constraints have been applied to reflect the constraints and extent of the MRO site.

This arrangement is based on [AD3] configuration co-ordinates provided in May 2016.5 It also reflects elements of the HV power design as understood at that point in time. It should be noted that the design cannot be further validated until final requirements around the design of the HV power distribution and the final locations and design of the CPF and RPF buildings are provided by INAU.

5 [RD6] SADT.LINFRA.MDAL-0206

2018-11-20 Page 27 of 158

SKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.1 Products SADT LINFRA products are grouped into seven categories:  Pathway  Fibre ( Cabling )  Structures  Cabinets and Racks  Power ( Distribution, Reticulation and UPS )  Copper ( Cabling )  KVM  Special Test Equipment These reflect the hardware to be installed and the structure of installation for the SADT LINFRA sub- element. The breakdown of products is provided in the [AD4] SADT Product Breakdown Structure (PBS).

4.2 Equipment-Location Matrix (Nodes) LINFRA products are located across the whole LOW telescope. Specifically:  CPF o Cabinet Room o Maser Room  Outside Plant at the MRO  GNSS Antenna Location near CPF  RPF They also located across the observatory and telescope support locations. Specifically:  EOC  SOC  SPC  Observatory (refers to other locations at or near the MRO) The correspondence between the products, the components of the design and their installation/storage location is noted in the Equipment Location Matrix provided in Table 1 below.

2018-11-20 Page 28 of 158

158

SKA-TEL-SADT-0000260-DDDSKA-TEL-SADT-0000260-DDD Rev: 2.1 SKA1 LOW LINFRA DETAIL DESIGN REPORTSKA1 LOW LINFRA DETAIL DESIGN REPORT

†

LINFRA Product / Component / Part CIN

CPF MRO RPF Observatory EOC SOC SPC Trenching and Reticulation Network Infrastructure LOW 140-000000 Copper 140-010000

Copper_Cable 140-011000

Copper_Assembly 140-012000

Copper_Patch_Lead 140-012100

Pre-Terminated_Copper_Cable 140-012200

Copper_Termination 140-012210

Copper_Field_Outlet 140-012212

Copper_Patch_Panel 140-012211

Fibre 140-020000

Fibre_Cable 140-021000

Distribution_Fibre_Cable 140-021100  

Buried_Fibre 140-021110  

Pole_Fibre 140-021120

Backhaul_Fibre_Cable 140-021200 

Buried_Fibre 140-021210 

Pole_Fibre 140-021220

Fibre_Assembly 140-022000

Fibre_Patch_Lead 140-022100     

Fibre_Pig_Tail 140-022200  

Pre-Terminated_Fibre_Cable 140-022300 

Fibre_Termination 140-023000  

Splice_Tray 140-023100  

Fibre_Splice 140-023200   

Fibre_Tray 140-023300

CPF_Fibre_Tray 140-023310 

RPF_Fibre_Tray 140-023320 

EOC_Fibre_Tray 140-023340 

2018-11-20 Page 29 of 158 158

SKA-TEL-SADT-0000260-DDDSKA-TEL-SADT-0000260-DDD Rev: 2.1 SKA1 LOW LINFRA DETAIL DESIGN REPORTSKA1 LOW LINFRA DETAIL DESIGN REPORT

†

LINFRA Product / Component / Part CIN

CPF MRO RPF Observatory EOC SOC SPC

SOC_Fibre_Tray 140-023350 

SPC_Fibre_Tray 140-023360 

Observatory_Fibre_Tray 140-023370 

Fibre_Patch_Panel 140-023400

CPF_Patch_Panel 140-023410 

RPF_Patch_Panel 140-023420 

EOC_Patch_Panel 140-023440 

SOC_Patch_Panel 140-023450 

SPC_Patch_Panel 140-023460 

Observatory_Patch_Panel 140-023470 

Fibre_Joint 140-023500 

Optical_Distribution_Frame_(ODF) 140-023600

CPF_ODF 140-023610 

EOC_ODF 140-023620

SOC_ODF 140-023630

SPC_ODF 140-023640

Observatory_ODF 140-023650

NREN_Interconnect_Fibre 140-024000   

Pathway 140-030000       

Trench 140-031000  

Marker 140-032000

Marker_Tape 140-032100 

Marker_Post 140-032200 

Conduit 140-033000   

Cable_Tray 140-034000  

Trunking 140-035000  

Waveguide 140-036000  

Pit_Man-Hole_Hand-Hole 140-037000 

2018-11-20 Page 30 of 158 158

SKA-TEL-SADT-0000260-DDDSKA-TEL-SADT-0000260-DDD Rev: 2.1 SKA1 LOW LINFRA DETAIL DESIGN REPORTSKA1 LOW LINFRA DETAIL DESIGN REPORT

†

LINFRA Product / Component / Part CIN

CPF MRO RPF Observatory EOC SOC SPC

Enclosure 140-038000

Poles 140-039000

Racks 140-040000

CPF_Rack 140-041000 

RPF_Rack 140-042000 

Observatory_Rack 140-044000 

EOC_Rack 140-045000 

SOC_Rack 140-046000 

SPC_Rack 140-047000 

CSP-SDP_Amplifier_Rack 140-049000

Maser_Room_Rack 140-083000 

Structures 140-050000

GNSS_Calibration_Shelter 140-051000 

Power 140-060000

Power_Cables 140-061000

Power_Reticulation 140-062000

Power_Distribution_Units_(PDU) 140-062100

CPF_PDU 140-062110 

RPF_PDU 140-062120 

EOC_PDU 140-062130 

SOC_PDU 140-062140 

SPC_PDU 140-062150 

Observatory_PDU 140-062160 

Uninterruptible_Power_Supplies_(UPS) 140-063000

RPF_UPS 140-063100 

Observatory_UPS 140-063200  

Blanking_Panel 140-048000     

Wall_Mounted_Cabinet 140-071000 

2018-11-20 Page 31 of 158 158

SKA-TEL-SADT-0000260-DDDSKA-TEL-SADT-0000260-DDD Rev: 2.1 SKA1 LOW LINFRA DETAIL DESIGN REPORTSKA1 LOW LINFRA DETAIL DESIGN REPORT

†

LINFRA Product / Component / Part CIN

CPF MRO RPF Observatory EOC SOC SPC

Cable_Management 140-072000      

KVM_Switch 140-080000   Table 1: LINFRA Equipment Location Matrix

† This describes the areas in between the CPF and the RPF locations

Test equipment will be required to maintain and diagnose problems with the fibre infrastructure, and will be used at all locations. Table 2 below reflects where the equipment is kept when not in use.

LINFRA Product / Component / Part CIN

CPF MRO† RPF Observatory EOC SOC SPC

Special Test Equipment Optical Time Domain 140-071000 Reflectometer 

Light Source 140-079000 

Optical Power Meter 140-072000  Video Fibre Inspection 140-073000 Microscope  Visual Fault Locator 140-074000 

Connector cleaner 140-075000 

Talkset 140-077000 

PMD / CD tester 140-078000  Table 2: LINFRA Equipment Location Matrix – Special Test Equipment

These items will be used by personnel with network technician or communications cabler skillsets.

2018-11-20 Page 32 of 158 158

SKA-TEL-SADT-0000260-DDDSKA-TEL-SADT-0000260-DDD Rev: 2.1 SKA1 LOW LINFRA DETAIL DESIGN REPORTSKA1 LOW LINFRA DETAIL DESIGN REPORT

4.3 SKA1 LOW Outside Plant Fibre Reticulation Design

User Requirements

LFAA and the SADT sub-elements are the users of the fibre reticulation network. The fibre reticulation network carries:  their individual system signals of LFAA and the SADT sub-elements to and from the LOW CPF to the various LOW RPFs  NSDN networks to the Power Station Aside from LFAA, the overall design strategy for SADT is to only provide direct access to the fibre for SADT sub-elements and to provide localised equipment interfaces for all other elements. Specific requirements are contained within the IICD documents referenced in Table 3 and Table 4.

Element Relevant EICD

LFAA: [RD9] 100-000000-026 Interface Control Document SADT to LFAA Table 3: External Fibre interfaces with other SKA1 LOW telescope elements

SADT Sub - Element Relevant IICD [RD1] SKA-TEL-SADT-0000445_ICD- LINFRA to NSDN (LOW) Internal Interface NSDN: Document [RD2] SKA-TEL-SADT-0000439_ICD -LINFRA to SAT.STFR.UTC (LOW) Internal Interface STFR.UTC: Document [RD3] SKA-TEL-SADT-0000440_ICD_ LINFRA to SAT.STFR.FRQ (LOW) Internal Interface STFR.FRQ: Document (THU) Table 4: External Fibre interfaces with other SADT sub-elements

Key fibre reticulation requirements summarised from all interface documents are as follows in Table 5 and Table 6:

2018-11-20 Page 33 of 158 158

SKA-TEL-SADT-0000260-DDDSKA-TEL-SADT-0000260-DDD Rev: 2.1 SKA1 LOW LINFRA DETAIL DESIGN REPORTSKA1 LOW LINFRA DETAIL DESIGN REPORT

SADT Work Reference Transmit Receiving Average Average Average Average Optical Dispersion Reference Notes Package application Wavelength Wavelength launch launch receive receive Insertion power (max) power (min) power (max) power (min) Loss nm nm dBm dBm dBm dBm dB

LFAA6 Vendor Vendor Vendor Vendor Vendor Vendor CPF to outer dependant dependant dependant dependant dependant dependant most RPF - 24dB, end of life Vendor Dedicated solution will transmission be designed equipment on around all links capability of the fibre Table 5: Summary of external user fibre performance requirements

NSDN 10GBASE- 1550 1550 4.0 -4.7 -1.0 -15.8 10.1 [AS4] Used for inner ER RPF locations less than 40km from CPF Used for Power Station

6 [RD6] SADT.LINFRA.MDAL-0257

2018-11-20 Page 34 of 158 158

SKA-TEL-SADT-0000260-DDDSKA-TEL-SADT-0000260-DDD Rev: 2.1 SKA1 LOW LINFRA DETAIL DESIGN REPORTSKA1 LOW LINFRA DETAIL DESIGN REPORT

NSDN 10GBASE- 1550 1550 4.0 0.0 -7.0 -22.0 24.0 1280 ps/nm [AS5] Multivendor789 ZR (80km) standard referencing physical standard of SONET/SDH OC192/STM64 Long reach (LR-2)

STFR.UTC Custom 1560.21 1558.98 3.0 -2.0 -3.0 -24.0 21.0 DWDM SFP optics in use. (CPF End) Avoid use of mixed fibre grades

STFR.UTC Custom 1558.98 1560.21 3.0 -2.0 -3.0 -24.0 21.0 DWDM SFP optics in use. (RPF End) Avoid use of mixed fibre grades

STFR.FRQ-LOW Custom 1547.72 1548.53 3.0 -2.0 -1.0 -26.0 25.0 [RD4] Bidirectional optics in use. (CPF End) Losses must

7 https://www.finisar.com/sites/default/files/downloads/finisar_ftlx1812m3bcl_10g_80km_multi-rate_xfp_optical_transceiver_product_specification.pdf 8 http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html 9 https://www.juniper.net/documentation/en_US/release-independent/junos/topics/reference/specifications/transceiver-m-mx-t-series-10-gigabit-optical- specifications.html

2018-11-20 Page 35 of 158 158

SKA-TEL-SADT-0000260-DDDSKA-TEL-SADT-0000260-DDD Rev: 2.1 SKA1 LOW LINFRA DETAIL DESIGN REPORTSKA1 LOW LINFRA DETAIL DESIGN REPORT

be considered for both transmit and receive wavelengths.

STFR.FRQ-LOW Custom 1548.53 1547.72 3.0 -2.0 -1.0 -26.0 25.0 [RD4] Bidirectional optics in use. (RPF End) Losses must be considered for both transmit and receive wavelengths.

Table 6: Summary of SADT sub-element user fibre performance requirements

2018-11-20 Page 36 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.3.1.1 Requirements from the CPF to each RPF Cable capacity requirements from each RPF back to the Central Processing Facility (CPF) are shown in Table 7 and Table 8.

Telescope Element Requirement

LFAA: 2 fibre cores10 Table 7: Summary of external user fibre core requirements

SADT Sub Element Requirement

NSDN: 4 fibre cores

STFR.UTC: 1 fibre core (Production) 1 fibre core (Calibration)

STFR.FRQ: 1 fibre core(Production) 1 fibre core (Diagnostic) Table 8: Summary of SADT sub-element user fibre core requirements

In the context of the above table,  Production refers to a fibre core that is utilised continuously to provide the timing or frequency product.  Calibration refers to a second fibre core run parallel and with similar characteristics to the Production fibre core. The Calibration core will be used on an infrequent basis to enable the calibration of the timing or frequency product.  Diagnostic refers to a second fibre core run parallel and with similar characteristics to the Production fibre core. No interfaces are defined for this fibre and are noted solely for dimensioning the number of cores. It may be used for diagnostic purposes. Each length of fibre cable consists of multiple fibre cores run together in a parallel fashion. Cores in different cables may be joined (“spliced”) together to form longer optical paths. Each RPF requires a total of ten (10) parallel fibre cores to connect from the CPF to the RPF. Given the distances involved between the CPF and the RPF, the optical path will be assembled from several cables spliced together. An optical path of twelve (12) parallel fibre cores allocated to each RPF is deemed most practical. This is for the following reasons:  Twelve (12) fibre construction is a basic industry method of cable construction (either tube or ribbon) and is recommended by fibre cable manufacturers.  Backhaul cables running along the spirals to the RPF will be of standard “n by 12” fibre core construction.  Simplified splicing and future maintenance requirements due to a RPF being provisioned by fibres in a single cable that are spliced to fibres in a single tube of the backhaul cable. The fibre tubes are colour coded for easy identification. Splicing between fibre cores in different tubes would lead to complex splicing and repair processes when the splices appear in different splice trays within a single joint.

10 [RD6] SADT.LINFRA.MDAL-0258

2018-11-20 Page 37 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

 The surplus cores into each RPF will be reserved as spares (and fully terminated at both ends).11 In addition the backhaul will be designed with spare cores (4 tubes/ribbons of 12 cores = 48 cores) along each spiral for use by AIV, ancillary infrastructure, future systems, custom experiments, maintenance activities and as spares. The additional cost of the extra fibre cores is negligible relative to the cost of both the civil requirements and cable installation processes. The fibre cores will enter the CPF via waveguides and be terminated immediately on Optical Distribution Frame (ODF) on the inside wall. The LFAA, NSDN and the SAT.STFR fibre cores, both working fibre and spares, will be split as they enter the CPF and follow different paths for access into the CPF Cabinet Room and to the CPF Maser (Timing and Frequency) room. This will be via a cross connect on the ODF, connecting between the outside plant cables and individual cables/patch panels in the CPF. Care will need to be taken to minimise any possible disturbance to these STFR fibre cores during normal operations. 4.3.1.2 Requirements from the CPF to the Power Station Cable capacity requirements from the Central Processing Facility (CPF) to the Power Station are shown in Table 9.

SADT Sub Element Requirement

NSDN: 4 fibre cores Table 9: Summary of SADT sub-element user fibre core requirements

It is assumed that the Power Station will be approximately five (5) kilometres south of the CPF building12, and that the fibre will be installed alongside the power cable. The 24 core Distribution Cable (also used to connect from the Backhaul Cable into each RPF) is suitable for this application. This is for the following reasons:  Twelve (12) fibre construction is a basic industry method of cable construction (either tube or ribbon) and is recommended by fibre cable manufacturers. The 24 core Distribution Cable will be of standard “n by 12” fibre core construction.  The 24 core Distribution Cable will be obtainable as a single five kilometre length, and can be installed with only splices at the waveguides at each end (CPF and Power Station)  This cable will be the same as used elsewhere and minimise sparing required.  The surplus cores in the cable will be reserved as spares (and fully terminated at both ends).13 These may be used for ancillary infrastructure, future systems, custom experiments, and maintenance activities. The additional cost of the extra fibre cores is negligible relative to the cost of both the civil requirements and cable installation processes. The fibre cores will enter the CPF via waveguides and be terminated immediately on Optical Distribution Frame (ODF) on the inside wall. These will then be patched through to NSDN equipment in the CPF Cabinet Room. This will be via a cross connect on the ODF, connecting between the outside plant cables and individual cables/patch panels in the CPF.

11 [RD6] SADT.LINFRA.MDAL-0207 12 [RD6] SADT.LINFRA.MDAL-0286 13 [RD6] SADT.LINFRA.MDAL-0287

2018-11-20 Page 38 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Overview of the SKA1 LOW External Fibre Reticulation Design

The SKA1 LOW external fibre reticulation design follows a consistent arrangement across the SKA1 LOW telescope.

Haul Pits as required 192 core 24 Cores Quantity as required

FFIIBBRREE PPAATTCCH PPAANEELL FFIIBBRREE FFIIBBRREE FFIIBBRREE FFIIBBRREE (( MAASSEERR RROOM )) JJOIINTT JJOIINTT JJOIINTT JJOIINTT

SSPPLLIICCEE SSPPLLIICCEE SSPPLLIICCEE SSPPLLIICCEE

24 Cores 12 Spliced

PPIITT ((SSPPLLIICCEE)) PPIITT ((SSPPLLIICCEE)) PPIITT ((HAAULL)) PPIITT ((SSPPLLIICCEE))

C ENTRY PIT ENTRY PIT ENTRY PIT

Y G

I ENTRY PIT ENTRY PIT ENTRY PIT

L R

ODF L R

ODF E

P T

V WAVE GUIDE WAVE GUIDE WAVE GUIDE S

V WAVE GUIDE WAVE GUIDE WAVE GUIDE

E E

W SPLICE SPLICE SPLICE

FFIIBBRREE PPAATTCCH PPAANEELL FFIIBBRREE PPAATTCCH PPAANEELL FFIIBBRREE PPAATTCCH PPAANEELL

FFIIBBRREE PPAATTCCH PPAANEELL (( CCAABBIINEETT RROOM )) RRPPFF RRPPFF RRPPFF

24 Cores Quantity as required

12 RPFs in total per CCPPFF spiral LLOW Figure 3: Typical arrangement for each spiral14

The LOW telescope is laid out as an arrangement of stations consisting of a dense core and smaller collection of stations distribution along three spirals radiating out from the core. Each of these stations will deliver signals to a processing facility. For stations in the core, this will be the CPF, and for the stations along each spiral, these are the RPF buildings. SADT has the responsibility to transport the LFAA, NSDN, SAT.STFR.UTC, and SAT.STFR.FRQ networks and signals from each RPF to the CPF. There are twelve RPFs per spiral, for an overall total of thirty-six (36) RPFs that SADT are responsible for providing fibre connectivity at. The external fibre design consists of three components that are: a) Cable reticulation at the CPF from the equipment to the outside of the building b) Cable backhaul along the spirals consisting of large core count cables c) Cable reticulation into each RPF from the backhaul cable The backhaul cable will run out the length of each spiral, with each RPF utilising cores from the backhaul cable progressively out the distance of the spiral. This arrangement is described further in §4.3.2.2. For the Power Station, a Distribution Cable will running directly from the CPF to the Power Station. A demarcation point between the telescope and Power Station infrastructure will exist outside the Power Station.15 This is illustrated in Figure 4.

14 Drawing [D1] 15 [RD6] SADT.LINFRA.MDAL-0288

2018-11-20 Page 39 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Demarcation Point

24 Core

FFIIBBRREE JJOIINTT

SSPPLLIICCEE

24 Core

PPIITT

C ENTRY PIT

Y I

G ENTRY PIT

L R

ODF L R

ODF E

P T

V WAVE GUIDE S

V WAVE GUIDE

E E

W SPLICE

FFIIBBRREE PPAATTCCH PPAANEELL

FFIIBBRREE PPAATTCCH PPAANEELL (( CCAABBIINEETT RROOM )) PPOWEERR SSTTAATTIION

24 Cores Quantity as required

CCPPFF LLOW Figure 4: Typical arrangement for Power Station

4.3.2.1 Cable reticulation at the CPF Inside the CPF building there will be a single Optical Distribution Frame (ODF) shared by the Data networks (LFAA, NSDN) and the Synchronisation and Timing (SAT) systems (STFR.FRQ and STFR.UTC). The ODF is a demarcation point between cables that leave the building (outside plant) and the internal fibre distribution/patch leads/equipment in the building (inside plant). The outside plant fibre enters the CPF building via the cable vault and duct provided by INAU and enters the CPF Cabinet Room area via waveguides before being terminated on the ODF. 4.3.2.1.1 CPF Optical Distribution Frame The ODF enables connectivity from equipment and internal cables located inside the CPF building to the external cables outside, these being the three backhaul cables providing connectivity to the RPFs along each spiral – north, east and south, and the distribution cable to the Power Sation. The ODF will be located at the end of a row adjacent to other SADT cabinets. The fibre cores will then be spliced to fibre optic pig tails and presented on the ODF via LC APC connectors. Further internal cables will link off to patch panels in the:  CPF Maser (Timing and Frequency) Room. These will provide the connectivity from SAT.STFR.UTC and SAT.STFR.FRQ timing and frequency distribution equipment out to the telescope.  CPF Cabinet Room. These will provide the connectivity for the LFAA and NSDN equipment between the CPF and the RPF. The ODF will have further cables terminating there to provide additional internal fibre distribution. (See §4.4) There may be other cables terminating at the ODF to cater for other local interconnects to ancillary rooms, buildings and facilities (such as power generation, contractor sheds and work areas, and additional experiments).

2018-11-20 Page 40 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

The cable (and the associated termination) providing connectivity to the NREN for the CSP-SDP link will not be terminated on the ODF. See §4.5.2 for implementation details. The ODF (Figure 5) will provide a patching mechanism between the outside plant and inside plant. This permits future reconfigurations for equipment maintenance and replacement. Cables at the ODF will be presented on a common consistent connector type. To maximise density and performance, LC APC (angled polished connector) is preferred.

Figure 5: Typical ODF – Commscope NG416

The ODF will: a) Be a dedicated, fit for purpose, commercial off the shelf fibre distribution frame b) Consist of a high density fibre patch panel arrangement with a consistent connector type c) Have an ability to house splices and splice trays (even if omitted during initial installation through use of pre-terminated cable tails). d) Provide sufficient capacity to terminate both outside and inside plant cables as one contiguous patch field. e) Contain cable management and strain relief for fixed cables. f) Have patch cable management. g) Only require front access for cable installation and termination h) Be of suitable dimensions for installation into the CPF Cabinet Room i) Be fitted with front and side panels, with a mechanism to lock/restrict access j) Have appropriate mechanisms to ensure laser safety during operation (for example, angled patch panels and covers). Several suitable products exist in the market. [AS6] GR-449-CORE may be used as guidance in suitable product selection.

16 Image – Commscope NG4access - http://www.commscope.com/catalog/solution_wn_centralofc_hdflexibleodf/2147496434/product_details.aspx?id=6320 1

2018-11-20 Page 41 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

The safety requirements for the ODF include: a) Appropriate laser safety labelling. b) Mechanisms to restrict access to authorised personnel. c) Mechanisms and design features to prevent accidental exposure to laser radiation. The ODF shall comply with mandatory labelling requirements and display Warning (indicates hazard) and Explanatory (explains hazard and caution) labels per AS/NZS IEC 60825 series of standards (see [AS7], [AS8] and [AS9]). The choice of suitable ODF product will be driven by spatial requirements of and in the cabinets in the CPF and other design requirements such as laser safety. For a freestanding solution, ideally the ODF will be of similar width to the selected CPF cabinets. The ODF will terminate both external (A Side) and internal cables (B Side). Table 10 summarises the requirements. Cables will be terminated on the ODF in multiples of 12 cores. Between the A and B sides patch leads will be used to create a cross-connect. A Side Cores

RPF 1 to 36 432 GNSS 36 Campus 72 Power 24 Station

B Side Cores

LFAA Rack 1 24 LFAA Rack 2 48 LFAA Rack 3 48 CPF04 96 CPF05 96 CPF06 24 CPF07 24 M01 96 End of Row and BMS 192

Total A Side Cores 564 Total B Side Cores 648

Total A+B Cores 1212 Table 10: Summary of ODF Capacity Requirements

All cables terminating on the ODF will be run to their corresponding location through overhead cable trays provided by INAU. This corresponding location may either be inside the CPF or to the waveguides to exit the CPF. The external splicing arrangement for an RPF is show in Figure 6. The ODF will comply with [AS10] AS/CA S008 and its design and implementation will comply with [AS7] [AS8] [AS9] AS/NZS IEC 60825 series of standards where appropriate. For the purposes of clarity, cables associated with the outside plant portion of the core stations (LFAA) will not terminate at the ODF. This work is outside the scope of this design.

2018-11-20 Page 42 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Fibre Tail

SPLICE 001 001 002 SPLICE 002 003 SPLICE 003 004 SPLICE 004 005 SPLICE 005 PATCH PANEL 006 SPLICE 006 TUBE-01 ON 007 SPLICE 007 ODF 008 SPLICE 008 009 SPLICE 009 010 SPLICE 010 011 SPLICE 011 SPLICE

012 012 E D

013-024 TUBE-02 I U

025-036 TUBE-03 G E

OPTICAL DISTRIBUTION FRAME WAVEGUIDE SPLICE CABINET V

037-048 TUBE-04 A

049-060 TUBE-05 W 061-072 TUBE-06 073-084 TUBE-07 085-096 TUBE-08 192 CORE G.652.D J27 097-108 TUBE-09 109-120 TUBE-10 121-132 TUBE-11 133-144 TUBE-12 145-156 TUBE-13 157-168 TUBE-14 169-180 TUBE-15 181-192 TUBE-16

Figure 6: Splicing arrangement for RPF S8 (typical) at the ODF17

The arrangements of connections on the ODF should aim to mitigate disturbance to the timing products (SAT.STFR.UTC and SAT.STFR.FRQ). This is to be achieved by locating these terminations on separate parts of the frame from those of the data products. 4.3.2.1.2 Cable entry into the CPF building From the connection at the ODF each outside plant cable will need to transition through a splicing and waveguide arrangement. Each connection follows a similar structure: 1. Each fibre core will be presented on the ODF as an LC/APC connector. This will be via a tray (similar to Figure 11) containing a LC/APC duplex coupler. The tray will contain multiple couplers (typically enough for 12 cores) corresponding to an individual tube or ribbon in the outside plant cable. Effectively this corresponds to one tray per RPF. 2. Multiple indoor rated fibre cables (“fibre tails”), corresponding to one per tray, 12 cores per cable, will run from the ODF (Figure 13) to the Waveguide Splice Enclosure Cabinet (Figure 10). These cables will run on overhead tray between the two locations. Each fibre tail will be pre-terminated on the ODF end and cut to length when installed into the Waveguide Splice Enclosure cabinet. 3. The inner buffer tubes or the ribbons of the outside plant cable are brought through the waveguide (Figure 12) from the area outside the RFI/EMI shielded room individually and spliced to the fibre tails originating at the ODF. Sufficient waveguide capacity will be required to fit all the tubes or ribbons. The waveguides are specifically designed and machined items to restrict RFI and EMI leaving the shielded room. 4. The outside plant cable passes through the Optical Fibre Entry Box (Figure 7) and leaves the building via conduits to the cable vault outside the building (Figure 8). From the cable vault, it may be run through further conduits in the immediate area of the CPF to provide protection against other activities in the area before making the transition to the cable pathway along the spirals.

17 Drawing [D2]

2018-11-20 Page 43 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Figure 8: Cable Vault and Conduits under building – photo from ASKAP

Figure 7: Optical Fibre Entry Box

Figure 10: Photograph of ASKAP waveguide splice tray cabinet

Figure 9: Photograph of ASKAP splice trays

2018-11-20 Page 44 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Figure 11: Individual ODF Tray18 Figure 12: Photograph of Waveguide

Figure 14: ODF Submodule20

Figure 13: ODF19

Other waveguides and splice trays will be required for the cable providing the CSP-SDP connectivity back to the NREN (See §4.5.2), GNSS equipment (See §4.6.2) and for other cables interconnecting from other elements with SADT systems. 4.3.2.2 Cable backhaul along the spirals Each spiral consists of 12 RPF locations. With the CPF as the root of the tree, all 36 RPF locations have been modelled and a minimum spanning tree produced providing the shortest distances between each RPF, with optimisations to minimise the depth of the tree and the lengths of cable required.

18 Image – Canovate Angora ODF, http://mcldatasolutions.co.uk/optical/management/distribution-frame/angora-odf.html 19 Image – Canovate Angora ODF, http://canovate.com/product/angora-odf-system/ 20 Image – Canovate Angora ODF, http://mcldatasolutions.co.uk/optical/management/distribution-frame/angora-odf.html

2018-11-20 Page 45 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Each of the spirals at the inner most point of the spiral requires a minimum of 144 cores to service 12 RPFs with 12 cores spliced as direct unbroken optical path from the CPF to each RPF. To permit integration, future maintenance and other activities (for example, connection to other experiments) a larger cable size is required to be selected, with 192 cores being the next manufactured and practical size. A consistent cable size needs to be selected to optimise procurement and sparing21. In this case, this means an additional 48 cores (4 x 12) is provided. The cable size stated is a recommended minimum and some manufacturers may supply their equivalent nearest sized cable that meets this requirement (for example, 216 cores). At the centre of the spiral all 144 cores are in use, with this number progressively declining until the outermost RPF on that spiral is reached. The spare capacity at the outermost portions of the spiral enables future telescope growth (eg SKA2 or other experiments) with only minimal overbuild potentially required nearer the core. The arrangement of the fibre along the spiral can be drawn as a sequence of segments. With each segment, there can be multiple locations where splicing will occur. These can be split into two broad categories: a) Through Splice. This occurs when the length of the segment exceeds the amount of cable that can be transported on a single drum. All cores are spliced straight through between the two cables to their corresponding matching core. b) Drop Splice. This occurs at the far end of each segment near to where the RPF is located. From the backhaul cable an allocated 12 cores are cut and split off for connection to the RPF. These are spliced to the first 12 cores of the cable running from the RPF. All other cores in the backhaul cable will in most circumstances be spliced straight through between the two cables onto the next segment. Even for those cores that are not in use, the latter is done to assist with testing and repair (for example, to affect a repair using the spare cores a visit to each intermediate location is no longer required if they are already spliced together) The cable pathway along the spiral will consist of conduits and pits. The pits (hand holes) will fall into three categories based on function: a) Haul. A smaller pit used to interconnect two conduit segments and enabling access to pull (haul) the cable during installation procedures. b) Splice. A larger pit which provides the required space for fibre jointing enclosures and to maintain a cable bending radius c) Service Loop. A larger pit which interconnects two conduit segments and provides the required space to hold a loop of cable used to enable future maintenance and repairs. The pit will have adequate spare space to be able to enclose a fibre jointing enclosure. These splices will all occur in splice pits along the route, and will be housed in industry standard underground fibre jointing enclosures. The fibre optic cable used must comply with and be labelled in accordance with [AS10] AS/CA S008, and be appropriately rated for the application used. The cable jacket will be:  Of appropriate construction for mechanical hauling, including a sacrificial jacket  Rodent and termite resistant  Compliant with [AS11] AS1049.1 where relevant.

21 See further discussion of alignment with other procurement in §5.4

2018-11-20 Page 46 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

The expected diameter of the cable to be installed will be between 15mm and 20mm. We can optimise here to minimise the number of through splices in two ways.  Procure longer drum lengths of cable. As we get progressively further out the spirals from the CPF, the distance between each RPF increases (to a maximum of 18302 metres between RPF N15 and RPF N16) and require intermediate through splices. 4000 metres is considered a standard length that would be shipped on a cable drum. This does vary, for example, Prysmian ship 12500 metre lengths of 72 core cable for Telstra in Australia. For the purposes of development of our cost model, an effective length of 3750 metres was used to allow for individual drum variances.  For the shorter runs that can accomplish two RPFs on a single drum, or if a design choice is made to have all drums of the same length, mid-span fibre access techniques (“ringbarking”) can be used to access only the necessary tubes/ribbons required for splicing into the RPF leaving the other tubes/ribbons and cores undisturbed. Both options need to be considered during procurement and in discussion with the fibre supplier and installation contractor. It is anticipated that the cable backhaul along the spirals will share a common trench with the HV power reticulation across the site being provided by INAU.22,23 Both will follow the same spanning tree to reach the RPF locations.

22 [RD6] SADT.LINFRA.MDAL-0001 23 [RD6] SADT.LINFRA.MDAL-0017

2018-11-20 Page 47 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Figure 15: Overlay of Power and Fibre routes

2018-11-20 Page 48 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

For the fibre to share the common trench with the HV, both cable systems will need to follow the relevant local (Australian) standards or codes of practice associated with their disciplines. For HV electrical cables, both [AS12] AS/NZS 3000 and [AS13] AS2067 provide guidance. For the fibre optic cable, the operative standard is [AS14] AS/CA S009 which dictates separation distances to be followed for buried cables in proximity to HV. Both [AS12] AS/NZS 3000 and [AS14] AS/CA S009 are mandatory standards (per relevant legislation and regulations). [AS13] AS2067 §4.2.9.1 requires that a HV cable be buried at a depth of 750 mm where mechanical protection (eg tiles, slabs) is provided to meet [AS12] AS/NZS 3000. If mechanical protection is not provided, the requirement is 900mm min. As proposed by INAU element design, the HV cable will be trenched and mechanical protection provided. [AS14] AS/CA S009 §18.6 requires the fibre to be buried at minimum depth of 300mm (450mm under roads) in normal soils to the top of either the cable or conduit. [AS14] AS/CA S009 Table 4 dictates a minimum of 300 mm of segregation between the fibre (either in conduit or not) and the HV cable where the HV cable has been installed in accordance with [AS12] AS/NZS 3000. In accordance with good practice and [AS14] AS/CA S009 §18.3, the fibre optic cable and conduits will have marker tape placed above them in the trench, and marker posts installed above ground to highlight the route. The marker posts will highlight the presence of buried services and provide appropriate operational contact details upon them. The marker tape will be compliant with [AS15] AS/NZS 2648.1 and be of an appropriate colour to identify the service. Conduits used shall comply with [AS16] AS/NZ 4130 for HDPE and [AS17] AS/NZS 2053.2 for PVC, and be conformant with [AS10] AS/CA S008. 63mm HDPE will be used. The final design and configuration of the shared trench will be by INAU and subject to their engineering sign-off. It will comply with the standards (and relevant clauses) mentioned above. Figure 16 below shows one potential arrangement. The configuration of the trench for cables direct buried is similar. (See §4.1.2.3.3 for discussion of the merits of each installation technique)

GROUND LEVEL

m m

AS/CA S009:2013 §18.6.2

3 ~ AS/NZS 3000 §3.11.4.5 MARKER TAPE

(OPTIONAL) m

m AS/CA S009:2013 §18.3

1 m

0 MARKER TAPE

5 7

AS2067 §4.2.9.1 FIBRE OPTIC IN CONDUIT

m BARRIER m 0

m 0

3 m AS/NZS 3000 §3.11.4.3

5 AS/CA S009:2013 Table 4 7

DEPTH OF COVER AND MARKER TAPE COMPLIANT SEPARATION BETWEEN HV AND HV WITH AS/NZS 2648.1 FIBRE OPTIC CABLES IN CONDUIT N.T.S. COMPLIANT WITH AS/NZS 3000 Figure 16: Cross section of shared trench (typical)24

24 Drawing [D3]

2018-11-20 Page 49 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

HDPE conduit is transported wound into a coil. The standard manufacturing length of a single coil of 63mm HDPE conduit is 140 metres. Pits and couplers/fusion joins will be placed between each segment of conduit. The location of each pit/coupler/fusion joint will be flagged with marker posts.

TOP VIEW

420 m

140 m 140 m 140 m

HDPE CONDUIT SPLICE OR HAUL HDPE CONDUIT HDPE CONDUIT HDPE CONDUIT SPLICE OR HAUL HDPE CONDUIT PIT PIT

COUPLER OR FUSION JOIN Figure 17: Typical pit spacing25

Pits will be installed every 420 metres. This equates to three (3) lengths of HPDE conduit. The segments of conduit will be joined via a coupler piece, or via fusion welding of the two conduits together. The latter is more likely to be used if the conduit is being ploughed in. The spacing of pits does exceed the 100 metre spacing recommended by §18.3.5.3 and Table 3 of [AS14] AS/CA S009. However, this installation will be more akin to a carrier/utility network with long straight level runs of conduit and with minimal difficulties anticipated with the installation of the cable. Cable will be installed in the conduit with a mix of manual and mechanical hauling For locations where a crossing of a road is required, standard details will be developed by the contractor and road designer to establish appropriate levels of cover for the conduit and location of the pits in relation to each side of the road, informed by the [RD17] Utility Providers Code of Practice for Western Australia and [AS14] AS/CA S009. Likewise, similar details will be required for the crossing of any watercourse, with appropriate changes to method of construction as needed. Where the fibre installation cannot follow the INAU HV cable installation (for example, due to localised soil conditions or on the approach to buildings), the fibre optic cable will need to be installed at a minimum depth of 300mm (with the exception of road crossings and other technical requirements) to comply with AS/CA S009. The total quantity of these locations is not yet known but is expected to be minimal.26 These locations will be highlighted during the route surveys for the shared trenches.27 “Shop drawings” will be created to show the general arrangements required during survey and construction. Some of the initial geotechnical reports [RD5] (assuming consistency over much wider area for SKA1 LOW) indicate that ploughing to direct bury a cable or a conduit to a minimum depth of 300mm could be feasible for extended stretches. Other installation approaches including horizontal directional drilling and open trenching are also available to the contractor for these sections. The selection of technique will be chosen on site by the contractor (in accordance with an engineering and project change management process) in response to site conditions, time and installation cost, whilst maintaining overall project cost, quality and schedule control.

25 Drawing [D4] 26 [RD6] SADT.LINFRA.MDAL-0209 27 [RD6] SADT.LINFRA.MDAL-0208

2018-11-20 Page 50 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.3.2.3 Cable backhaul for the Power Station The cable to the Power Station will follow a direct point-to-point route from the Central Processing Facility (CPF) to the Power Station. It will follow alongside the HV power cable that will power the CPF and the telescope. The cable pathway for this route will consist of conduits and pits. The pits (hand holes) will fall into two categories based on function: a) Haul. A smaller pit used to interconnect two conduit segments and enabling access to pull (haul) the cable during installation procedures. b) Service Loop. A larger pit which interconnects two conduit segments and provides the required space to hold a loop of cable used to enable future maintenance and repairs. The pit will have adequate spare space to be able to enclose a fibre jointing enclosure. It will generally follow the same construction and installation methodology as the cable along the spirals, as it applies to:  Cable manufacture and labelling  Cable hauling  Trench configuration, Installation depth and segregation from the HV cable  Selection of conduits, pits and installation spacing  Marker posts and marker tape 4.3.2.4 Cable reticulation into each RPF For the short segment from the location of the Drop Splice to the RPF (Figure 18), each RPF will be serviced by a 24 core cable from the Drop Splice into the RPF.

Figure 18: Indicative 3D of Remote Processing Facility (See [RD7] Figure 35)

2018-11-20 Page 51 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Each cable into an RPF follows a similar arrangement: 1. At the pit containing the Drop Splice, the 12 cores of the backhaul cable heading towards the CPF will be spliced to 12 cores to the distribution cable to enable connection from the RPF back to the CPF (Figure 19). The distribution cable to the RPF will consist of 24 cores, but the remaining 12 cores in the cable will be left for future use that may include repairs, AIV activities or the connection of other “last-mile” fibre into the NSDN switch at the RPF (eg RFI monitors, weather stations, future experiments). 2. At the pit containing the Drop Splice the distribution (24 core) cable exits the pit via a 100 mm conduit towards the RPF. The conduit will provide additional physical protection for the cable in the vicinity of the RPF where many trades will be working during the construction of the structure and electrical services to the building. 3. Outside the RPF, a small cable entry pit will exist to enable easier hauling of the cable from the Drop Splice and into the RPF. A conduit will exit this pit into the RPF and interface with the waveguide arrangement associated with the building. 4. The inner tubes or the ribbon inside the outside plant cable will be brought through the waveguide from the area outside the RFI/EMI shielded room components of the RPF building. The waveguides are specifically machined items to restrict RFI and EMI leaving the RPF. 5. Inside the waveguide splice enclosure cabinet, which is located on the inside of the RPF, each core of the indoor rated cable is spliced to the corresponding core of the outdoor plant cable. These cores may be spliced individually or collectively depending on the construction of the respective cables, either loose tube or ribbon respectively. 6. A section of 24 core indoor rated cable runs from the waveguide splice enclosure to a rack mounted fibre optic splice tray/patch panel located inside the RPF. 7. All 24 cores in the cable are terminated on the patch panel with LC APC (angled polished) connectors inside the RPF. This is to avoid future need to disturb this patch panel. The 12 cores that are spliced through to the CPF will be labelled appropriately. Unlike the ODF at the CPF, both Data and Timing products will be present on the same patch panel.

2018-11-20 Page 52 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

TUBE-01 001-012 TUBE-01 001-012 TUBE-02 013-024 TUBE-02 013-024 TUBE-03 025-036 TUBE-03 025-036 TUBE-04 037-048 TUBE-04 037-048 TUBE-05 049-060 TUBE-05 049-060 TUBE-06 061-072 TUBE-06 061-072 TUBE-07 073-084 TUBE-07 073-084 J27 192 CORE G.652.D TUBE-08 085-096 TUBE-08 085-096 192 CORE G.652.D TUBE-09 097-108 TUBE-09 097-108 J25 TUBE-10 109-120 TUBE-10 109-120 TUBE-11 121-132 TUBE-11 121-132 TUBE-12 133-144 TUBE-12 133-144 TUBE-13 145-156 TUBE-13 145-156 TUBE-14 157-168 TUBE-14 157-168 TUBE-15 169-180 TUBE-15 169-180

TUBE-16 181-192 TUBE-16 181-192

2 4

1 2

0 0

- -

1 3

0 1

0 0

1 2

0 0

- -

E E

B B

U U

T T

24 CORE G.652.D

TOWARDS CPF

Figure 19: Remote Processing Facility Splice Arrangement (RPF S6, typical)28

28 Drawing [D5]

2018-11-20 Page 53 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

The route of the conduit from the splice pit to the RPF will take the most direct route, whilst avoiding other infrastructure – electrical (EPR zones, conduit routes and building access) and civil (access roads) to the greatest extent feasible. For the purposes of costing we’ve assumed that this is a distance of approximately 40 metres.29 No particular buffer zone is required for the fibre itself and none is required to mitigate RFI/EMC. It is assumed that the RPF is suitably located with a buffer distance from the stations it services.

IT U D N O C m m Shared trench 3 6 A R F IN L

V SADT LFAA H LINFRA 100mm CONDUIT ENTRY PIT T IT D P SA E C LI SP

REMOTE PROCESSING CRAC FACILITY

IT U D INAU N O HV RMU C m TRANSFORMER m 3 LV DB 6 A Edge of HV EPR Zone R F TYPICAL ARRANGEMENT OF SERVICES AT RPF IN L NOT TO SCALE

V H

Shared trench

TOWARDS CPF Figure 20: Remote Processing Facility Site Services Layout (typical)30

4.3.2.5 Cable reticulation into the Power Station The cable to the Power Station is expected to follow this arrangement: 1. A pit will occur at the perimeter of the Power Station complex. This pit will be the demarcation between telescope and power station infrastructure. Ideally, there will be no splicing at this location and the cable will continue up to the building. 2. A conduit will exit this pit and run up to the Power Station building and interface with the waveguide arrangement associated with the building. 3. The inner tubes or the ribbon inside the outside plant cable will be brought through the waveguide from the area outside the RFI/EMI shielded room components of the Power Station building. The waveguides are specifically machined items to restrict RFI and EMI leaving the Power Station building. 4. Inside the waveguide splice enclosure cabinet, which is located on the inside of the Power Station building, each core of the indoor rated cable is spliced to the corresponding core of the outdoor plant cable. These cores may be spliced individually or collectively depending on the construction of the respective cables, either loose tube or ribbon respectively.

29 [RD6] SADT.LINFRA.MDAL-0278 30 Drawing [D6]

2018-11-20 Page 54 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

5. A section of 24 core indoor rated cable runs from the waveguide splice enclosure to a rack mounted fibre optic splice tray/patch panel located inside the Power Station building. 6. All 24 cores in the cable are terminated on the patch panel with LC APC (angled polished) connectors inside the Power Station building. This patch panel is located within a suitable rack to house the NSDN equipment that interfaces with equipment inside the building.

Analysis of the LOW Fibre Reticulation Design

4.3.3.1 Site Constraints The design is constrained at a macro level by the layout of the RPFs within the boundary of the MRO (Figure 21). The choice of RPF location is driven by the science requirements, the LFAA Consortium design of the stations and the need to mitigate RFI/EMC impacts from the RPF and the associated electrical distribution infrastructure. The choice of RPF locations in turn, is reflected in the designs for power reticulation by INAU, and our design for the fibre reticulation (Figure 22 and Figure 23). Figure 24 represents this in schematic form and shows the cumulative fibre distances to each RPF from the CPF31.

Figure 21: MRO Site Boundaries with RPF Locations plotted

31 The AIV AA-1 “Temporary CPF” is discussed further in §4.9.2.1

2018-11-20 Page 55 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Figure 22: SKA1-Low Array Core Region. RPF, CPF and antenna station locations and cable topology

Figure 23: SKA1-Low Array Outer Station Region. RPF, CPF and antenna station locations and cable topology

2018-11-20 Page 56 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

SKA1 LOW FIBRE DISTANCES (CUMULATIVE) JOINT ID

SPIRAL 1 (East) RPF ID E16 J5 J8 E9 E12 10879 19533 J1 J2 J3 J4 J6 J7 J9 J10 J11 J12

E5 E6 E7 E8 E10 E11 E13 E14 E15 E16 1805 3245 5133 7614 11246 14166 22295 30008 41551 50061

SPIRAL 2 (North)

J17 N9

12971 J19 N11 18733 CPF J13 J14 J15 J16 J18 J20 J21 J22 J23 J24

N5 N6 N7 N8 N10 N12 N13 N14 N15 N16

3900 5345 7240 9720 13423 19065 24522 30556 39853 58154

SPIRAL 3 (South)

J25 S5 6233 J26 S6 4792 J29 J32 S9 S12

8650 19758 J27 J28 J30 J31 J33 J34 J35 J36

AA-1 S7 Temporary S8 S10 S11 S13 S14 S15 S16 2897 CPF 5387 9091 13882 19632 26855 35288 42450 J12 Figure 24: Schematic diagram and cumulative fibre distances to each RPF32

Both the power and fibre reticulation designs are strongly focussed on minimising cable length. Figure 15 shows where the overlay of the LOW fibre reticulation network occurs on the LOW power reticulation and trenching design as shown in Figure 25. (Both are based on the same underlying model and analysis). SADT will make maximum use of the trenches created for the HV cables to lay fibre cables.

32 Drawing [D7]

2018-11-20 Page 57 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Figure 25: HV Single Line Diagram provided by INAU for PDR33

Other fibre routes required are to be constructed at SADT expense. The anticipated length of these is short and occurs at the cable entries to the CPF and RPF buildings. Optimisation of the trenching requirements will continue to occur as:  INAU evolves their design for the INAU Critical Design Review (CDR).  Details of the construction program become available.  Site surveys occur to validate and constrain the design.  Cost constraints are worked through. This design is base-lined against the [AD3] SKA1 LOW Configuration Coordinates published in May 2016.

33 The INAU design has evolved since this drawing was provided and their design documentation should be consulted further. The purpose here is to illustrate the common path shared by the SADT fibre and INAU power cables.

2018-11-20 Page 58 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

For the Central Processing Facility and each of the Remote Processing Facilities, further localised layout and design constraints will occur in relation to the CPF and RPF is respect of: a) HV Power Distribution i. Location of HV Electrical equipment (transformers and switchgear) ii. HV Cable Reticulation iii. Earth Potential Rise zones iv. LV Power Distribution v. LV Switchboards and Pillars b) LV Cable Reticulation i. Earthing systems c) Site conditions i. Location of access roads ii. Geology iii. Topography iv. Ecology v. Heritage vi. RFI/EMC management 4.3.3.2 Equipment Constraints The SKA1 LOW fibre reticulation design is based on the [AD3] SKA1 LOW Configuration Co-ordinates. These are for 36 RPF locations with a maximum spiral length anticipated of less than 60km. The challenge with the spiral layout is to remain within the equipment capability, especially that of fibre loss and fibre dispersion for the element LFAA and the three sub-elements – NSDN, SAT.STFR.UTC and SAT.STFR.FRQ. The LFAA34 and NSDN networks will be utilising COTS (Commercial Off The Shelf) equipment for which the capabilities are well documented and known. The SAT.STFR.FRQ components are a different matter with the SAT.STFR.FRQ equipment being developed by the Tsinghua University, Beijing. The SAT.STFR.UTC equipment is a developed and implemented (although not widely commercially available) technology (White Rabbit), being modified and tested for the purpose of distributing the UTC timing signals over large distances. Based on the station geographic co-ordinates, the locations of the CPF and RPFs have been geographically positioned and the length of fibre routes determined and plotted. Loss calculations have been modelled for each RPF. RPF N16 is the further most RPF by distance from the CPF. In Figure 26, for conventional ITU G.652.D fibre with assumed loss of 0.24dB per km @ 1490 nm, an estimated loss of 20.62 dB was calculated. This estimate falls within the known capabilities of the equipment for all elements and sub-elements (Table 5 and Table 6). Detailed calculations for fibre loss are included in the [RD8] SKA1 LINFRA LOW Cost Model, on the “Spatial Model” and “Fibre Loss” sheets. A discussion of the calculation methodology is given on the “Fibre Loss” sheet.

34 [RD6] SADT.LINFRA.MDAL-0259

2018-11-20 Page 59 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Fibre Loss Calculation RPF ID: RPF_N16

Cable Selection 0.3 dB/km is worst case for G.652.D, assuming 0.24 dB/km as typical of a modern cable @ 1490nm

Fibre Type 8 Corning SMF-28e+ @ Fibre Description 1490nm Fibre Loss per km (dB) 0.24

Count Metres Loss Splices (m) (dB)

RPF Internal Distribution

Pig Tail Loss 0.50 db 0.5000 Splice @ RPF Patch Panel 0.05 dB 1 0.0500 Cable 0.24 dB by 20 m / 1000 = 0.0048 Splice @ RPF Waveguide 0.05 dB 1 0.0500

Spur to Spiral

Cable 0.24 dB by 40 m / 1000 = 0.0096

Segment RPF_N16 to RPF_N15 Depth to CPF 10 hops

Splice to Segment (Through Splice) 0.05 dB by 1 splice 0.0500 Cable 0.24 dB by 18301 m / 1000 = 4.3922 Splice on Segment (Through Splice) 0.05 dB by 4 splice 0.2000

Segment RPF_N15 to RPF_N14 Depth to CPF 9 hops

Splice to Segment (Through Splice) 0.05 dB by 1 splice 0.0500 Cable 0.24 dB by 9297 m / 1000 = 2.2313 Splice on Segment (Through Splice) 0.05 dB by 2 splice 0.1000

Segment RPF_N14 to RPF_N13 Depth to CPF 8 hops

Splice to Segment (Through Splice) 0.05 dB by 1 splice 0.0500 Cable 0.24 dB by 6034 m / 1000 = 1.4482 Splice on Segment (Through Splice) 0.05 dB by 1 splice 0.0500

Segment RPF_N13 to RPF_N12 Depth to CPF 7 hops

Splice to Segment (Through Splice) 0.05 dB by 1 splice 0.0500 Cable 0.24 dB by 5457 m / 1000 = 1.3097 Splice on Segment (Through Splice) 0.05 dB by 1 splice 0.0500

Segment RPF_N12 to RPF_N10 Depth to CPF 6 hops

Splice to Segment (Through Splice) 0.05 dB by 1 splice 0.0500 Cable 0.24 dB by 5642 m / 1000 = 1.3541 Splice on Segment (Through Splice) 0.05 dB by 1 splice 0.0500

Segment RPF_N10 to RPF_N8 Depth to CPF 5 hops

Splice to Segment (Through Splice) 0.05 dB by 1 splice 0.0500 Cable 0.24 dB by 3703 m / 1000 = 0.8887 Splice on Segment (Through Splice) 0.05 dB by 0 splice 0.0000

Segment RPF_N8 to RPF_N7 Depth to CPF 4 hops

Splice to Segment (Through Splice) 0.05 dB by 1 splice 0.0500 Cable 0.24 dB by 2480 m / 1000 = 0.5952 Splice on Segment (Through Splice) 0.05 dB by 0 splice 0.0000

Segment RPF_N7 to RPF_N6 Depth to CPF 3 hops

Splice to Segment (Through Splice) 0.05 dB by 1 splice 0.0500 Cable 0.24 dB by 1895 m / 1000 = 0.4548 Splice on Segment (Through Splice) 0.05 dB by 0 splice 0.0000

Segment RPF_N6 to RPF_N5 Depth to CPF 2 hops

Splice to Segment (Through Splice) 0.05 dB by 1 splice 0.0500 Cable 0.24 dB by 1445 m / 1000 = 0.3468 Splice on Segment (Through Splice) 0.05 dB by 0 splice 0.0000

Segment RPF_N5 to CPF Depth to CPF 1 hops

Splice to Segment (Through Splice) 0.05 dB by 1 splice 0.0500 Cable 0.24 dB by 3900 m / 1000 = 0.9360 Splice on Segment (Through Splice) 0.05 dB by 1 splice 0.0500

CPF Internal Distribution

Splice @ ODF 0.05 dB 1 0.0500 Pig Tail Loss 0.50 db 0.5000

Allowances

Maintenance & Ageing allowance - dB 3.00 dB 3.0000 Patch Cord Allowance - both ends - dB 1.50 dB 1.5000

TOTAL Splices 23 TOTAL Length (metres) 58214 Corning SMF-28e+ @ Fibre Type 1490nm TOTAL Fibre Insertion Loss plus Allowances 20.6214 Figure 26: Fibre Loss Calculation for RPF N16 (generated by [RD8])

2018-11-20 Page 60 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.3.3.3 Cable reticulation Four potential approaches exist for the reticulation of the majority of the cable (in this case, for the spirals) across the site. These are overhead, underground in conduit, underground through microduct and underground direct buried.  Overhead requires the use of either wooden or metal poles, and the cable is of a construction to enable it to be strung between them.  Underground in conduit describes the cable being run through PVC or HDPE pipework between locations, usually designed with access pits placed on a regular basis to permit installation access.  Underground through microduct describes the use of specialised tube that permits the installation of cable through the duct by the use of compressed air to minimise friction and push the cable along. As above, this is usually designed with access pits placed on a regular basis to permit installation access.  Underground direct buried refers to the practice of installing the cable directly in the trench with no protective conduit. The exterior of the cable is manufactured of suitable covering (typically nylon composite) to permit this. For the Australian site, it exists as a single site with little or no requirement to obtain easements (wayleaves / servitudes) for installation of the cables. The observatory is presently managed under the auspices of CSIRO in conjunction with the pastoral activity on the site. Routes for the cables on the site will follow a direct point-to-point path. The cable routes will be established initially via a desktop survey informed by constraints presented by the topography, geology, ecology and cultural heritage of the intended route. Each route will be validated with relevant stakeholders and the contractor. Additional small scale surveys may be required around areas of significance to provide final clearance for the route. This desktop survey will provide a bounded description of the route for the contractor (for both shared and dedicated trench segments. The contractor will be asked to provide a Work Method Statement, inclusive of approvals process, to address procedures to address any localised obstacles or otherwise necessary changes of route. The majority of the existing ASKAP cables have been installed as underground direct buried. However, based on the experience providing fibre connectivity to the new power station for ASKAP with the fibre underground in conduit, it is proposed to install the cable along the spirals in a similar manner. This method of installation provides both maintenance and installation program co-ordination benefits.35 Cost, labour and logistics make the other alternative solutions (overhead, underground through microduct) unfeasible. Overhead and underground in microduct have been used successfully on the South African MeerKAT site. Underground in conduit has the following advantages: a) Protected by the conduit the cable is isolated from the extremes of temperature changes caused by the weather b) Mechanical protection of the cable for soil/earth movement and other hazards c) Long term maintenance is simplified.

35 [RD6] SADT.LINFRA.MDAL-0210

2018-11-20 Page 61 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

The work involved in using the shared trench includes multiple steps, including by the INAU element - excavation of the trench, installation of the HV cable, installation of electrical service barriers and marker tapes, backfill and compaction. LINFRA will co-ordinate with these steps to install the conduit and communication service marker tapes. The contractor excavating the trench be responsible for management of any spoil and re-instatement arising. The costing and methodology for the use of a shared trench shall be aligned with the INAU element design before construction. The impact on cost and program will be driven by their approach to the rollout of the HV infrastructure on site, and the construction contractor’s interpretation of their specification. It should be noted, that shared trenching has the following disadvantages: a) Minor increase in risk of damage due to shared trench arrangements b) Installation works are required to be scheduled together 4.3.3.4 Distance and Optical Constraints Finally, it is necessary to consider the fibre losses of the link that the equipment would be required to function with. We have applied a distance bin of 80 km of standard Single Mode [AS1] ITU-T G.652.D fibre cable in conceiving our design. This aligns with industry standards around commonly used enterprise grade fibre optics transceivers. For the long range optics, we have noted a total optical insertion loss of 24 dB as our design limit. The individual spiral fibre distances, from the furthest RPF back to the CPF based on the minimum spanning tree, were 58.1km (north spiral, RPF N16), 50.0 km (east spiral, RPF E16), and 42.4km (south spiral, RPF S16), all remaining within the 80 km distance bin limit.36 The Power Station is assumed to be located 5.0 km south of the CPF.37 This then leaves the design with the constraint of ensuring that the amount of anticipated optical loss remains within the capability of the equipment to handle. In Table 11, extracted from calculations in the [RD8] SKA1 LINFRA LOW Cost Model, losses have been modelled with the following considerations:  the use of standard [AS1] ITU-T G.652.D fibre  wavelengths in use are 1547.72, 1548.53, 1550, 1560.21 and 1558.98 nm  optical loss at 0.24 dB per km  cable routes follow the spanning tree layout. For the purposes of our analysis, we have used as our reference Corning SMF-28e+ fibre, which has the following maximum optical attenuation characteristics:  0.24 dB per km at 1490 nm  0.20 dB per km at 1550 nm Newer cable variants improve on these figures.

36 Note that these distances are not to be confused with the length of cable required for each spiral. The total length of cable required including all branches is 58.7 km (north spiral), 66.8 km (east spiral) and 55.0 km (south spiral). Actual installed distances and cable required will change as later survey and design work will influence these figures. Our modelling has included a 5% allowance to account for this. 37 [RD6] SADT.LINFRA.MDAL-0286

2018-11-20 Page 62 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Nearest Depth TOTAL Cumulative RPF distance from LINK FIBRE Distance node ID CPF LOSS RPF_E5 CPF 1 1805 6.15 RPF_E6 RPF_E5 2 3245 6.54 RPF_E7 RPF_E6 3 5133 7.05 RPF_E8 RPF_E7 4 7614 7.69 RPF_E9 RPF_E8 5 10879 8.53 RPF_E10 RPF_E8 5 11246 8.61 RPF_E11 RPF_E10 6 14166 9.36 RPF_E12 RPF_E11 7 19533 10.75 RPF_E13 RPF_E11 7 22295 11.47 RPF_E14 RPF_E13 8 30008 13.47 RPF_E15 RPF_E14 9 41551 16.44 RPF_E16 RPF_E15 10 50061 18.63 RPF_N5 CPF 1 3900 6.70 RPF_N6 RPF_N5 2 5345 7.10 RPF_N7 RPF_N6 3 7240 7.60 RPF_N8 RPF_N7 4 9720 8.25 RPF_N9 RPF_N8 5 12971 9.08 RPF_N10 RPF_N8 5 13423 9.19 RPF_N11 RPF_N10 6 18733 10.56 RPF_N12 RPF_N10 6 19065 10.64 RPF_N13 RPF_N12 7 24522 12.05 RPF_N14 RPF_N13 8 30556 13.60 RPF_N15 RPF_N14 9 39853 15.98 RPF_N16 RPF_N15 10 58154 20.62 RPF_S5 RPF_S6 3 6233 7.31 RPF_S6 RPF_S7 2 4792 6.91 RPF_S7 CPF 1 2897 6.41 RPF_S8 RPF_S7 2 5387 7.06 RPF_S9 RPF_S8 3 8650 7.89 RPF_S10 RPF_S8 3 9091 8.00 RPF_S11 RPF_S10 4 13882 9.25 RPF_S12 RPF_S11 5 19758 10.76 RPF_S13 RPF_S11 5 19632 10.73 RPF_S14 RPF_S13 6 26855 12.56 RPF_S15 RPF_S14 7 35288 14.73 RPF_S16 RPF_S15 8 42450 16.55 Table 11: Chart of cumulative distance and link loss for each RPF on each spiral

2018-11-20 Page 63 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Nearest Depth TOTAL Cumulative Location distance from LINK FIBRE Distance node ID CPF LOSS Power CPF 1 5050 6.96 Station Table 12: Chart of cumulative distance and link loss for the Power Station

Table 11 and Table 12 demonstrate that for all optical paths our calculation of loss is:  Within our standard distance bin statement of 80 km  Inside our optical loss budget meets our design limit of 24 dB It has not been necessary to use low loss fibre optic cable variants. Care should however be taken during procurement and with the selection of cable offered by vendors to ensure the best of that grade of cable is supplied. The fibre input loss figures and splice losses are based on information received from fibre manufacturers and operators as to what figures are realistically achievable. Splices occur at the termination of each fibre, the waveguides into the CPF and RPF buildings, near each RPF location, and as required due to the drum length of the fibre. The input table to the model is given in Figure 27 below:

Design Fibre Losses

Cable Selection Fibre Type Loss/km dB ODF to Waveguide @ CPF G.652 D 0.24 Spiral G.652 D 0.24 Spur to RPF G.652 D 0.24 Waveguide to Patch Panel @ RPF G.652 D 0.24

Splice @ ODF 0.05 Splice @ Spiral Pit 0.05 Splice @ Through Splice on Spiral 0.05 Splice @ RPF Waveguide 0.05 Splice @ RPF Patch Panel 0.05

Maintenance & Ageing allowance - dB 3.0 Patch Cord Allowance - both ends - dB 1.5 Figure 27: Fibre Loss Input Table to Solution Modelling

2018-11-20 Page 64 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.3.3.5 Technologies and components for the design The choice of cabling technologies and components is going to be a critical driver for the success of the SKA1 LOW Outside Plant Fibre Reticulation Design. The selection and performance requirements for choice of these products will need to consider:  Durability and service life  Reliability  Flexibility  Cost  Reflection of underlying telescope and network design  External system requirements (eg RFI minimisation and control)  Internal design requirements (eg fibre loss and performance)  COTS / commodity nature of product  Regulatory compliance of product

Key components that are to be used are described below. Fibre Fibre cable (Figure 28) is used to provide the connectivity between telescope locations. For the purposes of our design we have divided this into two categories: Backhaul and Distribution. Backhaul is a high core count cable used on the spirals. Distribution is a low core count cable used to branch off the spirals and within the CPF.

Figure 28: Typical Loose Tube Fibre Optic Cable Outdoor Rated38

The fibre can be constructed in several different ways – ribbon, loose tube or tight buffered – and with differing sheathes depending on whether used indoor or outdoor. The [RD8] SKA1 LINFRA LOW Cost Model assumed loose tube was used for the Backhaul and Distribution components.39 The outside sheath of the cable will be marked at regular intervals (typically 1 metre) with details of manufacturer, date of manufacture, cable type and size, standards and regulatory compliance, and serial number. Other details (such as project or owner) may also be recorded on the sheath.

38 Image – Corning, http://catalog.corning.com/opcomm/en- AU/catalog/ProductDetails.aspx?cid=loose_tube_outdoor_cables_web&pid=10179&vid=77529 39 [RD6] SADT.LINFRA.MDAL-0282

2018-11-20 Page 65 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Fibre Patch Panel Patch panels (Figure 29) provide a means to connect the equipment to the fibre via patch leads. In our design, these will be rack mounted. They are used at the RPF, but will also be used for the termination of cables elsewhere.

Figure 29: Typical 1 and 2 RU Fibre Patch Panels installed at MRO40

For our design, LC Angled connectors will be used. Waveguide and Optical Fibre Entry Box At the CPF and RPF buildings, a waveguide and associated entry box will be provided by the INAU element. The waveguide permits the inner tubes or ribbons of our fibre cables to enter the building whilst containing the RFI emissions from the equipment inside the building. The entry box (Figure 30) permits access to the cables whilst retaining the RF integrity of the outer skin of the building (in normal use)

Figure 30: Optic Fibre Entry Design – INAU concept41

A similar mechanism will exist at the RPF

40 Image – SADT Site Visit to MRO, July 2016 41 Drawing – INAU, [RD41] SKA-TEL-INAU-0000015-BLDS-MROLFAA-RE Buildings Draft Detail Design Report

2018-11-20 Page 66 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Cable Vault This is a structure (Figure 31) provided adjacent to the CPF building, again provided by the INAU element, which provides the mechanism to physically contain and aggregate the conduits and cables before they enter the building via the Optical Fibre Entry Boxes.

Figure 31: Cable vault outside (left of image) ASKAP Control Building42

Splices and Splice Tray A splice joins individual fibre cores together to form a longer optical link from one or more cable segments. The spliced fibre core is then mechanically protected with a splice protector which in turn is held in a splice tray (Figure 32). For our design, all fibre cores will be fusion spliced together. The splicing arrangements for ribbon fibre (if used) are similar (Figure 33), except that multiple cores are spliced simultaneously.

Figure 33: Splice tray holding ribbon fibre splices44

Figure 32: Single core splice tray (typical)43

42 Image – SADT Site Visit to MRO, July 2016 43 Image – AFL Global, https://www.aflglobal.com/Products/Fiber-Outside-Plant/LightLink-Splice-Closure-Accessories- Aerial-Acc/LightLink-Splice-Trays/LightLink-2450-Fiber-Optic-Splice-Trays.aspx 44 Image – SADT Site Visit to MRO, July 2016

2018-11-20 Page 67 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Optical Distribution Frame A Optical Distribution Frame (ODF) (Figure 34) is a specialised, higher density version of the fibre patch panel and used where large volumes of fibre cores need to be interconnected. They can either be free standing in their own cabinet or rack mounted. For our design, we are providing a single rack mounted in the Cabinet Room.

Figure 34: Typical Optical Distribution Frame (ODF)45

Conduit – HDPE Conduit is used as a pathway to reticulate the fibre outside the buildings. HPDE conduit (Figure 35) is extruded as a single length and is pliable making it suitable for following curved paths. It is typically manufactured and shipped as 140 metre coils. In this design, it will be used along the spirals. Segments may be joined together with thermal welding.

Figure 35: Coils of HPDE conduit46

45 Image – Corning, Centrix ODF http://csmedia.corning.com/opcomm//Resource_Documents/product_family_specifications_rl/Centrix_Platform_NAFTA _AEN.pdf

2018-11-20 Page 68 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Conduit – PVC PVC Conduit (Figure 36) is used for short, straight runs. In our design, we have used this between the splice pit on the spiral and the RPF. The conduit typically comes in 6 metre lengths with a socket connection at one end. The conduit is jointed using a solvent to weld the segments together.

Figure 36: PVC Communications Conduit47

Marker Post Marker posts (Figure 37) are used to denote the presence of pits and services below. They typically include a sign on the top portion with contact details of the operator of the service. They are advantageous to enable maintenance staff to find the location of pits in situations where the ground level has changed and the pit has been covered with soil.

Figure 37: Marker posts besides AARNet pit at MRO48

46 Image – Australian Custom Pipes, http://www.acpipes.com.au/gallery/PolyPipes/68/ 47 Image – Vinidex, http://www.vinidex.com.au/products/electrical-and-communications/ordering-information-2/ 48 Image – SADT Site Visit to MRO, July 2016

2018-11-20 Page 69 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Pit / Manhole / Handhole Pits provide installation and maintenance access to the conduit and fibre optic cables. In Australia, pits are manufactured in standard sizes reflecting industry practice inherited from the incumbent carrier. In this design, the smaller P5 pit (Figure 38) is used for hauling cables directly through, and the larger P6 pit (Figure 39) is used to hold service loops and cable joints. Pits should be fitted with gaskets (Figure 40) between the body of the pit and the lid to minimise silt, insects and animals entering the pit.

Figure 38: Typical Australian P5 polyethylene pit Figure 39: Typical Australian P6 polyethylene pit (shown with concrete lid)49 (shown with concrete lids)50

Figure 40: Typical gasket fitted in plastic pit51

49 Image – Auscon Industries, http://acindustries.net.au/catalogue?route=product/product&path=37&product_id=75 50 Image – Auscon Industries, http://acindustries.net.au/catalogue?route=product/product&path=37&product_id=76 51 Image – ACO Australia, http://www.acoaus.com.au/pdf/nbn-compliant-gaskets.pdf

2018-11-20 Page 70 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Underground Fibre Optic Joint Enclosure To join segments of fibre optic cable together, or to allow individual cores to be taken and connected to other cables, splice trays are placed in sealed underground enclosures (Figure 41).. These are then securely installed in a pit (Figure 42). In this design, these will be to connect each RPF into the backhaul fibre along the spirals, in addition to joining segments of cable together.

Figure 42: Splice enclosure installed in P8 pit (typical)53

Figure 41: Underground fibre optic splice and jointing enclosure52

Cable labels Labels will be attached to patch leads (Figure 43) and both internal and external fibre cables (Figure 44). This will permit easy identification.

Figure 44: Outside Plant cable tag55

Figure 43: Patch lead labels54

52 Images – Commscope, http://www.commscope.com/Docs/FIST-GCO2-FIST-Generic-Closure-Organizer-318885EU.pdf 53 Image - UWA 54 Image – Brady, https://www.comtecdirect.co.uk/product/brady-tls2200-fibre-optic-tags/PG0632/776929 55 Image – Lazertek Australia, http://www.lazertekaustralia.com.au/tags1.php

2018-11-20 Page 71 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Marker Tape To protect services buried underground and to alert anyone excavating of their presence (if other safety and engineering controls have failed), marker tape (Figure 45) is used. The tape is a thin plastic film with writing indicating the nature of the service below. Illustrated below is the communications variant, similar variants exist for the electrical and other services in differing colours. Detectable and traceable variations of marker tape exist.

Figure 45: Underground Marker Tape56

4.3.3.6 Compliance with User requirements To confirm compliance we have checked against our requirements stated in §4.3.1, and specifically the constraints for performance of the fibre as installed that are noted in Table 5 and Table 6. This is undertaken by modelling the fibre losses from the CPF to the most distant RPF. Ultimately all links to the RPFs must fall within specification. Selection of SADT equipment has been undertaken around a maximum loss of 24 dBs (including aging and patch lead allowances) As modelled, our selection of cable compliant with [AS1] ITU-T G.652.D fibre cable meets the requirements at all RPFs, and at the Power Station.

56 Image – Pakaflex, http://www.pakaflex.com.au/prodgn09.htm

2018-11-20 Page 72 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.4 SKA1 LOW Inside Plant – Fibre and Other Cable Distribution

User requirements

The SADT sub-elements require reticulation of connectivity between elements and sub-elements inside the SKA1 buildings:  via fibre o from the ODF to locations inside the CPF (for example, to facilitate a NSDN connection out to an RPF), o directly between locations (for example, the link between NSDN equipment and CSP-SDP equipment located in the Cabinet Room at the CPF) , and  via high quality coaxial cable for reticulation of 1 PPS, 10 MHz and 100 MHz signals (for example, between SAT.CLOCKS equipment and LFAA) For this design, at the CPF building, these will occur between different rows of cabinets, and from the Cabinet Room to Maser (Timing and Frequency) Room. Elsewhere, at the RPF, EOC, SOC and SPC buildings similar reticulation will occur between cabinets and equipment locations. With the exception of timing and frequency signals, all connectivity between rows of cabinets and rooms will be via fibre. This may be a mix of fixed cables, patch leads, pre-terminated MTP cables and adapters. The intention of this design is that no copper cables carrying Ethernet will extend between rows, although cables may run between adjacent cabinets but will be the responsibility of individual sub-elements. SADT will provide some reticulation of fibre connectivity between SADT and other elements inside SKA1 buildings. These will be run on cable tray provided by INAU. Specific requirements are contained within the interface documents referenced in Table 13 and Table 14.

Element Relevant EICD

LFAA: [RD9] 100-000000-026 Interface Control Document SADT to LFAA

INAU [RD10] 100-000000-024 SKA1 LOW Telescope Interface Control Document SADT to INAU Table 13: Inside Plant interfaces with other telescope elements

SADT Sub-Element Relevant IICD [RD1] SKA-TEL-SADT-0000445_ICD- LINFRA to NSDN (LOW) Internal Interface Document NSDN: [RD2] SKA-TEL-SADT-0000439_ICD -LINFRA to SAT.STFR.UTC (LOW) Internal Interface STFR.UTC: Document [RD3] SKA-TEL-SADT-0000440_ICD_ LINFRA to SAT.STFR.FRQ (LOW) Internal Interface STFR.FRQ Document (THU) [RD11] SKA-TEL-SADT-0000438_ICD LINFRA to CSP-SDP (LOW) Internal Interface Document CSP-SDP Table 14: Inside Plant interfaces with other SADT sub-elements

2018-11-20 Page 73 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Key cable connectivity summarised from all interface documents above are shown below in Table 15 through Table 20.

Cable Type Interface Type Description LINFRA Provision

Fibre LFAA ODF to LFAA Fibre racks in CPF Cabinet Room

Fibre NSDN ODF to NSDN Fibre racks in CPF Cabinet Room

Fibre SAT.STFR.FRQ ODF to Fibre SAT.STFR.FRQ racks in CPF Maser Room

Fibre SAT.STFR.UTC ODF to Fibre SAT.STFR.UTC racks in CPF Maser Room

Fibre SAT.CLOCKS ODF to Fibre SAT.CLOCKS racks in CPF Cabinet Room Table 15: Key Connectivity at CPF – from ODF

2018-11-20 Page 74 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Cable Type Interface Type Description LINFRA Provision

Fibre NSDN to NSDN Inter-room and inter-row Fibre cables connecting NSDN core routers to NSDN top of rack and utility switches

Fibre NSDN to INAU Fibre and copper If required, Installation interconnects assistance

Fibre NSDN to LFAA Fibre interconnects If required, Installation assistance

Fibre NSDN to TM Inter-row cables Fibre connecting NSDN core routers to TM

Fibre NSDN to CSP Inter-row cables Fibre connecting NSDN core routers to TM

Fibre NSDN to CSP-SDP Fibre interconnects If required, Installation assistance

Fibre CSP-SDP to CSP-SDP Fibre interconnects If required, Installation assistance

Fibre CSP-SDP to CSP Fibre interconnects If required, Installation assistance

Coaxial SAT.CLOCKS to SAT.FRQ 1 PPS and 10 MHz Signal If required, Installation Cables assistance

Coaxial SAT.CLOCKS to SAT.UTC 1 PPS and 10 MHz Signal If required, Installation Cables assistance

Coaxial SAT.CLOCKS to LFAA 1 PPS and 10 MHz Signal If required, Installation Cables assistance Table 16: Key Connectivity at CPF - Direct

2018-11-20 Page 75 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Cable Type Interface Type Description LINFRA Provision

Fibre NSDN to NSDN Fibre and If required, copper Installation interconnects assistance

Fibre NSDN to INAU Copper If required, interconnects Installation assistance

Coaxial SAT.FRQ to LFAA 10 MHz Signal If required, Cables Installation assistance

Coaxial SAT.UTC to LFAA 1 PPS Signal If required, Cables Installation assistance Table 17: Key Connectivity at RPF

Cable Type Interface Type Description LINFRA Provision

Fibre NSDN to NSDN Fibre and If required, copper Installation interconnects assistance

Fibre NSDN to INAU Copper If required, interconnects Installation assistance

Fibre NSDN to TM Copper If required, interconnects Installation assistance

Fibre NSDN to CSP-SDP Fibre If required, interconnects Installation assistance

Fibre CSP-SDP to CSP- Fibre If required, SDP interconnects Installation assistance Table 18: Key Connectivity at EOC

2018-11-20 Page 76 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Cable Type Interface Type Description LINFRA Provision

Fibre NSDN to NSDN Fibre and If required, copper Installation interconnects assistance

Fibre NSDN to INAU Copper If required, interconnects Installation assistance

Fibre NSDN to SDP Fibre If required, interconnects Installation assistance

Fibre NSDN to CSP-SDP Fibre If required, interconnects Installation assistance

Fibre CSP-SDP to CSP- Fibre If required, SDP interconnects Installation assistance

Fibre CSP-SDP to SDP Fibre If required, interconnects Installation assistance Table 19: Key Connectivity at SPC

Cable Type Interface Type Description LINFRA Provision

Fibre NSDN to NSDN Fibre and If required, copper Installation interconnects assistance

Fibre NSDN to INAU Copper If required, interconnects Installation assistance

Fibre NSDN to TM Copper If required, interconnects Installation assistance Table 20: Key Connectivity at SOC

Where “Installation assistance” is noted, this is regarded as outside the scope of the LINFRA sub-element design and is the responsibility of the other elements and sub-elements. The LINFRA cost-model will however make an allowance to assist these elements and sub-elements to install the cable on the cable pathways provided by LINFRA.

2018-11-20 Page 77 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Design of Internal Fibre and Other Cable Distribution at CPF

4.4.2.1 Overview Reticulation of the Inside Plant Fibre at the SKA1 LOW CPF will occur in three different ways:  Inter-room – Connecting equipment in separate rooms of the building together  Inter-row – Connecting equipment in different cabinet rows together  Intra-row – Connecting equipment in the same cabinet row but different cabinets together The topology of the Inside Plant Fibre can be further categorised:  Originate from the ODF – cabling forms a star topology with the ODF as the hub where cross connects occurs between different cables  Point to Point – cabling runs directly between equipment located in different cabinets Fibre cables originating from the ODF will be terminated on fibre patch panels at the cabinet end. Point to Point cables may either be long simplex or duplex patch leads, multicore cables terminated with MPO connectors at each end, or cables terminated with fibre patch panels at each end. Where a cable is terminated on a patch panel, a patch is lead is then used to connect from the patch panel to the equipment. 4.4.2.2 Inter-room connectivity Cables that provide Inter-room connectivity will terminate at the ODF. These will need to be installed:  In a permanent fashion via appropriate cable pathways (eg designated overhead cable tray shown in Figure 46)  Labelled appropriately at both ends and along pathway as appropriate  Ensuring compliance with RFI and EMC mitigation measures present in the building  Ensuring fire protection systems remain effective and compliant  In compliance with local regulatory arrangements

Figure 46: Photograph of overhead cable tray

A fibre cross connect cable (patch lead) may be required at the ODF. For example, to connect an item in the Cabinet Room to the Maser Room the connection could follow the following path:

2018-11-20 Page 78 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

To connect equipment between rooms (Figure 47):  cables will exist from each rack back to the ODF  a patch lead in the first rack will connect to a patch panel in the same cabinet  similarly in the second rack the same type of connection will be made from the equipment to the patch panel  at the ODF, a patch lead will interconnect the cables that run to the two racks

ODF

TIMING AND FREQUENCY ROOM CABINET ROOM

Figure 47: Typical inter-room cabinet elevation57

The cables connecting the GNSS receiving equipment will be treated similarly to the above.58

57 Drawing [D8] 58 [RD6] SADT.LINFRA.MDAL-0211

2018-11-20 Page 79 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.4.2.3 Inter-row connectivity (Cabinet Room only) To connect equipment between rows (Figure 48):  cables will exist from each cabinet back to the ODF  a patch lead in the first cabinet will connect to a patch panel in the same cabinet  similarly in the second cabinet the same type of connection will be made from the equipment to the patch panel  at the ODF, a patch lead will interconnect the cables that run to the two cabinets

ODF

CABINET ROOM Figure 48: Typical inter row cabinet elevation59

The preferred mechanism to reticulate these connections will be pre-terminated cables. In some circumstances, it may be required to run an MPO cable between cabinets to effect this connection as opposed to running via the ODF These will need to be installed:  Via appropriate cable pathways (eg designated overhead tray)  Labelled appropriately at both ends and along pathway as appropriate 4.4.2.4 Intra-row connectivity A patch lead of suitable length will be run directly between the equipment (Figure 49). It will be routed through each rack/cabinet up to cable tray or trough above and then back down to enter the second cabinet.

59 Drawing [D8]

2018-11-20 Page 80 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Figure 49: Typical intra-row cabinet elevation60

These will need to be installed:  Via appropriate cable pathways (eg designated overhead tray)  Labelled appropriately at both ends

Design of Internal Fibre and Other Cable Distribution at RPF

It is expected that most connectivity between elements and sub-elements at the RPF will be formed of intra-row cables (similar to §4.4.2.4).61

Design of Internal Fibre and Other Cable Distribution at EOC

It is expected that most connectivity between elements and sub-elements at the EOC will be formed of intra-row cables (similar to §4.4.2.4).62 There will be interfaces to the building cabling that will occur, but these are outside the scope of the LINFRA sub-element.

60 Drawing [D8] 61 [RD6] SADT.LINFRA.MDAL-0212 62 [RD6] SADT.LINFRA.MDAL-0213

2018-11-20 Page 81 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Design of Internal Fibre and Other Cable Distribution at SPC

It is expected that most connectivity between elements and sub-elements at the SPC will be formed of intra-row cables (similar to §4.4.2.4).63 There will be interfaces to the building cabling that will occur, but these are outside the scope of the LINFRA sub-element. It is assumed at the SPC that any facility cabling will be undertaken by the operator of the facility.64

Design of Internal Fibre and Other Cable Distribution at SOC

It is expected that most connectivity between elements and sub-elements at the SPC will be formed of intra-row cables (similar to §4.4.2.4).65 There will be interfaces to the building cabling that will occur, but these are outside the scope of the LINFRA sub-element.

Analysis of SKA1 LOW Internal Fibre and Other Cable Distribution

4.4.7.1 Site Constraints The design of the CPF building will present constraints on the Internal Fibre Distribution. These include:  Allocation of equipment locations within the building  Scale and dimensions of the building  Location and routing of cable tray  Shared resources  Radio-frequency and emissions controls within the building Where not explicitly reserved (for example, a cable tray shared with other consortia such as LFAA and CSP), co-ordination will need to occur to ensure correct ordering of installation. 4.4.7.2 Equipment Constraints and Selection of technologies for the design Connections between equipment will be effected by the use of pluggable optics. These will comply with the relevant industry or IEEE standard. The standard will note the suitable fibre optic cable (and reference to the relevant ITU standard), its installed performance, and the functional distance limitations. For example, 100GBase-SR4 has a distance limitation of 150 metres over OM4. It is not expected, however for any distance limitation to be encountered due to the restricted size of the building and the potential interconnections. It is expected that a mix of single and multi-mode cable will be used within the CPF. This will involve a mix of configurations (duplex, simplex, MPO parallel optics) with preference for LC/APC, LC/UPC and MPO connectors. From the lessons learnt from ASKAP (See [RD12]), there is a strong recommendation that single mode cable be selected as first preference for any deployment. 4.4.7.3 Cable reticulation Cables within the buildings will be run along cable pathways. The type of cable pathway used will depend on the permanence of the cable (patch lead versus permanently installed). Illustrations and further

63 [RD6] SADT.LINFRA.MDAL-0214 64 [RD6] SADT.LINFRA.MDAL-0279 65 [RD6] SADT.LINFRA.MDAL-0215

2018-11-20 Page 82 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

description of these are discussed in §4.7.2.1 PVC Cable Trough is the preferred means of reticulating individual fibre patch leads, with basket tray being preferred for all other cable types. These are installed above cabinet/rack height. Spills will be incorporated to manage cable bend radius and strain. Cable troughs and trays shall follow the room grid, longitudinally following above each row of cabinets/racks, with crossings between rows located at their ends. The exact routes will be co-ordinated with other items attached to the ceiling (for example, lights and fire detection/prevention) Separate cable troughs and trays may be provided to segregate cables based on type and function. For example, coax cables associated with timing and frequency signals would be run separately to minimise disturbance. Individual cable routes between locations will be chosen to minimise cable length and congestion. Where necessary, provision will be made to spool excess cable length (in the case of calibrated cables and standard cable lengths). 4.4.7.4 Compliance with User requirements To confirm compliance we have checked against our requirements stated in §4.1.4.1, and specifically the constraints for performance of the fibre as installed. In Table 21, all cable types from the requirements are within the maximum link length and meet the performance requirements of the type. Interface Type Cable Type Maximum Length Anticipated Comments Recommended Distance

1000BASE-SX ITU-T G.651.1 550 m < 100 m OM4 MM

1000BASE-LX10 ITU-T G.652.D 10 km < 100 m OS1 / OS2 SM

IEEE 802.3ab Category 6A 90 m < 90 m

10GBASE-SR ITU-T G.651.1 400 m < 100 m OM4 MM

10GBASE-LR ITU-T G.652.D 10 km < 100 m OS1 / OS2 SM

100GBASE-SR466 ITU-T G.651.1 100m < 100 m OM4 MM (MPO/MTP)

100GBASE-LR467 ITU-T G.652.D 10 km < 100 m OS1 / OS2 SM

IEEE 802.3an Category 6A 90 m < 90 m

IEEE 802.3 Category 6A 90 m < 90 m 10 Mb and 100 Mb

1 PPS See [RD13] < 100 m

10 MHz See [RD13] < 100 m

100 MHz See [RD13] < 100 m

Table 21: Summary of Inside Plant Interfaces

66 Not presently used, but anticipated over life of telescope 67 Not presently used, but anticipated over life of telescope

2018-11-20 Page 83 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.5 SKA1 LOW Fibre for CSP-SDP connectivity

User Requirements

SADT.CSP-SDP is the SADT sub-element that provides connectivity from the CPF to the SPC via the EOC for the interconnection between the CSP and SDP elements, and the inter-location connectivity for the SADT.NSDN sub-element. Interfaces between SADT.LINFRA and SADT.CSP-SDP are defined in [RD11] SKA- TEL-SADT-0000438_ICD-LINFRA to CSP-SDP (LOW) Internal Interface Document. Figure 50 from the [RD14] CSP-SDP Detail Design Report (LOW) provides an overview of the design for SADT.CSP-SDP.

Figure 50 Extract from [RD14] CSP-SDP Detailed Design Report showing logical connectivity

Assuming that the EOC will be co-located with the existing MRO Support Facility (MSF)68 in Geraldton, fibre infrastructure and connectivity exists between the MRO and Perth to the likely location of the SOC and SOC6970 via services provided via AARNet. No additional work is anticipated. Existing connectivity from the MSF in Geraldton to the MRO site is provided by an existing cable from the MRO Support Facility to the ASKAP Control Building. CSIRO engaged AARNet, the Australian NREN, to install and operate this cable. Similarly existing connectivity from the MSF in Geraldton to the Pawsey Centre in Perth, in Figure 50 from the EOC to SPC, is by AARNet in arrangement with CSIRO. This connection is outside the scope of this document and will likely be covered by the host country arrangements. The design for the SADT.CSP-SDP sub-element requires optical connectivity (fibre) between the ASKAP Control Building and the SKA1 LOW Central Process Facility to connect the Wavelength Selector Switches together. This connection will carry both traffic between the CSP and the SDP, and provide backhaul for the NSDN infrastructure between the CPF and both the SPC and the EOC.

68 SADT.LINFRA-MDAL-0254 69 SADT.LINFRA-MDAL-0255 70 SADT.LINFRA-MDAL-0256

2018-11-20 Page 84 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Summary of proposed implementation

[AD17] ECP-170024 was raised and the option to “Extend the existing CSIRO/AARNet cable” was selected (Figure 51).

ASKAP APL Control J70 Building

19237m SPIRAL 3 (South)

J25 4089m S5

J26 S6 CPF J29 J32 S9 S12

J27 J28 J30 J31 J33 J34 J35 J36

AA-1 S7 Temporary S8 S10 S11 S13 S14 S15 S16 CPF

EXISTING CABLE JOINT ID J12

CABLE EXTENSION

RPF ID E16 BACKHAUL TRENCHES 9091m

Figure 51: Schematic overview of the fibre extension required for CSP-SDP connectivity (in blue)

This implementation proposes:  to intersect the existing CSIRO cable operated by AARNet (Figure 52, green line) that runs between the MSF in Geraldton and the AKSAP Control Building at the MRO. From the point of intersection at APL Joint 70, located in APL Pit 26, run a 48 core optical fibre (NREN Interconnect Fibre) eastwards to RPF S10 and then alongside Spiral 3 (South) with the SADT.LINFRA sub-element fibre servicing that spiral to the CPF  to trench and install further conduit from the APL Joint 70 to RPF S10 (Figure 52, thick red line)  to either expand the size of conduit installed along Spiral 3 (South) or run a second conduit along that path of similar size (second option assumed for costing71) following the same route (Figure 52, thick red line with blue overlay) towards the CPF to accommodate this cable

71 [RD6] SADT.LINFRA.MDAL-0280

2018-11-20 Page 85 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Figure 52: Geographic overview of “Fibre Gap” connection

This enables the following items to occur:  A connection to temporary CPF for AA-1, likely via temporary over-ground fibre will be used (from the newly established pit adjacent to the CPF location)  Connection of temporary construction and accommodation camps  Achieves better alignment with construction programme  Provides shorter distance for future transmission upgrades Further activity is required to develop a scope of work for CSIRO and AARNet to implement, likely as part of the host country arrangements, and as such will not be considered further in this design. It is assumed that the construction techniques required to provide the trench, conduit and cable to extend the cable will be similar to that for the Outside Plant.72 The cable will be terminated at the CPF on a dedicated patch panel of a standard type used by the NREN. The connector type will likely be SC APC.

72 SADT.LINFRA.MDAL-0284

2018-11-20 Page 86 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.6 SKA1 LOW Fibre and Enclosures for GNSS Equipment

User Requirements

SADT.CLOCKS is the SADT sub-element that provides the timescale for the telescope with equipment housed at the CPF. Interfaces between SADT.LINFRA and SADT.CLOCKS are defined in [RD15] SKA-TEL- SADT-0000441_ICD-LINFRA to SAT.CLOCKS (LOW) Internal Interface Document. SADT.CLOCKS uses GNSS receivers and antennas to accomplish time transfer between the local timescale and remote timescales with reference to UTC(k). This requires the presence of antennas outside the CPF building with a view of the sky. To enable SADT.CLOCKS to mitigate RFI emissions and comply with the [AD5] SKA EMI/EMC STANDARDS AND PROCEDURES, the SADT.LINFRA sub-element includes:  Optical fibre to enable GNSS signals to pass through the double skin of the CPF  A suitable shelter to house the portable GNSS receiver equipment and references used for calibration of the three permanent GNSS antennas, frequency to optical convertors, and receivers.  Optical fibre to enable 10MHz reference signals to pass through the double skin of the CPF73 The location of the GNSS Antennas and GNSS Calibration Shelter require further design work post-CDR and co-ordination with the INAU element. [RD13] SKA-TEL-SADT-0000330_DDD SADT Clocks Detailed Design Document discusses the requirements for the antenna and mounts. The optical fibre for the GNSS signals and 10MHz reference signals will consist of single mode [AS1] ITU G.652.D fibre74

Summary of proposed implementation

The implementation will consist of three parts:  Optical fibre for GNSS signals  GNSS Calibration Shelter  Optical fibre for reference signals to GNSS Calibration Shelter The integration and verification for these will be the same as the other Outside Plant 4.6.2.1 Optical fibre for GNSS signals The GNSS antennas will be co-located at the same location outside the building. Each antenna requires an individual fibre for transmission of the GNSS signal. Three antennas are required and a single enclosure provided by the SADT.CLOCKS sub-element can contain the three Frequency-to-Optical Convertors needed. The fibre will run from a patch panel located in the CPF Maser room, spliced at a wave guide to exit the building, and extended via a suitable pathway to the location of the enclosure housing the Frequency-to- Optical Convertors for the GNSS signals. In the enclosure housing the Frequency to Optical convertors the fibre will enter the enclosure via a suitable gland, and be spliced to pigtails inside the enclosure. The fibre cable used will have a spare core reserved for each convertor, and be a minimum of 12 cores.

73 [RD6] SADT.LINFRA.MDAL-0263 74 [RD6] SADT.LINFRA.MDAL-0264

2018-11-20 Page 87 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.6.2.2 GNSS Calibration Shelter The GNSS Calibration Shelter will be a shielded enclosure to mitigate emissions from the SADT.CLOCKS GNSS Travelling Receiver Box (which contains a GNSS receiver, timescale generator, distribution amplifier and laptop) The shelter will provide to the GNSS Travelling Receiver Box:  Power  Housing for termination of the fibre carrying the reference signal  Environmental controls to meet [AS18] ETSI EN 300 019-1-3 Class 3.2 that provide a stable temperature and humidity The shelter will be suitable for the chosen installation location. Further development of the specification for the GNSS Calibration Shelter will occur as part of post-CDR work. 4.6.2.3 Optical fibre for reference signals to GNSS Calibration Shelter The GNSS Travelling Receiver Box requires a 10MHz reference signal. This will be transmitted over optical fibre from the CPF Maser Room. The fibre will run from a patch panel located in the CPF Maser room, spliced at a wave guide to exit the building, and on via a suitable pathway to the location of the GNSS Calibration Shelter where it will be terminated on a suitable patch panel. The fibre cable will have a single core used for the reference signal, alongside a spare, and be a minimum of 12 cores.

2018-11-20 Page 88 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.7 Ancillary Items

User requirements

The LINFRA work package has been assigned additional ancillary items necessary for the complete installation of the SADT work packages for SKA1:  Management of cabling within cabinets and racks  Blanking and ventilation panels within cabinets and racks  Provision of PDU (Power Distribution Units) within allocated cabinets and racks  Provision of an Uninterruptible Power Supply (for NSDN) at: o each RPF o at the SOC o at ancilliary locations such as the Power Station  Power distribution within cabinets and racks  Provision of KVM Switches at the CPF and SPC buildings  Earth bonding within cabinets and racks  Documentation systems to record infrastructure installed under SADT.LINFRA These items are either:  Needed for the installation of either the Inside or Outside Plant cables; or are  Emergent requirements from the design of other work packages. Individual work package requirements are summarised in the relevant IICD documents. These ancillary items are described in general terms to ensure their design parameters are captured. These items will exist across the LOW telescope and will primarily be located in the CPF, RPF, EOC, SPC and SOC buildings. Hardware such as cable ties, screws, brackets, and the like are assumed to be supplied as part of these works.

Overview

4.7.2.1 Cable Pathways To permit the reticulation of cables between locations inside the CPF and RPF facility, there are a variety of means to provide a pathway for this to occur. Typically most of these pathways will be provided by INAU but LINFRA may need to augment/extend in certain circumstances. These pathways are typically provided by:  Metal Cable Tray (Figure 53)  Metal Cable Basket(Figure 54)  PVC Cable Trough(Figure 55)  PVC Duct  PVC Conduit These will be installed with associated mounting hardware and cable spills.

2018-11-20 Page 89 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Figure 53: Metal Cable Tray75

Figure 54: Metal Cable Basket76

Figure 55: PVC Cable Trough77

75 Image – EzyStrut, http://www.ezystrut.com.au/products/cable-support-systems/cable-tray-systems/et3/et3/et3g/ 76 Image – Legrand, http://www.legrand.com.au/uploads/tx_sbdownloader/Cablofil_Technical_Guide_2010.pdf

2018-11-20 Page 90 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.7.2.2 Racks and Cabinets 4.7.2.2.1 CPF Cabinet Room The SKA1 LOW CPF Cabinet Room consists of 6 rows of cabinets. These rows are formed of 22 cabinets arranged into 3 groups of 10/11/11 cabinets respectively. The [RD10] SKA1 LOW Interface Control Document SADT to INAU documents the allocation of floor space to SADT element. For the cabinet room, this is a requirement for the space for 9 racks. Each cabinet (Figure 56) has a dedicated fan coil heat exchanger, which is connected to a centralised cooling water system. The heat exchanger will have the ability to reject up to 20kW of sensible heat.78 Details of the cooling water provision are specified in [RD10] SKA1 LOW Telescope Interface Control Document SADT to INAU.

Figure 56: Example of CPF Cabinet (including fan coil unit)79

The cabinet will comply with [AS3] IEC 60297, provide 42 RU of usable space for equipment and provide space for the installation of two (2) power distribution units (PDU). The PDU will be fitted to the rack in a manner which consumes zero RU of usable rack space. The cabinet will have transparent front and rear doors to allow for inspection of the contents without requiring the door to be opened. These doors shall have the capacity to have suitable locks fitted.

77 Image – Panduit, http://www.panduit.com/heiler/ApplicationGuides/SA- FRCB02%20Fiber%20Runner%20Brochure%20PDF%20for%20web%201-07.pdf 78 [RD6] SADT.LINFRA.MDAL-0216 79 Image Schroff. VARISTAR LHX 20 shown, as used for ASKAP

2018-11-20 Page 91 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.7.2.2.2 CPF Maser (Timing and Frequency) Room The SKA1 LOW CPF Maser (Timing and Frequency) Room consists of a single row of five (5) open frame racks (Figure 57) bayed together. The [RD10] SKA1 LOW Interface Control Document SADT to INAU documents the allocation of floor space to SADT for these racks and the masers.

Figure 57: Example of Open Frame Rack with Vertical Cable Manager80

Each rack will comply with [AS3] IEC 60297, provide 42 RU of usable space for equipment and provide space for the installation of two (2) power distribution units at the rear. Between each rack will be vertical cable management at both front and rear. Figure 58 below illustrates a typical open-rack installation as implemented by NPL for timing equipment.

80 Image – Schneider Electric. AR203A, AR8615, AR8652 shown

2018-11-20 Page 92 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Figure 58: Existing timescale installation at NPL with open frame racks. The use of open frame racks in this application, and the associated use of vertical and horizontal cable management, is important due to:  Length and volume of leads in use  Access due to the varied items of equipment  Tracing and maintenance of cables and connections 4.7.2.3 Cable Management To improve reliability and serviceability, cable management (Figure 59) is provided to:  provide localised cable pathway for cables  provide physical protection for the routing of the cables  provide strain relief on cables and enough necessary bending radius  improve the neatness of the cable installation  permit cables to be more easily be installed, removed and replaced Cable management is typically provided locally with in a rack or cabinet. The management provides either a horizontal or vertical path for the cables to be run.

2018-11-20 Page 93 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.7.2.3.1 Examples Typical examples include rings, fingers or trough. They can be made of metal or plastic depending on application and strength required. Locations typically installed are:  Between items of equipment  Incorporated in cabinet as trough / guide at the top or vertically at the sides

Figure 59: Typical 1RU Horizontal Cable Manager81

4.7.2.4 Blanking and Ventilation Panels To manage internal airflow and cooling behaviour inside cabinets and racks, blanking and ventilation panels (Figure 60) are used in the spare rack units (RU). These are manufactured out of either plastic or metal, and are sized in increments of rack unit.

Figure 60: Blanking Panel82

Blanking panels are used to ensure that the cooler air is drawn through the equipment fans correctly. Ventilation panels have a similar appear to blanking panels, but have perforations to enable cooling airflows.

81 Image – Amdex, http://www.amdex.com.au/cable-management.html 82 Image – Cable Away, http://www.cableaway.com.au/cablestore/data-cabinet-shelf-cable-manager-cantilever-shelf- blank-panels/64-blanking-panel-1ru.html

2018-11-20 Page 94 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.7.2.5 Power Distribution LINFRA will provide a Power Distribution Unit (PDU) at the racks (in the CPF, RPF, EOC locations and elsewhere) allocated to the SADT consortium to distribute power to all equipment located within the rack. At a minimum, there will be at least one PDU per rack. Additional PDU will be added where either diverse power supplies are required or capacity of a single PDU is exhausted. The power supply connected to the PDU will either be protected (on UPS), subject to short break (generator backed) or unprotected. This will be dependent on the requirements of the equipment connected. The PDU will have a quantity of standard [AS19] [AS20] IEC 60320 sockets – typically either C13 or C19 – for the connection of equipment. Connections to the PDU are defined in the IICD documents with other SADT sub-elements. Each PDU will have remote monitoring (current and voltage) and remote switching of its outlets. Monitoring of current and voltage will be at a whole of PDU level. Remote switching will be provided at an outlet level and is to enable individual items of equipment attached to the PDU to be power cycled remotely avoiding the needing for physical attendance of a technician. Each PDU will connect to NSDN and interact with NMGR for monitoring (SNMP), software transfer (TFTP) and logging (Syslog). Ideally, the PDU will be configured and operated via SNMP, but may also have a command line (SSH) or web interface (HTTPS). Network addressing (DHCP) will be provided by others as an Enterprise function. PDUs come in either vertical or horizontal form factors. Vertical PDUs are mounted attached to the metal framework of the cabinet/rack. Horizontal PDUs typically consume a RU within the rack.

Figure 62: Horizontal PDU84

Figure 61: Vertical PDU83

The PDU will connect to the electrical supply for that location at a service connection point located in near proximity to the rack. This connection, as a first preference, will be in the form of an [AS21] [AS22] IEC 60309 series plug and matching socket.

83 Image – Servertech, https://www.servertech.com/families/ci67 84 Image – APC, http://www.apc.com/shop/my/en/products/Rack-PDU-Switched-1U-16A-208-230V-8-C13/P-AP7921B

2018-11-20 Page 95 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.7.2.5.1 States and Modes In operation, the PDU may be in one of the following states described in Table 22. These states are a summary for reporting via NMGR to TM.

System State Description

Functional, as designed, no deviation from planned operational conditions ON State The system function is being fulfilled.

Functional, as designed, some deviation from planned operations conditions. The system may require maintenance or intervention by personnel or other actor to Alert State return the system state to “ON State.”

The system function is being fulfilled. Non-functional, by way of critical fault. The fault has affected the operation of the system. This state will require maintenance or intervention by personnel or other Fault State actor to return the system state to either “Alert State” or “ON State.”

The system function is not being fulfilled. Non-functional, either by way of critical fault or intentional downtime. This state will require maintenance or intervention by personnel or other actor to return the system OFF State state to either “Alert State” or “ON State.”

The system function is not being fulfilled. Table 22: Definition of States and Modes

4.7.2.5.2 Conditions and Events The correspondence between system state and component condition is described in Table 23 and Table 24.

Node/Component Description Condition Operating A non-fault condition, in which the system operates as intended. Non-Critical Fault A fault condition, in which the system fulfils the fundamental system functions A fault condition, in which the system fails to fulfil the fundamental system functions. Critical Fault This affects equipment connected downstream of the PDU and may cause faults in that equipment. Table 23: Definition of conditions

Component Condition System State CPF, RPF, EOC, SOC, SPC and Observatory PDU ON State Operating Alert State Non-Critical Fault Fault State Critical Fault OFF State Critical Fault Table 24: Definition of system states

2018-11-20 Page 96 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.7.2.6 Uninterruptable Power Supplies Each RPF will require an Uninterruptable Power Supply (UPS) (Figure 63) unit to provide power to the NSDN switch and other critical SADT equipment. The UPS will have a capacity to provide to maintain the load for a minimum of 24 hours.

Figure 63: Typical rack mounted UPS85

UPS units will also be deployed at the SOC and other observatory locations (for example, power station), with runtimes proportional to the requirements. Each UPS will:  Consume a single phase input power supply compliant with the [RD16] SKA1 Power Quality Standard. At the RPF, this supply will originate from the main switchboard for the building.  Internally be of a double conversion type. The UPS will be installed with both internal and external bypass mechanisms.  Output a single phase power supply compliant with the [RD16] SKA1 Power Quality Standard.  Have a management card network interface. This will enable remote monitoring (input and output power) and control of the UPS.  Connect to NSDN and interact with NMGR for monitoring (SNMP), software transfer (TFTP) and logging (Syslog). Ideally, the UPS will be configured and operated via SNMP, but may also have a web interface (HTTPS). Network addressing (DHCP) will be provided by others as an Enterprise function.  Be compliant with [AS23] AS 62040.1.1 and [AS24] AS 62040.1.2. EMC performance of the UPS will meet the requires of Category C1 of [AS25] AS 62040.2-2008 Uninterruptible power systems (UPS) - Electromagnetic compatibility (EMC) requirements and comply with [AD5] SKA EMI/EMC STANDARDS AND PROCEDURES The UPS shall be specified per [AS26] AS IEC 62040.3-2012 Uninterruptible power systems (UPS)-Method of specifying the performance and test requirements

85 Image – APC, http://www.apc.com/shop/my/en/products/APC-Smart-UPS-SRT-6000VA-RM-230V/P-SRT6KRMXLI

2018-11-20 Page 97 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.7.2.6.1 States and Modes In operation, the UPS may be in one of the following states described in Table 25. These states are a summary for reporting via NMGR to TM.

System State Description

Functional, as designed, no deviation from planned operational conditions ON State The system function is being fulfilled.

The system function is being fulfilled.

Functional, as designed, some deviation from planned operations conditions. The system may require maintenance or intervention by personnel or other actor to return the system state to “ON State.” On Battery State

The system function is being fulfilled, but will enter “Fault State” or “OFF State” at exhaustion of battery capacity.

Non-functional, by way of bypass. The activation of the Internal Bypass has affected the operation of the system. This state will require maintenance or intervention by On Internal Bypass personnel or other actor to return the system state to either “Alert State” or “ON State State.”

The system function is not being fulfilled.

Non-functional, by way of critical fault. The fault has affected the operation of the system. This state will require maintenance or intervention by personnel or other Fault State actor to return the system state to either “Alert State” or “ON State.”

The system function is not being fulfilled. Table 25: UPS Operational States

4.7.2.6.2 Conditions and Events The correspondence between system state and component condition is described in Table 26 and Table 27.

Node/Component Description Condition Operating A non-fault condition, in which the system operates as intended. Non-Critical Fault A fault condition, in which the system fulfills the fundamental system functions A fault condition, in which the system fails to fulfill the fundamental system functions. Critical Fault This affects equipment connected downstream of the PDU and may cause faults in that equipment. Table 26: Definition of conditions

2018-11-20 Page 98 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Component System Condition State RPF UPS ON State Operating Alert State Non-Critical Fault On Battery Non-Critical Fault State On Internal Bypass Critical Fault State Fault State Critical Fault OFF State Critical Fault Table 27: Definition of system states

4.7.2.7 KVM Switch LINFRA will provide KVM (Keyboard, Video, Mouse) switches at the CPF and SPC buildings to enable on-site management of servers from the NSDN, CSP-SDP, SAT.LMC, SAT.CLOCKS and NMGR sub-elements. The KVM Switch (Figure 64) may be one-integrated unit, or be constituted of individual components (switch, console and KVM dongles/cables). An 8 port KVM Switch will typically be deployed at each location.

Figure 64: Typical KVM Switch and Console86

The KVM Switch will connect to the server equipment via industry standard VGA and USB connections, and operate with standard video modes and USB HID profiles. The KVM Switch will use the standard power supplies available in the rack it is housed in. No remote network connection or interface is required. 4.7.2.8 Earthing Each rack will be earthed. Equipment located in the racks may require an earth bond to be provided. This may be for functional or protective use. Functional uses include providing an earth reference or as part of the EMI/EMC shielding system of the attached equipment. The earth will be distributed inside a rack from a common earth bonding point. This bonding point will be affixed to the rack and connect to the wider bonding system available inside the building. In Australia, this bonding point will form part of the Communications Earthing System (CES) compliant with [AS14] AS/CA S009. (Refer to [AS14] AS/CA S009 Figure 3)

86 Image – ATEN, http://www.aten.com/global/en/products/kvm/lcd-kvm-switches/cl5708/

2018-11-20 Page 99 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.7.2.9 Test Equipment The LINFRA sub-element requires test equipment to be available to enable installation, commissioning and operational activities associated with the telescope. This suite of test equipment will consist of the following items:  Optical Time Domain Reflectometer (including Launch Cable)  Optical Power Meter  Light Source  Video Fibre Inspection Microscope  Visual Fault Locator  Connector cleaner  Talkset  PMD / CD tester The test equipment will be used to undertake:  Certification and verification of performance of newly installed or modified cables  Ongoing surveillance of the performance of cables  Inspection and Fault Diagnosis The equipment will be stored when not in use at the CPF building. 4.7.2.10 Cabling Installation Records Installation records generated and kept, in alignment with [AS27] AS3085.1, will record details of the cables and infrastructure installed to enable verification and provide a permanent record of the installation. These records will need to be suitable for import, future reference, and if necessary, modification into software and records systems to be used for the configuration management and operation of the telescope.87 Manual or paper based records are unlikely to meet the operational requirements of the telescope due to the need to be accessed from a variety of locations by different users and long term retention. At a minimum, these systems used to manage the cable records will need to track the utilisation and configuration of cables installed as part of SKA1 by the LINFRA work package. Broad requirements for information to be recorded are:  Multi-site – Ability to model multiple locations (RPF, CPF, EOC, SOC, SPC)  Cables – Can record details about each cable including: o Utilisation o Path (including GIS co-ordinates) o Test results including OTDR Traces o Circuits – Can record circuits formed of either patches or splices o Allocation  Test results including OTDR Traces

87 [RD6] SADT.LINFRA.MDAL-0217

2018-11-20 Page 100 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

 Calculation of Light Budgets  Racks – Can record utilisation of racks down to the RU level  Elevations [AS27] AS/NZS 3085.1 expands on information to be recorded in further detail. Software systems designed to record this information may need to offer and support use cases which include the following:  Security Levels – Limit access by user (for example, read only access) o Use by construction contractor  Work Management o Generation of Work Orders o Design Assistance o Cable Tracing  Different means of access o Web UI o Thick Client o Systems Interface to TM  Reporting o Search  Drawings o Visio export  Technology Agnostic o Can model different types of cables - fibre (single and multimode), copper These requirements will need to be further documented in a detailed software specification, addressing both technical needs and common use cases. This would likely be done during procurement with the selected contractor who will have expertise available.

Analysis and Compliance with User Requirements

To confirm compliance we have checked against our requirements stated in §4.7.1. The items selected are Commercial Off The Shelf (COTS) and comply with both local and international standards.

2018-11-20 Page 101 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.8 Software

Overview of requirements

4.8.1.1 Software for operation of equipment The LINFRA sub-element requires minimal software for its operation. Depending on the specific equipment procured for the PDU (140-062100) and UPS (140-063000) units, proprietary software to operate a “craft” interface on the PDU or UPS, connecting to either the devices serial or network port, may be required for initial configuration and fault diagnosis. 4.8.1.2 Software for Record Keeping The “As Constructed” documentation for the fibre cable infrastructure will produce a range of digital data and documents. These include OTDR traces, light meter readings, drawings, GPS co-ordinate sets and photos. Specialised software may be required to view and interpret some of this documentation (for example, the OTDR traces). Furthermore, this data may require further manipulation before recording and uploading into operational support and record keeping systems associated with the telescope.

Use Cases

Two example use cases are provided below to illustrate a couple of likely scenarios. The use cases are inter-dependent with the capability that will be selected for these tasks. It can be reasonably assumed that the items of software will meet most typical uses for the operation and maintenance of the SKA1 LOW telescope. 4.8.2.1 Use Case 1: Use of craft interfaces for initial configuration of PDU An un-configured PDU is to be installed in an equipment cabinet in the CPF. It will be installed by a technician who is responsible for the installation and commissioning of the PDU. The technician will install the PDU into the equipment cabinet and connect the PDU’s Ethernet management port to nearest NSDN switch before energising the PDU and attempting configuration. The technician may need to connect a laptop to the PDU/network and ensure the PDU “craft” software is loaded onto it. The technician will load the “craft” software and change settings on the PDU such as:  Use of DHCP to set PDU IP address  Address of Syslog server  Configuration of accounts, authentication and privilege levels on the management interface of the PDU  Setting of SNMP version and communities The PDU may have an integral display or indicator lights to indicate the correct configuration. The configuration can be verified through the “craft” software. At the end of the configuration, the PDU should be successfully configured and it should be possible to download a copy of the running configuration for future restoration. 4.8.2.2 Use Case 2: Review of OTDR result in case of fault In response to an alarm raised about a communications failure between an RPF and CPF, a technician is required to investigate and wishes to compare a current OTDR trace on the affected fibre with an earlier OTDR trace taken at the time of installation. There is proprietary software to view these OTDR traces. The technician has been tasked with the investigation by a sequence of interactions starting with NMGR

2018-11-20 Page 102 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

through to TM and manual and automated processes to organise his attendance on site. The technician will need to download the previous OTDR trace and the current OTDR trace onto the same computer. Once loaded onto the computer, the technician can:  Visually compare the OTDR traces  Zoom in and out on specific portions of either trace that are of interest  Take measurements from the OTDR traces  Print out copies and reports in relation to either OTDR trace.  Save the current OTDR trace for future reference These manipulations and comparisons of the OTDR trace will enable the technician to formulate a plan to resolve the underlying fault causing the alarm.

2018-11-20 Page 103 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.9 Implementation

Overview

The implementation of the LINFRA sub-element design will require the efforts of many differing stakeholders to complete the works. For the installation, this work will be undertaken with contractors. Atop this there will be a level of project management and co-ordination, logistics support and engineering supervision. The contractor types include:  Civil  Communications (fibre)  Electrical  Air-conditioning and Refrigeration (HVAC)  Networking These contractors will be required to liaise amongst themselves and with the contractors/staff working on other elements and sub-elements. The contractors (with the exclusion of networking) will be sourced in country (and where needed, locally in Western Australia) due to licensing, registration and capability requirements. Suitably qualified and experienced contractors can be sourced from the telecommunications, civil, construction and building services industries. Some items such as addressing the CSP-SDP connectivity may need to be undertaken by contractors working for others (for example, AARNet). Contracts will need to allow for co-ordination and the provision/set up/removal of support facilities on site during construction. This includes the collection and safe disposal of any waste arising from the construction activities. It is envisaged that several individual contractors will be undertaking the work, under either a prime contractor or operating in a consortia based model. Contractors that previously worked on ASKAP should be invited to tender to take advantage of their familiarity with the project and the site. Further steps to translate from this design to installation and commissioning are discussed further below in §4.9.3 4.9.1.1 Outside Plant Specific considerations for the implementation of the Outside Plant will be the management and mechanism for the installation of the conduit and pits to be installed in conjunction with INAU HV Cable installation along each spiral due to the use of a shared trench. Further investigation should occur post-CDR to investigate the possibility of using a common contractor to undertake this work as a single co-ordinated activity. Specific management of this work interface between the activities to implement each element will need to occur. This is to manage the quality of the conduit and pit installation in advance of the fibre cabling being hauled through it. 4.9.1.2 Internal Plant To arrive at a successful implementation from this design to installation and commissioning on site, the Internal Plant documentation will need to reflect:  Final CPF building design

2018-11-20 Page 104 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

 Final RPF building design  Confirmed internal connectivity requirements between and to other SADT sub-elements  Alignment with the Outside Plant design Contractors working inside the CPF and RPF facilities should be selected for their capability and past project experience working in data centre facilities containing critical infrastructure. Design and cabling best practices shall be followed with special attention to equipment density and cabling. Cabling manufacturer and equipment vendor guidelines shall be followed (improving workability, reduced failure risk and lower maintenance times). An example of a best practice document is provided in the [RD42] “Network Cabling Design Best Practices: 2017” white paper from CTC Technologies Inc & Cisco. 4.9.1.3 Ancilliary Items The Ancilliary Items are multi-sourceable through multiple vendors with installation able to be undertaken by suitably qualified and experienced personnel. Like the Internal Plant fibre, contractors working inside the CPF and RPF facilities should be selected for their capability and past project experience working in data centre facilities containing critical infrastructure. Items selected for use should be checked for their regulatory compliance and suitability for use in Australia. Availability of manufacturer and local agent support should be considered. Existing vendors of cabinets and PDUs from the ASKAP precursor should be specifically investigated so as to take advantage of any commonality and existing emissions qualification of devices on the site.

Configurations

The LINFRA sub-element will need to provide a functional infrastructure in two different configurations; a) An interim configuration to support the AIV Array Assembly AA-1 b) The final configuration of the telescope 4.9.2.1 Configuration for AIV Array Assembly AA-1 For AIV Array Assembly AA-1, as described in [AD6] SKA-TEL-AIV-4410001-SE-RP-MPL Roll-Out Plan for SKA1_LOW, a temporary structure providing “Temporary CPF” services will be placed near RPF S8.88 RPF buildings S8, S9 and S10 will be installed along with their corresponding antenna arrays. This work would need to be co-ordinated with the installation of the SKA1 LOW Fibre for CSP-SDP connectivity. (See §4.5) Dependent on final construction and AIV timelines, this configuration could be enabled by a fibre configuration (Figure 65) involving:  Fibre backhaul and distribution cable to RPF S9  Fibre backhaul and distribution cable to RPF S10  Fibre distribution cable to RPF S8  Fibre backhaul cable to the “Temporary CPF”  Appropriate fibre splicing at fibre joint at RPF S8 (J28) to join these cables together using the additional “spare” cores reserved for this purpose.  Additional fibre splicing as required at RPF S9 and RPF S10 to complete this arrangement

88 [RD6] SADT.LINFRA.MDAL-0218

2018-11-20 Page 105 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

 Internal fibre reticulation and splicing, and the installation of fibre patch panels and trays inside the “Temporary CPF” and RPF S8, RPF S9 and RPF S10 This configuration will enable the future removal of the “Temporary CPF” by disconnection and sealing of the backhaul cable to the building , avoiding disturbance to the fibre joints at RPF S8, RPF S9 and RPF S10.

TUBE-01 001-012 TUBE-01 001-012 TUBE-02 013-024 TUBE-02 013-024 TUBE-03 025-036 TUBE-03 025-036 TUBE-04 037-048 TUBE-04 037-048 TUBE-05 049-060 TUBE-05 049-060 TUBE-06 061-072 TUBE-06 061-072 TUBE-07 073-084 TUBE-07 073-084 J27 192 CORE G.652.D TUBE-08 085-096 TUBE-08 085-096 192 CORE G.652.D TUBE-09 097-108 TUBE-09 097-108 J30 TUBE-10 109-120 TUBE-10 109-120 TUBE-11 121-132 TUBE-11 121-132 TUBE-12 133-144 TUBE-12 133-144 TUBE-13 145-156 TUBE-13 145-156 TUBE-14 157-168 TUBE-14 157-168 TUBE-15 169-180 TUBE-15 169-180

TUBE-16 181-192 TUBE-16 181-192

T T

24 CORE G.652.D 192 CORE G.652.D 192 CORE G.652.D

J29 AA-1 S8 Temporary CPF TOWARDS CPF Figure 65: Potential AIV enabling splice configuration at RPF S889

The feasibility of this configuration is dependent on the timing of the installation of the NREN Interconnect Fibre and Backhaul fibre. Deployment of the main telescope fibre splices and cables shall be able to occur and operate in parallel. It should also be possible to incorporate further RPF locations along the southern spiral in a similar fashion if required. Design for this configuration will be subject to future revision as planning and designs for the AIV and INAU consortia evolve towards their Critical Design Review. 4.9.2.2 Final The Final configuration of the fibre infrastructure will have each RPF having connectivity back to the CPF. This is enabled by:  Installation of the Fibre backhaul cable along each spiral for its complete length  Installation of the Fibre distribution cable associated with each RPF  Appropriate fibre splicing for the connection of each RPF’s Fibre distribution cable to the Fibre backhaul cable associated with its respective spiral  Appropriate fibre splicing along the backhaul haul cable at fibre joint to effect the connectivity from the CPF to each RPF  Internal fibre reticulation and splicing, and the installation of fibre patch panels and trays at the CPF

89 Drawing [D5]

2018-11-20 Page 106 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

and at all RPF buildings Subsequent to the work above, the removal of Temporary CPF will occur and the fibre associated will be withdrawn from the building and made safe in the nearby pits. The exact scope of work will be dependent on the arrangements put in place for AIV. This configuration may be deployed progressively from the array centre (where the CPF is located nearby) outwards. This will occur in co-ordination with the INAU programme for trench and RPF installation.

Execution

To proceed from this design to installation and commissioning further work is required. To progress towards procurement a final documentation set needs to be developed to enable tendering processes to occur. This documentation set will include:  Complete set of drawings  Detailed specification  Schedules  Indicative construction programme To enable this documentation set to be put together a detailed site survey of geological features and cultural heritage clearance needs to occur. This will establish the final locations for the CPF and RPF buildings. The documentation set will first be distributed for review and to check cross element alignment. A tender version of the documentation set will be generated for the purposes of procurement. The indicative construction programme will follow the timing as described in [AD6] SKA-TEL-AIV-4410001- SE-RP-MPL Roll-Out Plan for SKA1_LOW and the [AD7] LOW Construction Plan.90 It is envisaged that the LINFRA work package activity will track that of INAU and be complete early in the plan. A tender process will likely enable the selection of a vendor. Once a contract has been awarded, a “For Construction” set of documentation will be issued enabling supply and installation to commence. It may be the case, that some elements of work are split amongst multiple parties. Where possible and relevant, prior to and during construction, construction activities for the outside plant should follow the procedures and recommendations of the [RD17] Utility Providers Code Of Practice For Western Australia.

Verification

4.9.4.1 Verification for Procurement 4.9.4.1.1 Identification of vendors The LINFRA design consists of large material and labour components. At a minimum, several categories of vendors are required for the successful implementation of the design. These include:  Cable manufacturer for the supply of cable  Suppliers of materials and components

90 [RD6] SADT.LINFRA.MDAL-0219

2018-11-20 Page 107 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

 Civil contractor for installation of conduit and pits, and hauling of cable  Communications contractor for termination and testing of the fibre optic cable, and installation of related items  Vendors and suppliers of racks, cabinets, PDUs, UPS, KVM and ancilliary items The design for the LINFRA sub-element covering the fibre optic network for the SKA1 LOW telescope is straight forward. In Australia, there exists a variety of small and large suppliers and contractors that would be able to accomplish the work. Indeed, the existing ASKAP fibre installation demonstrates that this is possible. An Expressions Of Interest process is recommended to assist with the identification of vendors before proceeding to a tender process. 4.9.4.1.2 Confidentiality In relation to the proposed work, there is minimal proprietary information as the design is composed of standard components. However, final configuration of equipment and infrastructure should be treated as sensitive. Vendors will be required to submit specification sheets for the components that they propose to use, and certification that these will produce a compliant design when used. Vendors may wish to disclose additional information during the procurement process (for example, how they plan to meet and exceed the technical requirements) and a process to maintain confidentiality and respect their intellectual property and trade secrets will likely be need. The procurement process should be constrained by documented procedures to ensure probity, manage conflicts of interest, confidentiality, clarifications, assessment of submissions and formal mechanisms for communicating with vendors. 4.9.4.1.3 Assessment of Vendors In assessing potential vendors, a variety of criteria need to be considered. Generally: a) Certification of compliance with design b) Use of quality management systems such as ISO 9001 and ISO 9002 c) Use of environmental management systems such as ISO 14001 d) Use of safety management systems such as [AS42] ISO 45001 For Suppliers: e) Specification sheets f) Samples g) Independent test results For Contractors: h) Project references i) Staff profiles and qualifications j) Schedules of rates and unit costs k) Sample documentation l) Draft programme of works

2018-11-20 Page 108 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.9.4.2 Functional Verification 4.9.4.2.1 Outside Plant For functional verification, we need to demonstrate conformance with the user requirements and specification. For the SKA1 LOW Outside Plant Fibre, this is a mix of qualitative and quantitative items. Qualitative items reflect the workmanship involved in the installation. Key items to be verified include:  Application and correctness of labelling  Workmanship and neatness of installation  Compliance with standards These are verified by:  Inspection of the installation  Collection and review of photographs of the installation  Collection of records  Submission of appropriate certificates by the installer (such as the Telecommunications Cabling Advice (TCA1)) Quantitative verification of the installed fibre includes:  Measurement of the optical loss using a Light Source and Power Meter (LSPM)  Measurement of the dispersion characteristics  Characterisation of the fibre with an OTDR These tests shall be undertaken per [AS28] AS/NZS ISO/IEC 14763.3:2012 Telecommunications installations - Implementation and operation of customer premises cabling-Testing of optical fibre cabling (ISO/IEC 14763-3:2011, MOD) and will be tested at three separate wavelengths (1310nm, 1550nm and 1625nm) from both ends of the cable. The OTDR traces taken at time of installation will be used for comparison over the life of the cabling system. At installation, they:  provide measurement of the optical length of the cable  verification of the quality and location of optical splices with the channel  show if any damage or stress has occurred to the cable  show poor or contaminated connectors  verify the absence of continuity and polarity mismatches All test data collected above shall be independently verified by an [AS29] AS/NZS ISO/IEC 17020 Type A Inspection Body and the report / statement of compliance shall be provided. All functional verification data should be included with the “As Constructed” documentation and manuals for the installation.

2018-11-20 Page 109 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.9.4.2.2 Inside Plant The functional verification of the SKA1 LOW Inside Plant fibre shall be the same as that for the Outside Plant, except that the OTDR testing may be omitted due to the short cable lengths involved. 4.9.4.2.3 Ancillary Items For functional verification, we need to demonstrate conformance with the user requirements and specification. Like the fibre, this is a mix of qualitative and quantitative items. Qualitative items reflect the workmanship involved in the installation. Key items to be verified include:  Application and correctness of labelling (where required)  Workmanship and neatness of installation  Compliance with standards These are verified by:  Inspection of the installation  Collection and review of photographs of the installation  Collection of records  Submission of appropriate certificates and commissioning sheets by the installer Quantitative verification includes:  Testing and measurement of performance of any fan coil units in the cabinets  Status and test reports from the UPS and PDU installations

2018-11-20 Page 110 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.10 Reliability, Availability and Maintenance The LINFRA sub-element is critical for the operation of the telescope. Inherently fibre networks are highly reliable. Where possible, LINFRA has been designed in a manner to increase reliability and availability.

RAMS Modelling

The reliability and availability for the LINFRA sub-element was analysed in the [AD8] SADT RAM Report (LOW).

Logistics and Long Lead Time Items

Logistics for the LINFRA sub-element was considered in [AD9] SADT ILS Report (LOW). It is expected that all LINFRA components and equipment will be delivered by the vendor or in-country distributor to the EOC or to the respective installation locations if more appropriate.91 There is existing logistics capability to transport and deploy the equipment and components as required. Typical lead time for telecommunications carrier grade equipment from the established equipment suppliers is between 4-8 weeks. This extends to 12-16 weeks for items that are required to be manufactured (eg cable). Some leads times are likely only to be confirmed closer to procurement due to the commercially sensitive nature of these figures.

Maintenance Scheduling

4.10.3.1 Fibre Maintenance procedures should follow the recommendations of [AS30] ITU-T Recommendation L.25. Areas of maintenance activity are summarised in the Table 28 extracted from the recommendation.

ITU-T L.25 Table 1 – General functions of optical fibre cable network maintenance

Maintenance category Maintenance activity Functions Status

Preventative Surveillance (e.g., Detection of fibre loss Optional maintenance periodic testing) increase Detection of fibre Optional (Note 1) deterioration Detection of water Optional penetration

Testing (e.g., fibre Measurement of fibre Optional degradation testing) fault location Measurement of fibre Optional (Note 1) strain distribution

Measurement of water Optional location

Control (e.g., network Fibre identification Optional element control) Fibre transfer Optional (Note 2)

91 [RD6] SADT.LINFRA.MDAL-0220

2018-11-20 Page 111 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Post-fault maintenance Surveillance (e.g., Interface with Required reception of transmission transmission operation system alarm or system customer trouble report) Interface with customer Required service operation

Testing (e.g., fibre fault Fault distinction between Required testing) transmission equipment and fibre network Measurement of fibre Required fault location Confirmation of fibre Optional (Note 3) condition

Control (e.g., cable Restoration/permanent Required repair/removal) repair Fibre identification Required Fibre transfer Required (Note 2) NOTE 1 – Further study is required. NOTE 2 – Fibre transfer may be achieved in a variety of ways, for example: – by use of fibre transfer splicing (optionally synchronous); – by switching the transmission equipment to prior connected standby circuits which may be provided by a ring topology or diverse or duplicated fibre feeds. NOTE 3 – Confirmation after installation is recommended.˙ Table 28: ITU-T L.25 Table 1

Testing required for maintenance activities is undertaken using OTDR and LSPM equipment. Comparison will be made with test results recorded during installation. Active monitoring of fibre deterioration will occur within other SADT sub-elements (NSDN) with equipment optical receive power measurement set to generate alarms through to NMGR and onwards to TM once power levels fall below commissioned thresholds. Preventative maintenance activities may be triggered by planned and un-planned network changes, and in response to external factors such as weather events. Good optical fibre hygiene practices such as connector cleaning and strict use of dust caps will be implemented on the infrastructure. For the associated pit and pipe infrastructure, maintenance activities will be minimal. These will include replacement of pit lids and gaskets (in response to breakages) and visual inspections in response to weather events. In addition, to the standard PPE and tools defined in §4.13.3 specific tools are required for the safe inspection of the infrastructure. These are:  Video microscopes to examine connector cleanliness and avoid direct exposure to laser radiation  Pit (lid) lifters to assist with the manual handling associated with the opening of pits

2018-11-20 Page 112 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.10.3.2 Ancillary Cabinets The cabinets located in the CPF and RPF buildings contain a fan coil unit to extract the heat from the equipment.92 This unit will require regular inspection and changing of air filters. This shall be done in accordance with manufacturer’s recommendations and relevant standards. Power Distribution Units The PDU in use at all locations may require “In-service safety inspection and testing of electrical equipment” (PAT testing) at regular intervals. The schedule for this will be in line with standards and localised risk assessments. This will likely need to occur during telescope maintenance periods due to the impact associated with the activity on equipment in use during observations. Test Equipment Some items of Test Equipment (such as OTDR units) have need for regular calibration to provide confidence and traceability to the measurements taken. This shall be done in accordance with manufacturer’s recommendations and to relevant standards.

Line Replaceable Units and Spares

The sparing strategy for the LINFRA sub-element needs to consider the following:  Uniqueness of each item and level of commercial availability  Logistics and shipping  Manufacturing lead times  Impact to telescope availability and associated costs  Cost of the spare  Shelf life of the spare and associated testing required before use  Size and associated storage requirements of the spare  Associated labour  Depth of spare inventory  Likelihood of replacement required  Sharing of spares Specifically, for the MRO site, the long logistic chain needs to be considered. For example, transport of parcels via road from the east coast (Sydney, Melbourne, Brisbane) to Geraldton typically takes 5 to 6 days. Given that there is limited warehousing by vendors in Western Australia, we will need to flag items with significant sparing impacts. Most LINFRA sub-element products will be used for the length of their operational life, failure, or obsolescence (whichever occurs first).

92 [RD6] SADT.LINFRA.MDAL-0283 Cabinets at EOC, SPC and SOC will not have integrated cooling.

2018-11-20 Page 113 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

There are four areas of sparing that need to be considered:  Outside Cable Plant  Inside Cable Plant  Cabinet and Racks  Power Distribution Units, Uninterruptible Power Supplies and KVM Switches Aside from the specific notes below, it is recommended that further investigation occur with the successful vendors regards inventory management in the Australian market. (That is, further information be sought on the general depth of stocks kept by the suppliers and be used to inform the sparing strategy) Outside Cable Plant Sparing for the Outside Cable Plant will target the capability to repair damage to the Backhaul Fibre Cable (140-021200) and Distribution Fibre Cable (140-021100) in a timely fashion. Stocks of cable and adequate quantities of associated hardware (splices, joints, pits, and conduit) will be required to be kept either on site, or at the EOC. The length of cable kept will be proportional to distance between access points (pits) on the installed cable and the expected number of faults anticipated. Due to the long lead times associated with the manufacture of the Backhaul Fibre Cable, it is recommended that sufficient stock of cable to handle multiple incidents be kept. Quantities of the associated hardware to be kept are proportional to the number of incidents. Inside Cable Plant The Inside Cable Plant encompasses all LINFRA sub-element components of the fibre plant inside the CPF, RPF, SPC, EOC and SOC buildings and includes permanently installed fibre, ODFs, fibre trays/patch panels, patch leads and cable pathways. Spares will be required but minimal in quantity. Quantity to be kept will cater for replacement due to defect or minor damage. Cabinet and Racks The cabinets and racks are largely a passive component. The cabinets used at the CPF (140-041000) and RPF (140-042000) will have a fan coil unit fitted for the removal of excess heat. This fan coil unit will require regular maintenance and spares (for example, air filters). Maintenance of the fan coil unit should be contracted out to a specialist service provider alongside a SLA (driven by telescope availability). This will in turn drive the service providers sparing requirements. Power Distribution Units, Uninterruptible Power Supplies and KVM Switches Power Distribution Units will be located inside cabinets and racks at the CPF, RPF, SPC, EOC and SOC buildings. These will either be minimally spared or placed under a maintenance arrangement with the vendor. Uninterruptible Power Supplies will require regular maintenance and spares (for example, battery replacement). In addition, periodic testing of the UPS under load should be undertaken. Maintenance of the UPS units will be contracted out to a specialist service provider alongside a SLA (driven by telescope availability). This will in turn drive the service providers sparing requirements. The KVM Switches require no specific maintenance aside from occasional cleaning of their screen and keyboard. The availability of the KVM Switch is not critical for telescope operation, however it is recommended that they be placed under a maintenance arrangement with the vendor.

2018-11-20 Page 114 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.11 Security The following is an extract from Revision 10 of the SKA requirements regarding security, it is followed by a full list of security requirements (Table 29) that were considered whilst developing the LINFRA design. Extract from Revision 10:

The SKA will be a very attractive target for criminals, including theft of infrastructure and cyber- attacks exploiting the HPC and networks. It will also be seen to be a 'soft' target with connections to the academic and research communities. The potential impacts include financial cost to replace equipment and to restore systems, loss of observing opportunities (telescopes could be rendered useless for weeks or months) and loss of reputation for the SKA and the host nations. The threats will exist from the outset and security will need to be established before physical installation starts (including security of information systems to deter Trojan horses from being installed early in the development phase).

There is currently no ISO Standard for a Security Management System, although DPC: 13 / 30278101 DC included Draft BS ISO 34001 Security Management System which forms the basis of the Security requirements. In addition, the UK Cabinet Office HMG Security Policy Framework (Version 11.0) has been used to derive requirements.

Requirement Description SKA1-SYS_REQ-2791 The SKA shall provide a security management system that includes : i. personnel security ii. physical security (asset) iii. security of information SKA1-SYS_REQ-2478 The observatory shall provide a secure environment for equipment including protection of generators, fuel, solar cells, equipment spare stores, and inter- station assets such as copper cables. SKA1-SYS_REQ-2482 It shall be possible to control on a per user basis which SKA1 facilities and resources (both hardware and software) may be accessed by the user. SKA1-SYS_REQ-2479 The SKA1_Low, and SKA1_Mid shall provide a secure environment for all its data archives. Table 29: SKAO Revision 10 Security Requirements

At the time of writing there is no SKA wide security policy document available. The only security requirements are stated above (Table 29). In the absence of a security policy a series of security risk assessments ([RD18], [RD19], [RD20], [RD21], [RD22], and [RD23]) was performed for all of the LINFRA locations in the array. Within each location all of the LINFRA assets were assessed against a number of threats and given a vulnerability score against confidentiality, integrity, and availability commonly known as CIA. Each threat was initially assessed with no mitigation in place providing a RAG status for all of the threats.

System Boundaries

The LINFRA sub-element has boundaries with the other SADT sub-elements (NSDN, SAT.CLOCKS, SAT.UTC, SAT.FRQ, NMGR, SAT.LMC, and CSP-SDP) and the INAU and LFAA elements. The physical boundaries of the LINFRA sub-element are the installed fibre cable, racks and power distribution equipment located in racks in the secure locations specified in Table 30, which also provides information on the element responsible for that physical security.

2018-11-20 Page 115 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Location Details of the environment Responsibility

CPF Racks are located in an RFI INAU shielded building in a locked compound. RPF Racks are located in an RFI INAU shielded building. Outside Plant Fibre installed in conduits and pits LINFRA & INAU EOC Racks are located in dedicated INAU room in a building SPC Racks are located in dedicated INAU room in a building SOC Racks are located in dedicated INAU room in a building Table 30: The physical boundaries of the LINFRA equipment and infrastructure

Logical boundaries for the LINFRA sub-element exist with the NSDN and NMGR sub-elements only for the UPS and PDU provided. Responsibility for the logical security of these is detailed in Table 31 below. Location Details of the environment Responsibility

CPF Connection of PDU to network NSDN (network) RPF Connection of PDU and UPS to NSDN (network) network EOC Connection of PDU to network NSDN (network) SOC Connection of PDU to network NSDN (network) SPC Connection of PDU to network NSDN (network) Table 31: The logical boundaries of the LINFRA equipment

Foreseeable Threats and Mitigations

The next phase was to include any mitigation that would be performed as part of the infrastructure design or rollout. As there is no security policy in place, and therefore no existing mitigating controls defined, the residual risk remained quite high. The table below (Table 32) identifies the risk and provides suggestions for mitigation: Risk Suggested Mitigation Failure / Fluctuation of Provide second PDU to minimize impact on dual supply equipment Power Take advantage of diverse supplies where present Hardware Failure Maintain onsite spares inventory Monitor software and equipment status. Software Failure / Only install official software updates and patches. Malfunction Proactively monitor software versions installed. Industrial Action / Staff Maintain skills inventory Shortage Keep site documentation up to date

Loss of Heating / Ventilation / Air Provide monitoring and alarms for key environmental parameters Conditioning / Humidity/ Utilities Damage to / Failure of For internal plant, provide cable management Communications Lines / For outside plant, survey and provide maps of infrastructure. Install cables marker posts to show location. Participate in asset location and

2018-11-20 Page 116 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Risk Suggested Mitigation protection schemes (Dial Before You Dig) Willful Damage Restrict physical access to the equipment and cables.

Natural Disaster (Flood, Earthquake, Volcanic Design building to appropriate standard per local building codes Activity, etc)

Provide fire detection Fire Provide fire extinguishers Damage by Wildlife Incorporate anti-wildlife and anti-pest features during installation. Undertake poison and baiting to control Theft / Loss Restrict physical access to the equipment and cables. Unauthorised Physical Restrict physical access to the equipment and cables. Access / Tampering Misuse of resources Order cable products labeled with SKA identification Require AAA to access configuration information of equipment. Restrict network access (firewalls, access control lists, network Denial of Service (DoS) segmentation). Install software updates and patches. Proactively monitor software versions installed. Keep up to date with software updates and releases. Illegal Import / export / Only install official software, updates and patches. use of software Proactively monitor software licenses and versions installed. Only install official software, updates and patches. Exploitation of Software Proactively monitor software licenses and versions installed. Vulnerabilities Keep up to date with software updates and releases. Spoofing / Impersonation Verify and validate staff credentials. Only allow trained personnel to work with LINFRA equipment and infrastructure. Maintenance Error / Use scripts. Operator Error / Follow a change control procedure. Misconfiguration Take backups of the configuration prior to making changes. Test the changes once implemented. Malware protection. Install software updates and patches. Malicious Software Implement an Intrusion Prevention System to identify and stop unusual activity. Install software updates and patches. Implement an Intrusion Prevention System to identify and stop Exploitation of unusual activity. Vulnerabilities Proactively monitor software versions installed. Keep up to date with software updates and releases. Disable unused ports. Communications Restrict network access (firewalls, access control lists, network Monitoring / segmentation). Eavesdropping Encryption of sensitive information (SNMP v3)

2018-11-20 Page 117 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Risk Suggested Mitigation Require AAA to access configuration information of equipment. Restrict network access (firewalls, access control lists, network Exfiltration of Information segmentation). / Information Disclosure Restrict physical access to the equipment. Prohibit the use of removable storage media. Monitor outbound internet and email traffic. Require AAA to access configuration information of equipment. Restrict network access (firewalls, access control lists, network Communications segmentation). Infiltration Restrict physical access to the equipment. Prohibit the use of removable storage media. Monitor the network for unusual activity. Require AAA to access configuration information. Restrict network access (firewall, access control lists, network segmentation). Unauthorised User Access Restrict physical access to the equipment. Generate alerts for failed logon attempts. Generate alerts for configuration changes. Require AAA to access configuration information. Restrict network access (firewall, access control lists, network Unauthorised Privileged segmentation). Access / Elevation of Restrict physical access to the equipment. Privileges Generate alerts for failed logon attempts. Generate alerts for configuration changes. Implement an asset disposal policy. Wipe the device configuration. Unsecure Disposal of Asset Remove and destroy the storage media if possible. Reset the device back to factory settings. Unauthorised use of Prohibit the use of removable storage media or limit its use to removable storage media trusted personnel only. Implement a removable storage policy. Repudiation of services / Store logs offline and restrict access to them. transaction messages Table 32: Suggested Mitigation

If these mitigating controls are included in the security policy and implemented, the identified risks are likely to be reduced to an acceptable level.

Recommendations and Compliance

The following section details each requirement stated above (Table 29) and documents how the LINFRA design complies with the requirement where applicable. Additionally, several general recommendations are made to improve the security and integrity of the infrastructure.

SKA1-SYS_REQ-2791 The SKA shall provide a security management system that includes: i. personnel security ii. physical security (asset) iii. security of information

2018-11-20 Page 118 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

For the LINFRA design we are concerned with the physical security of the asset, and to a lesser extent, the security of information within this requirement. The security of personnel (aside from consideration of Health and Safety) is outside of our responsibility. With the physical assets that form the LINFRA sub-element, security needs to be considered for:  Outside Plant (Cable and Pathways)  Inside Plant (Cable Terminations)  Racks and Cabinets  Power Distribution (PDU and UPS)  KVM Switches The primary security concern for these assets is the integrity of the asset and the assets proper function. This includes protection against inadvertent and wilful damage. Controls include good housekeeping, asset protection and integrity checks. Good housekeeping would include:  the proper management of waste, dirt and debris that could affect the assets  management systems around the assets (works notifications, planned events, and permitting)  correct storage of spares Asset protection addresses:  Correct records of assets  Provision of information about assets to those working around them  Safe systems of work Integrity checks include:  Use of locks and alarms  Protective barriers  Auditable records  Seals and passive surveillance Information in the LINFRA context is defined as the data that is transported over the LINFRA infrastructure to and from the elements/sub-elements, and any configuration data held on the LINFRA devices. Science data transported over the network is considered as uninteresting to a cyber-attacker as the data has no value to anybody other than a scientist interested in the observation. This data is therefore ignored as part of this analysis. Management data/access to and from a LINFRA network element is something we need to protect from a cyber-attack as it could allow an actor to infiltrate the device if it were compromised. Table 33 details a number of security features we expect all vendors to support and have enabled on any device that is used by LINFRA. Collectively these will provide security of information stored and transported.

2018-11-20 Page 119 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

 Disable unused ports on Network element  Network element(s) support Anti-spoofing features  Network element(s) logs number of failed logon attempts  Network element(s) default SNMP community names be deleted and new ones created  Network element(s) supports NTP  Network element(s) telnet and HTTP protocols can be disabled  Network element(s) HTTPS supports TLS v1.2 or higher  Network element(s) SSL be disabled and TLS be forced when using HTTPS  Network element(s) support AAA servers  Disable Network element(s) unused services  Network element(s) inactivity timeout for logged on users  Network Element supports FTPS  Disable Network Element FTP/TFTP  Network element software updates release frequency  Network element security patches release frequency  Network Element(s) support SNMPv3  Network Element(s) support HTTPS web interface  Network Element(s) support SSH login  Network Element(s) support multiple levels of user access  Network Element(s) monitors config changes on the device via SNMP or syslog  Network Element(s) local and remote syslog capable Table 33: Security features to be supported and enabled

Configuration data stored in an item of LINFRA equipment must also be protected as it may contain sensitive information. All configuration files will be backed up to a server in the NMGR domain so that any file can be re-installed should an issue occur on the device. Any passwords within the configuration file must not be stored in clear text. In order to maintain the integrity and security of the equipment and its data, it is recommended that LINFRA equipment is only administered by trained and experienced personnel. A formal change control procedure should also be followed for all hardware and software configuration changes so that any security risks can be identified prior to implementation.

2018-11-20 Page 120 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

SKA1-SYS_REQ-2478 The observatory shall provide a secure environment for equipment including protection of generators, fuel, solar cells, equipment spare stores, and inter-station assets such as copper cables.

This requirement is not considered to be a LINFRA responsibility but is noted here for completeness. LINFRA equipment such as racks/cabinets, PDU and UPS will be located within the secure areas of the CPF and RPF. Outside plant equipment is subject to overall site security. LINFRA spares and desirable items such as test equipment will be kept in secure stores and compounds provided by other elements.

SKA1-SYS_REQ-2482 It shall be possible to control on a per user basis which SKA1 facilities and resources (both hardware and software) may be accessed by the user.

It is expected that AAA servers will be used across the observatory to control access to all devices within the telescope. This includes the PDU and UPS installed as part of the LINFRA design. An integral part of the AAA design is to allow and log access for multiple users. Each user may have differing privileges depending on their role. The principle of least privilege will apply. Table 34 is a suggested set of rules that should apply for authenticated and authorised access; these are subject to change and currently non-binding.

2018-11-20 Page 121 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

 All users must complete a security induction prior to being granted access to any SKA Observatory networks.  All users must sign a network usage agreement prior to being granted access to any SKA Observatory networks.  Users must only have access to systems, services, and infrastructure required to undertake their duties using least privilege.  Privileged accounts must only be used for specific tasks and not used for general day to day access to the networks. Users with privileged accounts must use a non-administrative account for general day to day activities.  All privileged accounts must be unique for each user. The use of shared privileged accounts is prohibited, unless the device / software does not have the capability of individual accounts.  In the event that shared privileged accounts must be used, the password must be changed immediately upon the termination of employment or contract of one of the shared users.  A privileged user account form must be completed for every privileged user account created.  Service accounts must not have full privileged access and they must be restricted so that they can only log on to the required devices. A service account requisition form must be completed for every service account created.  Default usernames and passwords must not be used.  Usernames and passwords must not be hardcoded into software.  Usernames and passwords must not be stored in clear text.  Users must be forced to change their passwords every X months.  The password policy must require the use of complex passwords, with a minimum length of X characters, and automatically lock out after X invalid logon attempts.  User accounts must be disabled immediately upon termination of employment or contract. Table 34: Network Access Rules

SKA1-SYS_REQ-2479 The SKA1_Low, and SKA1_Mid shall provide a secure environment for all its data archives.

This requirement is not allocated to SADT / LINFRA but is noted here for completeness, this function being undertaken by NMGR. Thought must be given however for any configuration archives of the LINFRA equipment that needs to be stored in a secure manner. In addition to the above reference should also be made to the [RD24] Standards for SKA Networks which details some security features that should also be considered for any network within the SKA1 LOW telescope.

2018-11-20 Page 122 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

General Recommendations

The following general recommendations are made for the organisation that will be operating the infrastructure:  Seek and maintain membership of AusCERT, and subscribe to relevant member services including security bulletins, early warning service, and advice and incident management services.  Seek and maintain membership of Dial Before You Dig WA Ltd. DBYD (WA) provides a national referral service for underground asset owners to ensure plans are provided to those wishing to undertake excavation and that those assets are protected. In the case of this design, this encompasses the outside plant fibre.

2018-11-20 Page 123 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.12 Safety

General

Unique amongst the SADT work packages, key LINFRA products involve civil and building services construction work. This design must consider, through appropriate hazard identification, risk management and controls, the safety of the:  Products and material used  Construction and installation activities  Commissioning and operational use  Maintenance activities The life cycle, obsolescence and decommissioning of the system must also be considered. To ensure compliance with all necessary Legal, Regulatory, Environmental, Cultural requirements and Safety Standards, we must consider the following to ensure “Safety In Design”93,94:  Control of workplace and those with duty of care  Lifecycle and use of product  Application of systematic risk management  Application of Safe Design Knowledge and Capability  Information transfer regards hazards and risks

Safety In Design Principle 1: Persons with Control The SKA1 LOW LINFRA design will be realised at:  MRO (CPF, RPF, fibre and conduit network)  Geraldton (EOC)  Perth (SOC, SPC) The design, construction and management of the SKA1 Telescope are under the auspices of the SKA Organisation. At present, the Murchison Radio Observatory (MRO) at Boolardy and the EOC facility at Geraldton are operated by CSIRO. At the time of writing, the location and operation of the SPC and SOC facilities in Perth are yet to be determined. There may also be aspects of the assembly and construction that may occur elsewhere. These instances will need to be reviewed individually. The [AD10] SKA Project Safety Management Plan should cover these arrangements in detail.

93 See [RD33] Guidance on the principles of safe design for work 94 See [RD34] Model Code of Practice - Safe Design of Structures

2018-11-20 Page 124 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Safety In Design Principle 2: Product Lifecycle Safety risks are to be considered and analysed over the lifecycle of the telescope and the site. These break down into several, sometimes overlapping, periods of activity and use. These include as applying to LINFRA:  Construction and installation  Commissioning  Use  Decommissioning And given the nature of the LINFRA assets, we must also consider these periods:  Life of infrastructure  Life of telescope  Duration of operations at the site For example, it is likely that any pit and conduit infrastructure installed will be left in the ground for potential future use after decommissioning (removal of cables).

Safety In Design Principle 3: Systematic Risk Management Risks highlighted in this design should be managed as part of a broader system for the ongoing identification and management of risks associated with the LINFRA products, and the telescope more generally. A risk management system, based on [AS31] AS/NZS ISO 31000:2009 Risk Management – Principles and guidelines, should be used to capture and mitigate risks. Specific Occupational Health and Safety Risks should be managed in accordance with local legislative requirements. Systems of management such as [AS32] AS/NZS 4804:2001 and audit such as [AS33] AS/NZS 4801:2001 should be used. Specific Environmental Risks should be managed in accordance with local legislative requirements. Systems of management such as [AS34] AS/NZS ISO 14001:2016 should be used.

Safety In Design Principle 4: Safe Design Knowledge and Capability The telescope and the SADT elements are subject to a design review process. A Failure Mode, Effects and Criticality Analysis (FMECA) and Reliability, Availability, Maintainability, and Safety (RAMS) analysis have been undertaken against this design. (See [AD8], [AD11] and [AD9])

Safety In Design Principle 5: Information Transfer A risk management process should be adopted and a risk register should be kept both during the construction and ongoing operation of this infrastructure. Major hazardous items and scenarios are highlighted later in this design. The output of the FMECA and RAMS studies will be an input for the creation of an on-going risk register for this infrastructure. During construction, standard best practice for the management of safety and risk should occur. This includes:

2018-11-20 Page 125 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

 Creation and use of Health, Safety and Environment (HSE) manual for the works and the site  Induction for personnel, equipment and machinery  Proactive use of Job Safety Analysis (JSA) and Safe Work Method Statements (SWMS)  Application of the Hierarchy of Controls to all hazards and risks  Engagement of appropriately qualified and licensed contractors  Regular site meetings and daily toolbox sessions CSIRO have an extensive, existing [RD25] Murchison Radio-astronomy Observatory HSE AND SITE INFORMATION FOR CONTRACTORS manual that should be used as a primary reference.

Hazardous Items and Scenarios

All items required for the LINFRA will need to be assessed regards the hazards and risk they possess either inherently or through use. This is managed by the [AD12] SKA Product Assurance & Safety Plan which includes processes to:  Monitor the labelling, compliance and conformance of products supplied  Submission of Material Safety Data Sheets (MSDS) for these products  Undertake risk assessments against each product Hazardous items may affect multiple parties including:  Construction and installation personnel  Operations and maintenance personnel  Visitors to the telescope facilities The hazard associated with the item may:  Be inherent  Exist during transportation  Be present during installation and afterwards as a result of incorrect installation  Be associated with its use  Due to age, damage and deterioration  Occur when out of service  Exist during decommissioning and removal Some example risks and their mitigations are provided in Table 35.

2018-11-20 Page 126 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Area of risk Affected products Scenario and mitigation

Electrical shock PDU Worker receives an electric shock from the PDU or UPS UPS Mitigations:  Prohibitions against “live” work  Test to [AS35] AS/NZS 3760  Ensure RCD devices are present on all supplies and are testing regularly  Ensure adequate training of personnel.

Injury through exposure Fibre Worker receives fibre splinter under skin to sharps Mitigations:  Protective gloves  Proper disposal bins provided  Ensure adequate training

Injury through exposure to Fibre Worker is blinded by exposure to laser radiation laser radiation Fibre patch panels Mitigations  Avoid work on “live” fibre  Use cameras rather than naked eye to view fibre  Ensure caps are placed over unused fibre connectors  Use equipment with automatic optical shutdown  Attach laser warning notices  Ensure adequate training

Crushing injury Racks Worker receives injury by item falling upon them Cabinets Mitigations: Pits (During  Move with proper procedure installation)  Use appropriate tools to assist moving the item Fibre (On spool)  Use PPE  Ensure adequate training

Manual handling injury Racks Worker receives injury by moving item incorrectly Cabinets Mitigations: Pits (During  Move with proper procedure installation)  Use appropriate tools to assist moving the item Fibre (On spool)  Use PPE  Ensure adequate training

2018-11-20 Page 127 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Area of risk Affected products Scenario and mitigation

Trips and tangles Fibre Worker trips over cable Patch leads Mitigations: Power leads  Order cables of appropriate length  Provide appropriate cable management and pathways  Practice good workplace hygiene  Ensure adequate training

Exposure and sunburn Fibre Worker feels ill due to over exposure to sun Pit Mitigations Conduit  Limit working periods Trench  Supply PPE  Provide adequate shade  Provide adequate hydration  Monitor conditions  Ensure adequate trainining Table 35: Example risks, products and mitigations

2018-11-20 Page 128 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.13 Maintenance

Summary

All LINFRA products are subject to maintenance. For hardware items, this entails repair, adjustment or replacement as needed. For items with either embedded or associated software, (for example, PDU or UPS), this may include software upgrades and security patches. Maintenance activities will consist of both preventative and corrective actions. The model of maintenance applied to each product will fall into four categories:  Actively maintained (preventative and corrective)  Condition monitored (corrective if required)  Activity triggered (occurs as a result of other activity)  Run Until Failure (corrective in response to fault) For each product, this will include a means of:  Failure Detection (Conditional and Periodic)  Fault isolation (Automatic and manual)  Corrective action (Automatic and manual)  Operational verification For products under a maintenance contract with the vendor, the typical annual maintenance cost is approximately 10% of the list cost. This maintenance includes access to technical support for configuration and troubleshooting assistance, the replacement of faulty components and access to a software upgrade repository. The task of carrying out the maintenance will be a SKA responsibility. Logs of maintenance activities shall be kept, and where needed, telescope documentation and records shall be updated to reflect the current working system. Maintenance Model Products Actively maintained Racks and Cabinets (Fan Coil Unit) Special Test Equipment (Periodic Calibration) Uninterruptible Power Supplies Condition monitored Power Distribution Units and Power Cables (Electrical Safety Testing) Fibre Activity triggered Pits and Duct (Cleaning as required) Fibre Patch Panel and Fibre Patch Lead (Connector cleaning) Run Until Failure Racks and Cabinets (other parts)

ALL OTHER PRODUCTS Table 36: Maintenance models

The operational cost model for SADT is discussed in more general terms in the document [RD26] SKA-TEL- SADT-0000423 Operational Cost Model Description.

Maintenance Schedule

Un-scheduled maintenance activities for LINFRA will be carried out as required to attend to, resolve and clear faults. These may include items that have been run until failure or resolution of faults caused by external factors (eg cable cut). The timeframe for these activities will be driven by the nature of the fault, sparing, impact on telescope availability, staffing and site conditions.

2018-11-20 Page 129 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Schedule maintenance activities (for items that are actively maintained, condition monitored, or activity triggered) will predominantly occur during telescope maintenance periods. Specific scheduling of each activity will consider impacts on other systems. For example, an impact to NSDN will affect their capability to provide critical network services (e.g. telephony) during the maintenance scheduling of the telescope. The scheduling of these activities should therefore be done to ensure it does not affect other telescope maintenance activities, or limited to locations where other maintenance activities are not taking place. Some activities may be schedule to occur at the same time as other testing. For example, regular RCD circuit breaker testing affecting PDU circuits. Specific maintenance schedules are driven around regulatory and standards requirements.  Racks and Cabinets (Fan Coil Units) o [AS36] AS/NZS 3666.2:2002 Air-handling and water systems of buildings - Microbial control Operation and maintenance  Power Distribution Units, Power Leads, and Uninterruptible Power Supplies o [AS35] AS/NZS 3760:2010 In-service safety inspection and testing of electrical equipment For these activities records of such maintenance should be kept. For products with embedded software, upgrades, as per industry best practises, are scheduled annually. However, intermediate software patches may be released by the vendor to address security or functional bugs. These will be evaluated on a case by case basis to determine if the upgrade is required to address security or functional requirements. It may be the case that the installation of a software patch can be delayed until the next annual software upgrade where the patch will be automatically be included in the software release. Software upgrades are typically performed remotely, however it is always advised to have an on-site technician available in case the software upgrade has failed and there is a requirement to recover a device using console access. It is expected that during commissioning, the initial maintenance schedule will be higher. For example, air filter cleaning would occur more frequently due to activity in the surrounding environment.

Staffing, Training and Safety Equipment Required

The installation and maintenance of the LINFRA products will be undertaken by suitably qualified and experienced personnel. The installation and maintenance activities require a mix of skills and staffing, and will be required to be undertaken on-site. The required skillsets are:  Network Technician  Communications Cabler  Electrician  Air-conditioning and Refrigeration Technician (HVAC) It is not anticipated that the number of personnel or contracting arrangements will need to increase over time. Skillsets Following experience on the precursor telescopes, the Network Technician will be part of a small infrastructure group who look after server and network infrastructure including installation, responding to

2018-11-20 Page 130 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

security alerts, proactive monitoring and maintenance. The staff for this position will overlap with those provided for NSDN. The staff will travel to site to perform the activities as required, however on-site technicians should be available to perform emergency tasks (with the assistance of remote specialists), if required. A Network Technician will:  provide the liaison between staff and systems maintaining other SADT sub-elements and other telescope components with the other skillsets  assist with fault diagnosis and resolution A Network Technician will have tertiary, industry or vocational qualifications in IT and Networking. In Australia, the other skillsets require staff with formal trade qualifications, registrations and licenses. The qualifications consist of areas of study undertaken with Registered Training Providers under the Australian Qualifications Framework. It is anticipated that work involving these skillsets will be undertaken on a contracted basis by specialist firms. A Communications Cabler will hold an Open Cabling Registration with endorsements for structured, optical fibre, co-axial and underground cabling. The registration enables telecommunications workers to legally install and maintain telephone, security and fire alarm cabling in all types of customer premises under the Telecommunications Act 1997 (Cth). A Communications Cabler will:  undertaken all installation and maintenance work on the fixed portions of the inside and outside communications cabling plant A Electrician will be a licensed person under the Electricity (Licensing) Regulations 1991 (WA) in Western Australia. A Electrician will:  be the competent person to perform the [AS35] AS/NZS 3760:2010 inspection and testing An Air-conditioning and Refrigeration Technician (HVAC) will hold the necessary trade diploma or certificate required to maintain the fan coil unit in the cabinets at the CPF and RPF. They may also need to hold the appropriate Australian Refrigerant Council Refrigerant handling license. An Air-conditioning and Refrigeration Technician (HVAC) will:  undertake the required the installation, maintenance and repair work on the fan coil unit in the cabinets at the CPF and RPF Training The training and continued development of all personnel (staff and contracted) working on the SKA infrastructure should be considered before construction. Training should be continuous through the lifecycle of the telescope and tracked on a regular basis. Certifications and professional development should also be considered for all staff supporting the SKA networks. This approach ensures that personnel assigned to work on the SKA programme have the appropriate experience and skills to perform their assigned tasks. This is achieved via the processes shown in Table 37.

2018-11-20 Page 131 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Training Description Training Enablement Induction Training plan will need to be In-house developed for all new hires supporting the SKA infrastructure. Induction Induction process for new In-house contractors working on the infrastructure Induction Produce Manual for In-house Contractors Health and Safety SKA specific Health and Safety In-house or external. It may be training: that the employee training may need to be done by an o Employee external company whereas a o Contractor visitor to the site may need a o Visitor briefing performed by on-site personnel/an employee of SKA. Ongoing personal All staff should have a training In-house development plan for personal development in line with SKA policies Product Specific Training Vendor specific product Vendor or vendor partner training on the network products to be deployed. New product training on an as needs basis. Vehicles and equipment Formal training on the vehicles Training with industry partner and equipment that will be used by staff and contractors on the site, with reference to site specific hazards Certifications Formal training for staff in the Technical training with following areas: industry partner o Technical Support training for example o Support Management ITIL based training – Industry o Project Management Project Management training – Prince or PMP - Industry Qualifications Formal training to: Technical training with industry partner o augment existing qualifications of staff as required o maintain qualifications/endorsemen ts of staff Table 37: Training Requirements

2018-11-20 Page 132 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

The exact amount and mechanisms of training required will depend on the support model selected for the SKA1 LOW infrastructure. Safety Equipment Safety equipment appropriate to the tasks undertaken during maintenance should be provided. Appropriate training shall be given in the use of all safety equipment. At a minimum, it should include basic Personal Protective Equipment (PPE) such as:  Safety Boots  Safety Glasses  Gloves  Appropriate high visibility clothing

2018-11-20 Page 133 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.14 Integration The LINFRA sub-element consists of multiple work streams that need to be successful assembled to provide infrastructure to other SADT sub-elements and in turn other telescope elements. The LINFRA sub-element splits into two major areas of activity:  construction work to assemble and install fibre infrastructure needed other SADT sub-elements, and  installation of ancillary infrastructure products to other SADT sub-elements. Each of these areas of activity translates to work streams in the installation.

Component Integration

For the LINFRA sub-element, there are two key areas of component integration. These are shown in Figure 66 and Figure 67, and are: a) The fibre cable with the respective pathway. For example, the Backhaul fibre with the conduit, pits and trench. b) The fibre cable with the mechanisms for terminating or joining the cable. For example, the connection of the cable through the waveguides and splicing inside the CPF and RPF buildings. The remainder of the LINFRA products are Commercial Off The Shelf (COTS) components and do not require any significant integration. For example, the CPF PDU installs in a straight forward manner into the CPF Rack by attaching to pre-existing fixing points. For reference the LINFRA Product Breakdown Structure (PBS) [AD4] defines the LINFRA products down to a Line Replaceable Unit (LRU).

Fibre_Cable 140-021000

Pathway 140-030000

Conduit Cable_Tray Trunking 140-033000 140-034000 140-035000

Pit_Man-Hole_Hand- Trench Racks Hole 140-031000 140-040000 140-037000

Marker_Tape 140-032100

Figure 66: LINFRA Fibre to Pathway integrations

2018-11-20 Page 134 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Fibre_Cable 140-021000

Fibre_Tray Fibre_Joint Waveguide Optical_Distribution_Frame_(ODF) 140-023300 140-023500 140-036000 140-023600

Fibre_Patch_Pan Fibre_Patch_Pan Splice_Tray Fibre_Pig_Tail Splice_Tray Enclosure Splice_Tray Splice_Tray Fibre_Pig_Tail Fibre_Tray el el 140-023100 140-022200 140-023100 140-038000 140-023100 140-023100 140-022200 140-023300 140-023400 140-023400

Fibre_Splice Fibre_Splice Fibre_Splice Fibre_Splice 140-023200 140-023200 140-023200 140-023200

Figure 67: LINFRA Fibre to terminations integrations

For the integrations shown in Figure 66 and Figure 67 these will occur many times during the installation of the LINFRA sub-element. The installation of the fibre cables along and through the pathways (and by implication, integration) will occur before the termination of the fibre cable. Installation activities can however proceed in a parallel manner. For example, two sections of cable that meet once installed can be joined together whilst further cable sections are installed. This work will be undertaken by suitably qualified and experienced personnel. The timeframes for completion of these component integrations is subject to the related construction and installation activities by other telescope elements.

System Integration

Integration for LINFRA at a system level will follow the installation of assemblies of LINFRA products. The contractors installing these products will group and undertake similar activities down trade and skill lines.  At each RPF o Internal fibre reticulation along internal pathways o Installation of rack/cabinet and PDU o Installation of fibre patch panel and associated fibre splicing o Splicing of fibre at the Waveguides o Installation of UPS  External Plant o Installation of conduit and pits o Hauling of fibre cable through conduit o Jointing and splicing of fibre cables o Installation of fibre cable through waveguides

2018-11-20 Page 135 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

 At the CPF o Internal fibre reticulation along internal pathways o Installation of rack/cabinet and PDU o Splicing of fibre at the Waveguides o Installation of the ODF, fibre patch panels and associated fibre splicing o Installation of KVM Switches  At the EOC and SOC buildings o Installation of rack/cabinet and PDU  At the SPC o Installation of rack/cabinet and PDU o Installation of KVM Switch The tasks at each location will generally follow in the ordering above, although the actual installation programme will reflect how the work has been divided and procured, and how the successful contractor has optimised the ordering of tasks. The tasks above will be dependent on works being undertaken by the INAU element, namely the trenching, site preparation and construction of buildings (CPF and RPF). Systems integration in a LINFRA context means the complete installation of the necessary fibre links from the CPF to each RPF, internal links inside the CPF building, and the installation of racks and PDU in both buildings. Once this installation is complete, testing (and in the unlikely event of a problem, rectification) of the fibre links occur before wider systems integration begins with other sub-elements. Interdependencies are further discussed in §4.14.5 and §4.14.6 A high level process diagram is presented below (Figure 68) showing the integration steps required to install, integrate and commission a LINFRA fibre link.

• Where appropriate, site survey to Site Survey understand the readiness of the site • Internal to buildings install cable tray as described in the site specific design Install cable documentation pathways • External to buildings install manholes and conduit into trenches. Connect as described. • Internal to the building from the Install fibre along waveguides, along the cable trays to and through cable the rack containging the ODF pathways • External to the building haul the cable through the conduits

• Splice cables Splice together cable together in fibre segements external to buildings joints located in pits

Install, manage and • At both the CPF and RPF buildings, nstall splice at Waveguides the external cable through the waveguide, splice and connect to internal cables

Splice and • Terminate the terminate cables at internal cables at ODF and fibre patch the ODF and fibre panels patch panels

Test end to end in • Test end to end with line with the OTDR, make sure link applicable has no faults and commissioning conforms with documentation requirements

• Enable wider Handover system integration

Figure 68: System Integration for LINFRA fibre link

2018-11-20 Page 136 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

It is likely that the integration of the LINFRA fibre links will occur in a staggered basis with a progressive rollout of RPF structures. A restricted level of integration is required for the delivery of the AIV facility ahead of wider telescope rollout.

Wider System of Interest (WSOI) Integration

LINFRA provides fibre, rack, UPS, PDU and KVM interfaces requiring integration with the other SADT sub- elements. These are listed below in Table 38.

Integrations

Fibre Rack Uninterruptable Power Distribution Keyboard, Power Supplies Unit (PDU) Video, Mouse (UPS) (KVM)

NSDN NSDN NSDN‡ NSDN‡ SAT.LMC CSP-SDP NMGR NMGR‡ NMGR‡ NSDN SAT.STFR.UTC CSP-SDP CSP-SDP NMGR SAT.STFR.FRQ SAT.STFR.UTC SAT.STFR.UTC SAT.CLOCKS SAT.CLOCKS SAT.STFR.FRQ SAT.STFR.FRQ SAT.LMC SAT.LMC SAT.CLOCKS SAT.CLOCKS

‡ Includes management interface for UPS and PDU Table 38: Wider System of Interest integrations

These integrations reflect the interfaces are documented in detail in Internal Interface Control Documents between LINFRA and the other sub-elements.

SADT Sub- Relevant IICD Element

CSP-SDP [RD11] SKA-TEL-SADT-0000438_ICD LINFRA to CSP-SDP (LOW) Internal Interface Document

SAT.STFR.UTC [RD2] SKA-TEL-SADT-0000439_ICD LINFRA to SAT.STFR.UTC (LOW) Internal Interface Document

SAT.STFR.FRQ [RD3] SKA-TEL-SADT-0000440_ICD LINFRA to SAT.STFR.FRQ (LOW) Internal Interface Document (THU)

SAT.CLOCKS [RD15] SKA-TEL-SADT-0000441_ICD LINFRA to SAT.CLOCKS (LOW) Internal Interface Document

NMGR [RD27] SKA-TEL-SADT-0000442_ICD LINFRA to NMGR (LOW) Internal Interface Document

SAT.LMC [RD28] SKA-TEL-SADT-0000443_ICD LINFRA to SAT.LMC (LOW) Internal Interface Document

NSDN [RD1] SKA-TEL-SADT-0000445_ICD LINFRA to NSDN (LOW) Internal Interface Document Table 39: IICD documents relevant to integrations

2018-11-20 Page 137 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

All integrations require the physical components of the LINFRA installation to be completed before the integration can occur. For the Rack integration, the rack must be installed into the building concerned, and with appropriate services connected (for example, earthing and cooling) For the Fibre integration, the fibre link required must be fully installed and tested. By necessity, this requires the necessary racks and cable pathways to be installed. For the UPS integration, the necessary rack must be installed with UPS power connected to power distribution located within it. Power is required to the building for the verification of this integration/interface. For the PDU integration, the necessary rack must be installed with the PDU to its source of supply. Power is required to the building for the verification of this integration/interface. The UPS and PDU units have management and monitoring interfaces to NMGR, and network interfaces to NSDN. Power is required to the building for the verification of this integration/interface For the KVM Switch integration, the necessary servers must be present for the KVM cables to be connected.

Precursor Integration

There is no defined integration between the SADT LINFRA sub-element and the precursor telescope infrastructures at the MRO site. It is anticipated that the LINFRA sub-element may be required to:  Facilitate the connection for CSP-SDP sub-element to the existing infrastructure at ASKAP for connection between the CPF and the EOC.  Facilitate connection of the NSDN sub-element to the existing ASKAP pre-cursor network for integration to the IP Telephony facilities at the site. The exact details of these integrations are yet to be determined. It is anticipated this this integration will be required early in the construction phase in order to support construction and commissioning activities.

SADT Interdependencies

The dependencies of other SADT sub-elements on the LINFRA sub-element match the integrations noted in Table 38. For complete integration of the management and configuration interfaces of the PDUs and UPS units with NMGR, NSDN is required to be integrated both with LINFRA and NMGR. NSDN provides the underlying network connectivity between these sub-elements. The timing of these interdependencies will rely on the [AD6] AIV roll-out plan and the [AD7] LOW Construction Plan. Temporary arrangements may be required to effect initial integration until the permanent infrastructure is in place.

Non-SADT (External) Dependencies

The external integration dependencies for the LINFRA sub-element are reflected in the [AD7] LOW Construction Plan. For the installation and integration of the conduit and pits for the backhaul and distribution fibre this needs to occur concurrently with the HV electric cable being installed into the shared trench provided by INAU.

2018-11-20 Page 138 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

The completion of construction of the CPF and RPF buildings by INAU is required for the final installation of the external fibre to the waveguides at the building, plus the installation in the buildings of internal fibre, racks, power distribution units, and UPS. The integration of PDU and UPS can be completed once INAU has provided power to the building and energised the supplies. To verify the integration of management of alarms from the PDU and UPS units, NMGR and NSDN have to be integrated with the TM element at logical and network levels respectively. In turn, the CSP-SDP link (and underlying NREN connectivity) will need to be in place to enable the NSDN integration to be tested and proved.

2018-11-20 Page 139 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

4.15 Interoperability Wikipedia95 offers the following definition of interoperability: The ability to provide services to and accept services from other systems, and to use the services exchanged to enable them to operate effectively together. LINFRA has a mix of physical and logical interfaces across which services are provided.

SADT Interoperability

LINFRA interoperates with other SADT sub-elements to provide:  Racks and Cabinets to house equipment  Power Distribution Units and UPS to supply power to equipment  Fibre connectivity between locations to link equipment together  KVM Switches to enable management of server equipment Additionally, the NSDN and NMGR sub-elements interoperate with LINFRA to provide monitoring, management and alerts for the:  Power Distribution Units  UPS Specific interface requirements are contained within the IICD documents referenced in Table 40 below.

SADT sub-element Relevant IICD [RD1] SKA-TEL-SADT-0000445_ICD LINFRA to NSDN (LOW) Internal Interface NSDN: Document [RD2] SKA-TEL-SADT-0000439_ICD LINFRA to SAT.STFR.UTC (LOW) Internal Interface STFR.UTC: Document [RD5] SKA-TEL-SADT-0000440_ICD_LINFRA to SAT.STFR.FRQ (LOW) Internal Interface STFR.FRQ Document (THU) [RD11] SKA-TEL-SADT-0000438_ICD LINFRA to CSP-SDP (LOW) Internal Interface CSP-SDP Document [RD15] SKA-TEL-SADT-0000441_ICD LINFRA to SAT.CLOCKS (LOW) Internal Interface SAT.CLOCKS Document [RD27] SKA-TEL-SADT-0000442_ICD LINFRA to NMGR (LOW) Internal Interface NMGR Document [RD28] SKA-TEL-SADT-0000443_ICD LINFRA to SAT.LMC (LOW) Internal Interface SAT.LMC Document Table 40: Internal Interface Control Documents with other SADT sub-elements

Each of the sub-elements interoperates with LINFRA through standard connectors, interfaces and protocols.

95 https://en.wikipedia.org/wiki/Interoperability#Telecommunications

2018-11-20 Page 140 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Non-SADT (External) Interoperability

LINFRA interoperates with directly with the INAU element with:  equipment (rack) accommodation inside buildings  environmental management and cooling water provision to cabinets (including associated controls)  cable reticulation including cable trenches, cables trays, building cable entries and waveguides  power and earthing systems LINFRA interoperates with directly with the LFAA element with:  fibre connectivity between locations to link equipment together LINFRA interoperates indirectly via other SADT sub-elements by:  providing cable pathways to enable CSP, LFAA and SDP elements to connect with other SADT sub- elements  transmitting appropriate telemetry and alarms via NMGR to TM LINFRA interoperates indirectly with the NREN, AARNet, with:  equipment (rack) accommodation inside buildings  cable reticulation including cable trenches, cables trays, building cable entries and waveguides  power and earthing systems The exact nature of this interoperation with AARNet will be documented further in future work envisioned in §6.2.6.Specific interface requirements with other telescope elements are contained within the EICD documents in Table 41.

Other Element Relevant EICD

INAU [RD10] 100-000000-024 SKA1 LOW Telescope Interface Control Document SADT to INAU

LFAA [RD9] 100-000000-026 Interface Control Document SADT to LFAA

CSP [RD29] 100-000000-023 Interface Control Document SADT to CSP (LOW)

SDP [RD30] 100-000000-025 Interface Control Document SADT to SDP (LOW)

TM [RD31] SKA-TEL-SKO-0000153 SKA1 Interface Control Document TM to SADT Table 41: External Interface Control Documents between SADT and other elements

Each of the sub-elements interoperates with LINFRA through standard connectors, interfaces and protocols.

2018-11-20 Page 141 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

5 EVALUATION

5.1 Fitness for Purpose Paraphrasing from the [AD13] LINFRA Technical Requirements Specification, the LINFRA sub-element must provide six functionalities: i. Provide the fibre network infrastructure across the SKA1-LOW Telescope for other SADT sub- elements ii. Reuse existing fibre network infrastructure across the SKA1-LOW Telescope, wherever feasible. iii. Provide SADT-local “Enclosure” infrastructure items iv. Provide SADT-local “Power” infrastructure items v. Provide SADT-local “Environmental Control” infrastructure items vi. Provide SADT-local “Fibre and Copper Cable Reticulation” infrastructure items These functionalities allow the other SADT sub-elements to function. The key science enabler from the [AD14] SKA Design Reference Mission that the LINFRA sub-element enables is providing the physical medium for the transmission of astronomy data over the long baselines (with LFAA being the other key telescope element here) The technologies and components selected for the LINFRA sub-element are of a “commercial off the shelf (COTS)” nature and are being used in a traditional application (albeit in a unique environment). This enables other SADT sub-elements to make use of the infrastructure with the use of unique or special solutions. The COTS nature of the SADT technologies and components used means that no specialist knowledge is required for their maintenance and is within the capability of existing potential industry partners with competent, experienced and trained personnel. Likewise, these items be competitively priced due to their common use elsewhere in the telecommunications sector. Availability and diversity of the infrastructure component of the LINFRA sub-element is limited by the cost to provide. The FMECA and RAMS analysis for the LINFRA sub-element should be reviewed once the complete telescope design is available to ensure it is consistent with overall availability expectations for the telescope. The design, in the selection of cable sizes, includes some spare capacity for the purposes of maintenance, reconfiguration and to enable additional experiments. The selection of these cable sizes is based on various manufacturer’s standard product lines. This capacity, plus the flexible nature of the cable infrastructure in terms of re-use, enables limited opportunities for SKA2.

Functionalities

The SADT Element has three generalised responsibilities in the design of the SKA Telescopes: 1. Provide a timescale to synchronise science data across each Telescope; and to, maintain and manage the accuracy and coherence of the timescale to BIPM, thus making each SKA Telescope a UTC(k) instance. 2. Provide the networking infrastructure required to distribute science and calibration data from the pivotal Telescope locations; namely, Receptors, Remote and Central Processing Facilities (RPF/CPF), Science Processing Centre (SPC), and also beyond the SKA. 3. Provide separate, non-science data carrying networks across the SKA facilities which are required to support the operational aspects of the Telescopes. The LINFRA Sub-Element supports these functionalities by providing to other SADT sub-elements:

2018-11-20 Page 142 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

1. Fibre optic cable connectivity between the CPF and the RPF locations. This connectivity transports both science and non-science data, and timing and frequency reference signals. 2. Cabinets and racks to house SADT equipment across all telescope locations 3. Power distribution within cabinets and racks used by SADT equipment at all telescope locations 4. UPS equipment at the RPF locations to the support the NSDN sub-element 5. Cable reticulation pathways for the carrying of cables between elements and sub-elements interconnected with SADT Telemetry from the PDU and UPS units will not be required to meet the responsibilities but is necessary for the operational management of the telescope. The LINFRA sub-element has functional dependences on the INAU element for provision of: 1. CPF and RPF buildings to house the cabinets and racks 2. Power supplies to the PDU (including those backed by a common INAU UPS) 3. Power supplies to the SADT UPS units 4. Cooling water to the racks at the CPF 5. Cooling at the RPF 6. Waveguides for the entry of cables into the CPF and RPF buildings 7. During construction, the provision of a shared trench

Functional Opportunities

The configuration and design of the LINFRA sub-element follows the overall telescope design. The functionalities noted above can be extended, and alternative uses for the LINFRA products can be undertaken. All aspects of the LINFRA sub-element comprise assemblies of standard Commercial Off The Shelf (COTS) components. The components are modular and can be re-purposed, although may not necessarily be suitable for re-use. As implemented, the LINFRA design will have limited spare capacity in the external fibre plant. This can be used as the basis to extend the telescope, either geographically or with additional experiments. In addition, with the future development of new transmission equipment, optics and modulation techniques, the data carrying capacity of the LINFRA sub-element can be upgraded and expanded.

Functional Weaknesses

The LINFRA sub-element has been designed, due to cost constraints, with a single path for the fibre optic backhaul cables along the telescope spirals. This creates single points of failure for the LINFRA sub- element and the other elements and sub-elements that rely on the LINFRA infrastructure. This weakness also exists for the infrastructure providing the CSP-SDP link. Alternative cable paths could be added at a cost to provide diversity, but the design and technologies used would need to be adjusted.

2018-11-20 Page 143 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

5.2 Verification

Design Verification (Pre-CDR)

5.2.1.1 Overview The LINFRA design has been developed based on industry standard components and products. These are assembled and integrated in a straight forward manner. This approach has a number of advantages for the SKA1 LOW telescope:  Using standards-based products significantly de-risks the project.  Strong understanding in industry of the skills, equipment and techniques required to install these standard products  Interfaces with other sub-elements are with industry standard connectors, interfaces and protocols.  Ease of integration with other elements using the same standard.  No development required for the interface.  Equipment can be sourced from multiple vendors Racks, Power Distribution Units and Uninterruptable Power Supplies are commercial off the shelf items and are used widely in industry. The fibre network (including cable, conduits, pits and ODFs) is assembled out of standard components. The precursor telescopes contain examples of these items. Item Verification Performed by Fibre Analysis See §5.2.1.2 Fibre Similarity Similar components installed at ASKAP and MeerKAT precursors, and throughout telecommunications industry Racks Similarity Similar cabinets installed at ASKAP Power Distribution Similarity Similar PDU installed Units at ASKAP Uninterruptable Power Similarity Similar units used Supplies commercially across many industries KVM Switches Similarity Similar units used commercially across many industries Table 42: Design Verifications (Pre-CDR)

5.2.1.2 Calculations Undertaken for Outside Plant Fibre In response to the derived requirements from the other SADT sub-elements, as documented in the Internal Interface Control Documents, calculations have been undertaken to demonstrate that fibre network design will provide the necessary optical performance required. These were discussed previously in this document in §4.3.3 Analysis of the LOW Fibre Reticulation Design.

2018-11-20 Page 144 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

The calculation considers for the optical path from the CPF out to the RPF considering each component, splice, and length of optical fibre cable to arrive at a cumulative optical loss figure for the path. Typical values for each were assigned to perform the calculation. The result of the calculation was then compared against the requirements of equipment from the other sub-elements. 5.2.1.3 Known uncertainties and factors affecting Design Verification Complete design verification and modelling is constrained by the need for information outside the SADT element to be provided. This includes:  Confirmation of the location of the CPF building relative to the fibre routes  Confirmation of the design and internal layout of the CPF and the space to be allocated to the SADT element  Confirmation of the location of the RPF buildings relative to the fibre routes  Confirmation of the design and internal layout of the RPF buildings  Survey of the fibre routes, associated waypoints and lengths, and subsequent adjustments to the routes based upon addressing cultural, heritage and environmental issues. However, the locations of these items are constrained by other telescope elements (for example, locations of the stations) and the overall impact on the design is anticipated to be minimal.

Functional Verification (Post-CDR)

5.2.2.1 Overview Verification will occur at key milestones associated with the procurement, supply, installation, commissioning, integration and delivery of the fibre and ancillary infrastructure. The verifications required will be indicated in the construction specification. A mix of verification techniques (inspection, analysis, analogy, demonstration, test and sampling) will be used. For the Commercial Off The Shelf (COTS) items, these will primarily be verified by analogy, inspection and demonstration. Verification activities will be undertaken by various parties, and may be cross checked by others. During procurement, verification is required that the tender specification are aligned with the technical specifications, and that proposals from cable and equipment manufacturers and installers meet the functional requirements for LINFRA. Cable suppliers will need to provide samples of cable for testing and evaluation. Active equipment manufacturers (PDU, UPS, KVM, cabinet cooling system) will need to provide samples of candidate LINFRA equipment for RFI testing, prior to installation of equipment, to ensure that equipment meets the SKAO EMI standards. During the installation and commissioning phase, verification is required to ensure that the LINFRA items are correctly installed and that internal and external interfaces operate as expected. 5.2.2.2 Fibre and Pathway Verification will occur before, during and subsequent to the installation of this collection of products. Individual drums of fibre will be tested before departure from the factory and upon delivery on site before installation occurs. This will confirm the cable is expected to perform as expected once installed. The complete installation will be required to meet the functional and performance standard specified in the

2018-11-20 Page 145 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

user requirements. Key verification will be testing of the installed fibre optic links between each RPF and the CPF. This will be done with industry standard test equipment. This will include:  Calibrated Light Source and Optical Power Meter to verify connectivity  Optical Time Domain Reflectometer (OTDR) to characterise the performance of each fibre optic link at different wavelengths Additional measurements of dispersion may be required. Standard quality and compliance verification of the installation will also be required. 5.2.2.3 Racks and Cabinets These will be a COTS item and will be installed without fundamental modification. Before installation, verification will include examination of sample where appropriate, and inspection of specification sheet submitted by the vendor to ensure compliance with relevant standards and specifications. After completion of installation, further inspection will occur to check the quality and compliance of the finished installation. 5.2.2.4 Power Distribution Units, Uninterruptable Power Supplies and KVM Switches These will be a COTS items and will be installed without fundamental modification. Before installation, verification will include examination of sample where appropriate, and inspection of specification sheet submitted by the vendor to ensure compliance with relevant standards and specifications. Testing may need to occur to confirm compatibility with the fixings on the associated racks. After completion of installation, further inspection will occur to check the quality and compliance of the finished install. Where interfaces exist, these units will be connected into NSDN, configured, and integration tests performed to ensure correct functioning when communicating with NMGR.

Validation of Requirements

The [AD1] SKA1 System Baseline Design describes the following needs that the infrastructure provided needs to enable:  Provide and transfer time, derived from the Observatory Clock, to components of the system that require ‘time stamps’ [AD1] §13.1  Provide and distribute a frequency reference derived from the Observatory Clock [AD1] §13.1  Transport of output data from dishes or stations to the correlator / beamformer subsystems [AD1] §13.2.1  Transport output data from the correlator (CSP) to the science data processing centre (SDP) in Perth [AD1] §12 The LINFRA design meets all the requirements as per the Technical Requirements Specification for LINFRA (See [AD15]) This design enables these needs by providing infrastructure to other SADT sub-elements – fibre connectivity, cabinets and racks, UPS and power distribution.

2018-11-20 Page 146 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

The fibre connectivity enables the transport of the timing signals, frequency reference, and science and control data from the CPF buildings to the RPF buildings, enabling the functions of the LFAA element, and the SAT.STFR.UTC, SAT.STFR.FRQ, and NSDN sub-elements The cabinets and racks, UPS, KVM and power distribution are provided to all other SADT sub-elements. Together with the fibre connectivity, this collectively allows the LINFRA sub-element alongside the other SADT sub-elements to meet the needs of the telescope providing fundamental infrastructure to enable its functioning. The requirements for the LINFRA sub-element will continue to evolve due to external factors. These include: a) Confirmation of fibre requirements driven by selection of equipment for the LFAA element, and further development of the SAT.STFR.FRQ sub-element b) Changes to the design and nature of the CPF building c) Changes to the design and nature of the RPF building d) Confirmation of the EOC, SOC and SPC locations e) Changes in other elements and sub-elements

Recommendations

It is recommended that the documentation and specification for the procurement of the LINFRA sub- element include:  Mandatory provision of samples and submission of shop drawings  Independent calculation and verification by the tendering parties of the performance of the fibre elements before cable procurement and manufacture  Mandatory inspection and witness points during the construction of the Outside Plant fibre  Independent review of submitted test results from the contractors.  Requirements to co-ordinate with other contractors delivering other elements and sub-elements to align the timing and delivery of testing and documentation submission. Additional, during the procurement process, illustrative examples of the items below should be sought from tendering parties:  Installation records and test results  Mandatory and statutory compliance certificates  Operation and Maintenance documentation  Instruction and training materials  Recommendations of levels of sparing and inventory

2018-11-20 Page 147 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

5.3 System Reliability, Availability, Maintenance and Safety The SADT work-package engaged Fraser-Nash Consultancy to undertake a Failure Modes, Effects and Criticality Analysis (FMECAs) against each SADT sub-element. Workshops were carried out in September 2016. The outcome of this analysis is located in [AD11] Signal and Data Transport Group Detailed RAMS Analysis Report

Key Points

At this stage, with the issue of the [AD11] Signal and Data Transport Group Detailed RAMS Analysis Report, the following recommendations were offered:  Continue to identify failure modes against each of the equipment types in the LINFRA FMECA  Refine the failure data available and align the common equipment types across the SaDT system  Continue to populate and expand details for the LINFRA FMECA, refining the data available and aligning the common equipment types across the SaDT system  By understanding the higher level system and sub-system criticality better, ensure end effects are correct and the result on the SaDT system (and SKA as a whole) are correctly categorised  Ensure that the severity and likelihood of similar failures are balanced and consistent throughout LINFRA and, if possible, across SaDT The report observed:  For the fibre infrastructure, this is reliable and likelihood of failure is relatively low. The report however stressed the importance of ensuring a quality installation.  For other LINFRA products, the equipment is straight forward and suffers from few failure modes.

Recommendations

The recommendations this design makes in response to the FMECA report are to: 1. Maintain records of failures and allocate ongoing capital funds to mitigate 2. Undertake a regular independent risk and reliability review of the infrastructure 3. Inform other elements and sub-elements of the inherent risks associated with the design 4. Have a documented maintenance and repair policy and procedure

2018-11-20 Page 148 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

5.4 Costs

Summary of Costing Assumptions

The [RD8] SKA1 LINFRA LOW Cost Model reflects this design. Several assumptions where made in the model, the key assumptions made were:  Design o There will be 36 RPFs (LFAA Remote Processing Facilities) associated with the 36 stations documented in [AD3] SKA1-LOW Configuration Co-ordinates, dated 2016-05-3196 o Power reticulation for the RPFs is being provided by conventional HV and LV (as opposed to a PV based solution)97 o Fibre shall be reticulated in conduit98 o UPS and PDU units are for the sole use of SADT equipment99  Trenching and Conduit o Where practical, fibre cables will be laid in conduit in the same trench as the power cables. The cost of this trench and backfill is born by INAU. The cost of placing the conduit in the trench is born by SADT100 o Cost of trenching where not running in a shared trench with HV or LV electrical services is SADT Cost101 o Connectivity to spiral arrays are treated individually, in practice will be co-ordinated with core trenching activities.102  Fibre o Fibre is a consistent type across each spiral for outdoor plant (for both inner and outer spiral)103 Costing based on these assumptions is found in [RD8] SKA1 LINFRA LOW Cost Model spread sheet

Cost Saving Opportunities

Opportunities could exist to revisit this design to generate cost savings. One example would be to consider different sizes of cable for the inner and outer portions of the spiral. Given the distances involved there is a cost saving that can be achieved here, although this may as a proportion of the LINFRA cost estimate be negligible and restrict future telescope development.

96 [RD6] SADT.LINFRA.MDAL-0221 97 [RD6] SADT.LINFRA.MDAL-0222 98 [RD6] SADT.LINFRA.MDAL-0231 99 [RD6] SADT.LINFRA.MDAL-0281 100 [RD6] SADT.LINFRA.MDAL-0223 101 [RD6] SADT.LINFRA.MDAL-0224 102 [RD6] SADT.LINFRA.MDAL-0225 103 [RD6] SADT.LINFRA.MDAL-0226

2018-11-20 Page 149 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Recommendations for Procurement

Bundle purchases together to increase procurement volumes There are opportunities aggregate purchases of materials and components for the telescope, and provide opportunities with vendors:  Internally within the SADT element, for example, fibre and copper patch leads could be procured in bulk  Externally with other elements, procure common items together  Align LINFRA fibre procurement with the larger fibre volume being procured by LFAA. LFAA will be procuring a similar / slightly larger cable size which can be substituted for the LINFRA backhaul fibre. This will also assist with sparing for the telescope with common cable type and size used across elements.  Procure related purchases together. For example, racks with power distribution units or, splices with splice tray Rationalise procurement activity Due to the SADT.LINFRA sub element being required to be available early on in the build of the telescope, and a requirement to co-ordinate with the installation activities of the INAU element:  Purchase all fibre cable and componentry at the same time  Purchase all CPF and RPF rack and power distribution equipment at the same time  Aim to consolidate suppliers and deliveries Address perceived risk To avoid vendors unnecessarily pricing in larger allowances for risk associated with the site, engage in a formal program of:  Site visits  Tender documentation briefings

2018-11-20 Page 150 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

5.5 Assumptions A number of assumptions have been made in order to progress the design. All working assumptions are made in the [RD6] SADT Master Data Assumptions List. Some are also documented in the [RD32] SADT Risk Register, as some of these assumptions pose a risk which needs to be managed. A summary of the most pertinent assumptions that should be highlighted for the LINFRA detail design are the following:  Location of CPF and RPF buildings The location of these buildings has been assumed in relation to the array centre and station locations provided in [AD3] SKA1 LOW Configuration Co-ordinates104  Shared trench with INAU and arrangement of trenches The assumption that a co-ordinate engineering solution for the trench will be designed, and a cost saving will be realised with the shared trench105  Routes are straight lines Cable pathways between locations follow the most direct route with no deviations106  Uninterrupted work That the work to undertake the construction of the LINFRA sub-element can be accomplished in a singular programme of works with no interruptions. This includes the completion of buildings to enable the fibre and other LINFRA items to be installed.107  Quality contractors That the installation of the optical fibre will be undertaken by experienced contractors with appropriate quality control mechanisms in place108

104 [RD6] SADT.LINFRA.MDAL-0227 105 [RD6] SADT.LINFRA.MDAL-0017 106 [RD6] SADT.LINFRA.MDAL-0230 107 [RD6] SADT.LINFRA.MDAL-0228 108 [RD6] SADT.LINFRA.MDAL-0229

2018-11-20 Page 151 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

5.6 Risks All risks associated with the LINFRA project can be found in the [RD32] SADT Risk Register. At the time of writing there are eleven risks outstanding on LINFRA for the SKA1 LOW telescope, none of which have a high risk rating. Nine risks are medium risk rated and the remaining two are low risk. The twelve medium risks are as follows: SKA.SADT.RSK.156 – Final locations of RPF buildings for SKA1 LOW are yet to be known SKA.SADT.RSK.157 – Routing details of power for SKA1 LOW affecting LINFRA design SKA.SADT.RSK.159 – Final location of CPF building for SKA1 LOW is yet to be known SKA.SADT.RSK.165 – Allocation of space for racks in CPF remains unknown SKA.SADT.RSK.166 – Likelihood of changes of configuration in design of LOW SKA.SADT.RSK.184 – The EOC location is yet to be firmly defined. SKA.SADT.RSK.185 – The SOC location is yet to be firmly defined. SKA.SADT.RSK.186 – The SPC location is yet to be firmly defined. At the time of writing, the finalised locations of the CPF , RPF, EOC, SOC and SPC buildings are not known for LOW. As mitigation, assumed locations and distances have been used in lieu when modelling this design. Further work is required with the SKA Office and other telescope work packages to address these risks.

SKA.SADT.RSK.160 – Poor sequencing construction of LINFRA for SKA1 LOW with others Construction work for the LINFRA sub-element is required to be co-ordinated with the installation of the INAU element and AIV work package activities. At the time of writing, minimal information is available to ensure these are well co-ordinated.

SKA.SADT.RSK.161 – Incorrect understandings around trench product "ownership" with INAU for SKA1 LOW The trench product requires further design work to ensure co-ordination with the INAU element.

SKA.SADT.RSK.162 – Incorrect understandings around rack product "ownership" with INAU for SKA1 LOW SKA.SADT.RSK.164 – For SKA1.LOW, rack specification is inconsistent with other elements

Further clarification is required around the allocation of cabinet products at the CPF and RPF buildings.

2018-11-20 Page 152 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

6 RECOMMENDATIONS AND DEVELOPMENT ROADMAP

6.1 Summary of Deliverables The design report provides a design, technical requirements and costing for the Local Infrastructure (LINFRA) for the SADT element of the SKA1 LOW telescope. The design report is not suitable to serve as a substitute for tender specification, which needs to be compiled during the procurement phase. The technical requirements and design specifications in this document can be used as input to a tender specification process when the outstanding issues and work as outlined in this section have been addressed. It is suggested that an Expressions of Interest (EOI) process be run with suppliers and contractors to ensure capability is available in the market. It is recommended that any procurement efforts for LINFRA be co- ordinated with similar activities for the INAU and LFAA elements. The acceptance of the chosen bidder is dependent upon sample equipment made available for testing to ensure compliance with the SKA RFI standard.

6.2 Outstanding Issues/Work The following outstanding items need to be addressed before the procurement and installation of LINFRA for the SKA1 LOW telescope can commence.

Further design work for GNSS antennas and GNSS calibration shelter

A suitable location needs to be selected and agreed for the GNSS antennas. This will require input from the SADT.CLOCKS and SADT.LINFRA sub-elements and the INAU element. With the location, the design can be completed with provision of:  Cable route between the CPF Maser Room and the GNSS Antennas  Power supplies by INAU  Mounts for the GNSS Antennas  Suitable plinth for installation of GNSS Calibration Shelter  Safe access to the GNSS Antennas and GNSS Calibration Shelter  Specification and selection of the GNSS Calibration Shelter

Completion of site survey work

The finalisation of the route of the cable along the spirals and for the CSP-SDP connectivity requires further desktop and site survey work to be undertaken to adjust the route to: 1. Accommodate features and conditions of the terrain (eg rock, watercourse) 2. Determine and minimise environmental impact (particularly to flora and fauna) 3. Consider and minimise impact to cultural and indigenous heritage 4. Finalise and confirm locations of CPF and RPF buildings 5. Integrate and incorporate SADT.LINFRA, INAU and LFAA designs 6. Provide the necessary level of detail to meet agreements with stakeholders and minimise construction risk

2018-11-20 Page 153 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

This work is assumed to be undertaken by others (for example, INAU and Host Country).109 This design and associated [RD8] cost model may require updates to reflect the outcome of the survey. A key outcome of the survey work will be the confirmation of quantities of cable and conduit that are required, and will enable procurement activities to proceed.

Alignment of design with work by INAU

INAU have been undertaken the design work for the CPF and RPF buildings, and the electrical distribution infrastructure separately. This also includes any “last-mile” fibre connections for INAU specific equipment.. During verification of the [RD10] Interface Control Document between INAU and SADT, both CDR document sets for both consortia should be read side-by-side in a holistic fashion to ensure alignment and to check for any negative interactions. It is recommended that further work be undertaken to bring both designs into closer alignment in respect of:  Trenching – Confirm routes and complete extent of power and fibre route overlay. Further establish methodology for installation of conduit and cables in shared trench.  Fibre Wave Guides – Confirm design detail of wave guides into the CPF and RPF building  Power Distribution at CPF and RPF – Confirm spatial configuration and location of connection  Floor and Rack Layouts of CPF and RPF – Confirm location of equipment and utilisation of cable routes between equipment  GNSS Antennas and Equipment Shelters – Confirm nature and configuration of proposed solution  Power Station Connectivity – Establish route of fibre at arrival at the Power Station

Alignment of design with work by LFAA

LFAA have been undertaking the design work for the antenna and digitisation systems separately. During verification of the [RD9] Interface Control Document between LFAA and SADT, both CDR document sets for both consortia should be read side-by-side in a holistic fashion to ensure alignment and to check for any negative interactions. It is recommended that further work be undertaken to bring both designs into closer alignment in respect of:  Fibre – Establish the physical fibre interface between the LINFRA fibre patch panels and LFAA in respect of quantity and location of fibres, termination and interface points.

Alignment of design with work by AIV

AIV are undertaking planning and co-ordination work to support the rollout of the telescope. LINFRA configurations to support this activity will need to be updated to reflect revisions to their plans post SADT CDR. This includes fully scoping any work associated with the temporary structure providing “Temporary CPF” services for the AIV AA-1 milestone.

109 [RD6] SADT.LINFRA.MDAL-0232

2018-11-20 Page 154 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

Arrangements for extension of the existing CSIRO/AARNet cable to the CPF

A scope of work will need to be developed with CSIRO and AARNet to enable this connection to take place. This will likely be under the aegis of any host agreement.

Arrangements for the EOC, SOC and SPC facilities

The Australian Science Operations Centre (SOC), Australian Engineering Operations Centre (EOC) and Australian Science Processing Centre (SPC) are only referenced by their broad location (city level) in [AD2] SKA Phase 1 System Requirements Specification. Once their final locations and operational arrangements are known, this design may require updates to ensure alignment, and to reflect interactions with other elements designs. This will also include ensuring appropriate arrangements and infrastructure is in place for the connectivity between these locations.

Corrections and harmonisation for System CDR process

Updates and corrections will be needed to this design document and others as a result of the System CDR process. This includes implementing the outcomes of any System level ECP documents.

Transformation of design into specification for procurement

This design will need to be transformed into a suitable document for procurement. This is to:  lower the level of design uncertainty  narrow the specification of components where necessary  manage the procurement risk  formalise the language used into the legal and technical language of specification and contract  address any documentation (design, drawing) gaps  add the additional of the necessary contractural preliminaries  reflect the procurement model and stakeholders in the process The resulting specification will need a final review of its technical and compliance aspects before issue. This would likely be undertaken by both the project team and an independent third party. Ongoing validation of the costs will also need to occur.

6.3 Recommended Actions It is recommended to address the outstanding issues as described in §6.2.

2018-11-20 Page 155 of 158 SKA-TEL-SADT-0000260-DDD Rev: 2.14.0 SKA1 LOW LINFRA DETAIL DESIGN REPORT

7 STATEMENT OF COMPLIANCE

This design complies with the following requirements documents:  [AD2] SKA-TEL-SKO-0000008 SKA1 Phase 1 System Requirements Specification, Rev 11, 31 July 2017  [AD13] SKA-TEL-SADT-0000107_SPE LINFRA Technical Requirements Specification (TRS)  [AD15] SKA-TEL-SADT-0000520 SADT Element Technical Requirements Specification The following specific non-compliances against [AD13] SKA-TEL-SADT-0000107_SPE LINFRA Technical Requirements Specification (TRS) exist: Requirement ID Requirement Description Compliance LINFRA shall provide diverse network connectivity between the Central Processing Facility (CPF) and the Remote Processing Facilities (RPFs) (LOW) for: LINFRA_REQ-105- Not Non-science data networks 183 compliant Science data networks Timing and synchronisation signal networks LINFRA shall provide diverse network connectivity between the Central Processing Facility (CPF) and the LINFRA_REQ-105- Not Engineering Operations Centre (EOC) (LOW) for the: 184 compliant Non-science data networks LINFRA shall provide diverse network connectivity between the Central Processing Facility (CPF) and the LINFRA_REQ-105- Science Processing Centre (SPC) (LOW) for the: Not 185 Science data networks compliant Non-science data networks LINFRA shall provide diverse network connectivity between the Central Processing Facility (CPF) and the LINFRA_REQ-105- Science Operations Centre (SOC) (LOW) for the: Not 186 Science data networks compliant Non-science data networks Table 43: List of non-compliances with requirements

8 CONCLUSIONS/RECOMMENDATIONS

8.1 Conclusions & Recommendations In conclusion, this design meets the requirements as stated in:  [AD2] SKA-TEL-SKO-0000008 SKA Phase 1 System Requirements Specification, Rev 11, 31 July 2017  [AD13] SKA-TEL-SADT-0000107-SADT.LINFRA.XXX-TRS-01-LINFRA Technical Requirements Specification (TRS)  [AD15] SKA-TEL-SADT-0000520 SADT Element Technical Requirements Specification Given those requirements this design is in response. Based on the verification and validation information included in this document, a working implementation can be constructed based on this design.

The design is comprehensive with the following limitations: 1. Further co-ordination is required with the INAU on the design of trenches, CPF and RPF buildings 2. Further modelling may be required, co-ordinating the outputs of the modelling exercises with the dependant SADT work package equipment, with recommendations to follow optimising both equipment and fibre cable performance with respect to system reliability and performance, and economic viability Further design work is required to develop this design into a “For Construction” specification. As submitted, this design is recommended for use.

2018-11-20 Page 156 of 158 SKA-TEL-SADT-0000260-DDDSKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORTSKA1 LOW LINFRA DETAIL DESIGN REPORT

9 Appendix A – Fibre Splice Diagrams

Figure 69, Figure 70, Figure 71, and Figure 72 show typical fibre splicing configurations along each spiral.

TUBE-01 001-012 TUBE-01 001-012 TUBE-02 013-024 TUBE-02 013-024 TUBE-03 025-036 TUBE-03 025-036 TUBE-04 037-048 TUBE-04 037-048 TUBE-05 049-060 TUBE-05 049-060 TUBE-06 061-072 TUBE-06 061-072 TUBE-07 073-084 TUBE-07 073-084 CPF 192 CORE G.652.D TUBE-08 085-096 TUBE-08 085-096 192 CORE G.652.D TUBE-09 097-108 TUBE-09 097-108 J28 TUBE-10 109-120 TUBE-10 109-120 TUBE-11 121-132 TUBE-11 121-132 TUBE-12 133-144 TUBE-12 133-144 TUBE-13 145-156 TUBE-13 145-156 TUBE-14 157-168 TUBE-14 157-168 TUBE-15 169-180 TUBE-15 169-180

TUBE-16 181-192 TUBE-16 181-192

T T

24 CORE G.652.D 192 CORE G.652.D

J26 S7 TOWARDS CPF Figure 69: Typical Splice Arrangement – Branching (J27 near RPF S7)110

24 CORE G.652.D

S6 TOWARDS CPF Figure 70: Typical Splice Arrangement – On Branch (J26 near RPF S6)111

110 Drawing [D5]

2018-11-20 Page 157 of 158 SKA-TEL-SADT-0000260-DDDSKA-TEL-SADT-0000260-DDD Rev: 4.0 SKA1 LOW LINFRA DETAIL DESIGN REPORTSKA1 LOW LINFRA DETAIL DESIGN REPORT

TUBE-01 001-012 TUBE-01 001-012 TUBE-02 013-024 TUBE-02 013-024 TUBE-03 025-036 TUBE-03 025-036 TUBE-04 037-048 TUBE-04 037-048 TUBE-05 049-060 TUBE-05 049-060 TUBE-06 061-072 TUBE-06 061-072 TUBE-07 073-084 TUBE-07 073-084 J33 192 CORE G.652.D TUBE-08 085-096 TUBE-08 085-096 192 CORE G.652.D TUBE-09 097-108 TUBE-09 097-108 J35 TUBE-10 109-120 TUBE-10 109-120 TUBE-11 121-132 TUBE-11 121-132 TUBE-12 133-144 TUBE-12 133-144 TUBE-13 145-156 TUBE-13 145-156 TUBE-14 157-168 TUBE-14 157-168 TUBE-15 169-180 TUBE-15 169-180 TUBE-16 181-192 TUBE-16 181-192 2 4 1 2 0 0 - - 1 3 0 1 0 0 1 2 0 0 - - E E B B U U T T

24 CORE G.652.D

S15 TOWARDS CPF Figure 71: Typical Splice Arrangement – On Trunk (J34 near RPF S15)112

TUBE-01 001-012 TUBE-02 013-024 TUBE-03 025-036 TUBE-04 037-048 TUBE-05 049-060 TUBE-06 061-072 TUBE-07 073-084 J35 192 CORE G.652.D TUBE-08 085-096 TUBE-09 097-108 TUBE-10 109-120 TUBE-11 121-132 TUBE-12 133-144 TUBE-13 145-156 TUBE-14 157-168 TUBE-15 169-180

TUBE-16 181-192

T T

24 CORE G.652.D

S16 TOWARDS CPF Figure 72: Typical Splice Arrangement – End of Trunk (J36 near RPF S16)113

111 Drawing [D5] 112 Drawing [D5] 113 Drawing [D5]

2018-11-20 Page 158 of 158