Power Utilities Energy Automation Paulo Pereira, IoE Solutions Sales Agenda
• Cisco Next Generation WAN for Power Utilities – teleprotection
• IEC Standards for Energy Automation – status and outlook
• IEC 61850 Substation Automation – latest station bus design
• Distributed Energy Resources – a new distribution grid model
• Conclusion / Key Takeaways Thanks!
• Ruben Lobo • Maik Seewald
• Darren Cranston • Tim Lyon
• Motaz Elshafi • Patrick Grossetete
• Dave Schmitt • Mitko Vasilev
• Sean Song Jiang
4 Cisco NG WAN for Power Utilities - teleprotection Power Utility Energy Automation Networks
• The Power System has multiple overlay networks like the Wide Area Network, Substation Network, Field Area Network, or the Control / Data Center. • Each of these networks host several applications with very different network requirements
6 Power Utilities HV Example Use Cases Use Case SCADA (DNP3, Modbus, T101) serial tunneling with Raw Sockets
SCADA (DNP3, Modbus, T101) transport over E&M LMR
SCADA (DNP3-IP, Modbus-TCP & T104) IP transport
Wide Area Measurement Systems (WAMS) with C37.118.2
Monitor, Wide Area Measurement Systems (WAMS) with IEC 61850-90-5 Measure, Control, IEC 61850 GOOSE messaging for Feeder Protection over Station Bus Automation, IEC 61850 SV messaging with Merging Units over Process Bus & Protection Traditional Teleprotection (Current Differential) with legacy interfaces
IEC 61850 Teleprotection (Current Differential) with Ethernet interfaces
System Integrity Protection Schemes (SIPS)
Wide Area Measurement Protection and Control (WAMPAC)
7 Cisco Solution Validated Design and Lab
• Dedicated solution validation lab for substation automation
• Designed to support current and future real-world Power Utilities use cases
• Lab consists of complete end-to-end utility SA network: NOC, substations, DMZ, WAN
• End-to-end validation with RTU, Relays, IED, PMU etc (ex. Siemens and Alstom).
• Test validation results documented in Design and Implementation Guides
8 Substation Automation DIG
. Substation Automation DIG Release 1.5 . Covers LAN and WAN use cases with CGS 2520 and CGR 2010 . DIG focused on off-net substations using leased Service Provider transport
. Substation Automation DIG Release 2.0 . Extends the previous DIG to on-net substations and teleprotection use cases . Covers the use of ASR 900 product family
9 DIG Rel 1.x DIG Rel 2.x Power Utilities MPLS System Overview Common
Prime Prime Prime Prime NOC AAA DHCP DNS Network Provisioning Performance Optical
OAM Subsystem
On-Net Substation Utility Edge Network Utility Core Network Control / Data Center IEC 61850 T&D Substations T&D Substations Primary Control Center Substation Headend Router HMI RTU MPLS Router Control Center SCADA FEP EMS Station Bus Historian DFR/DC MPLS/IP Substation Ring Aggregation MPLS Router Control Center Analytics MPLS Router Process Bus IED/PMU CPAM Data Center VSOM Multiservice ACS CA On-Net Substation MPLS/IP Core LDAP IEC 61850 MPLS/IP Substation Linear Aggregation MPLS Router RTU Control Center MPLS Router Station Bus Substation Substation DFR/DC MPLS Router IP Router Substation Control / Data Center Process Bus MPLS Router Secondary IED/PMU HMI Control Center SCADA FEP Off-Net Substation EMS Historian IEC 61850 Service Provider Service Provider Analytics MPLS PE MPLS PE RTU CPAM Station Bus Service Provider Data Center Leased Transport VSOM ACS DFR/DC Substation CA IP Router (CE) Utility Control Center LDAP Process Bus CE router Headend Router IED/PMU
Fiber Fiber MPLS/IPoDWDM Optical Network (for Private WAN)
Packet / Hybrid Microwave Copper Utility WAN Edge - Platform Positioning
CGR-2010
. The CGR-2010 is a software switched enterprise branch router, ruggedized to meet substation environment compliance Utility Private WAN TDM Network CGR-2010 . It does not support TDM circuit emulation T1/E1
. Being a software switched platform, it is not suitable for low- Utility Private WAN Packet Switched Network latency use cases or ring topologies involving transit traffic ASR-900 Sub GE, 10GE . It is ideal for off-net substations connected over leased transport requiring VPN/encryption functions like FlexVPN/DMVPN, etc. SONET/SDH Sub Sub . Deploy in spoke topologies over SP leased transport or Sub Sub private TDM transport overlay environments Sub CC CC ASR-900 OC3/12 ASR-903 / ASR-902 / ASR-920 Sub Sub Sub . Position as SDH to MPLS packet transport replacement Sub platform for on-net substations Service Provider . Supports TDM circuit emulation to enable packet migration CGR-2010 Leased Transport GE, T1/E1 . Supports time sensitive low-latency transport use cases Sub . Needs a security appliance or a CGR-2010 to enforce the Electronic Security Perimeter (ESP)
11 ASR 903/902 Latest and Planned IoT Developments
HW / SW Feature SW Release
SW Port RS232 Raw Socket from RSP1 to RSP2 (Serial IM) 3.16 (Aug 2015
SW RS232 Mirrored bits - Teleprotection 3.16 (Aug 2015) HW ASR 902/903/920 Utility Environmental Certification: Nov/Dec 2015 • IEEE 1613 and IEC 61850-3 HW/SW 4w E&M IM - CESoPSN Pseudowire with CAS 3.17 (Nov 2015)
SW CESoPSN Pseudowire (Serial Module): Spring 2016 • X.21 SW Interface protocols (Serial IM): Summer 2016 • V.35, EIA 530, EIA 449 SW RS-485 TCP Raw Socket for RSP2 (Serial IM) Summer 2016
HW/SW C37.94 IM - CESoPSN Pseudowire Fall 2016 ASR 900 Series New 4-wire E&M IM for Utility Applications
• 6 RJ45 ports of E&M analog interfaces
• Software selectable E&M type, impedance, 2w/4w mode
• Each port configurable for 2 or 4-wire operation and signaling types I, II, III and V, supporting various signaling use cases: • SCADA transport using Land Mobile Radio with out of band signaling • SCADA transport using 4-wire "transmission only” • In-band signaling for audio tone-based teleprotection
• Point-to-point, low-latency transport over MPLS pseudowire using CESoPSN
• Compliant with IEC 61850-3, IEEE 1613 Teleprotection Migration to IP
C37.94, G.703 Co-Dir E&M, G.703 Co-Dir, C37.94 Preserving channel-bank E1/T1 E1/T1 CESoPSN or SAToP Pseudowire TPR Relay Line TPR Relay Timing Line Freq. Sync using SyncE Timing Migrate from existing Migrate from existing Channel-Bank to ASR-900 Channel-Bank to ASR-900
E1/T1, Serial, E&M Direct Attachment from legacy relays E1/T1, Serial Sync/Async CESoPSN or SAToP Pseudowire TPR Relay TPR Relay Line Future support for E&M, C37.94 etc. Line Timing Timing
Direct Attachment from IEC 61850 relays Ethernet Ethernet EoMPLS Pseudowire TPR Relay TPR Relay
ESP ESP RTU RTU MPLS/IP MPLS/IP MPLS/IP DC DC CGS-2520 CGS-2520 Transport Transport Transport
DFR IED/PMU IED/PMU DFR IE-2000U Substation Substation IE-2000U Router Router Substation Substation Edge Network Core Network Substation Edge Network Substation Network Requirements for Teleprotection (1/2)
• Network requirements depend on type of protection scheme being supported
• Teleprotection is a very generic term… Not all schemes require low-jitter, path-symmetry etc.
• Circuit Emulation for legacy E1/T1, X.21, E&M, G.703 co-dir relay protection interfaces
• In most cases structured (nxDS0) circuit emulation is needed
• Circuit Emulation for utility specific C37.94 relay protection interface
• IEEE C37.94 standard is structure (nxDS0, where n = 1 to 12) aware
• Ethernet Layer 2 transport for IEC 61850 based protection interfaces
• Network Frequency synchronization when Circuit Emulation is used for transport (SyncE, 1588)
• Path symmetry when “ping-pong” or “echo” based synchronization is used (MPLS-TP or MPLS-TE)
15 Network Requirements for Teleprotection (2/2)
• Low end-to-end latency and Packet Delay Variation (Jitter) across communication path (QoS PQ, TE, de- jitter buffer, etc)
• Communication delay budget is set by protection engineer (based on breaker operation time and relay settings)
• Generally, the higher the voltage class, the smaller the delay budget
• IEC 61850-90-1 suggests values ≤10 msec
• MPLS-TP Fast Recovery (HA)
• MPLS-TP Linear Protection provides simple sub 50ms restoration for Teleprotection services that require symmetric paths
• Protection switching is triggered by in-band OAM.
• Hardware offloaded 3.3ms BFD timers supported for sub 10ms failure detection and sub 50ms recovery.
• Fast Recovery for L2 and L3 services (ex. SCADA), that do not need traffic engineered paths, are protected by R-LFA
• Remote-LFA for Dynamic MPLS/IP LSPs provides fully automated sub 50 msec. restoration without the need for Tunnels
16 Teleprotection Topology – Cisco Lab
T0/1/0 10.25.1.1 T0/0/0 10.25.6.1 ASR-903 W2507 T0/0/0 T0/0/0 10.25.1.0 10.25.6.0 Symmetricom ASR-903 ASR-903 TP-5000 PRC W2501 W2506 T0/1/0 10.25.1.2 10GE T0/1/0 10.25.5.1 T0/1/0 10.25.1.3 • MPLS-TP & MPLS IP 3 T0/1/0 10.25.5.0 ASR-903 ASR-903 Backup Path W2502 TP-Tunnel(s) Anue-2 W2505 T0/0/0 10.25.2.0 • Prime for Network Management 5 Hops T0/0/0 G0/3/0 ASR-903 10.25.4.1 10.24.15.1 G0/0/1 W2504 T0/0/0 • Benchmarking with Ixia IxNetwork 10.24.15.0 T0/0/0 10.25.2.1 T0/1/0 10.25.4.0 ASR-902 10.25.3.1 W2415 • Benchmarking with Relays T0/1/0 ASR-903 G0/4/0 G0/0/0 1GE 10.25.3.0 10.24.13.1 Siemens 10.24.14.1 W2503 • Operation under full traffic load TPR Relay 2 G0/0/1 2 10.24.13.3 Anue-1 Failure • Operation under link failure condition G0/0/0 G0/0/0 10.24.14.0 ASR-902 G0/0/1 10.24.13.0 10.24.13.2 1 • SyncE for CESoPSN W2414 ASR-902 1 Hop W2413 TP- Tunnel(s)
Siemens TPR Relay 1 17 SIEMENS SIPROTEC-4 Differential Relay Details
. 7SD5 Multi-end differential & distance protection relay Connectivity Diagram Siemens SIPROTEC-4 . Quantity x2 7SD5231 Relays . Model: 7SD5231-5AB99-9CJ1 +L0R +M0A +N0A Siemens 7XV5662-0AD00 . Two protection Interfaces T1/E1 Serial Converter . Communications converter for serial data to E1 format . Quantity x2 . Model: 7XV5662-0AD00 . GPS-time synchronization unit . Quantity x1 . Model: 7XV5664-0AA00 . Opto-electrical time synchronization converter Siemens 7XV5664-0AA00 GPS Receiver Siemens 7XV5654-0BA00 . Quantity x1 Opto-electrical Sync Converter . Model: 7XV5654-0BA00 . Y-Bus cable for the time synchronization bus . Quantity x2 . Model: 7XV5105-0AA10
18 Teleprotection Test Results
1 0.07ms with Relays Connected Back-to-Back
G.703 E1 8xDS0 = 512kbps
2 1.82ms Across 1-hop Primary Path 87L / CDP Relay 0.035ms 1.75ms 0.035ms
* Latency delta between 1-hop CESoPSN Pseudowire Relay Protection Interface Unit and 10-hops is only 130usec MPLS PSN due to ASR-903 centralized architecture and Cisco low- 1.95ms Across 10-hop Backup Path latency ASIC MPLS Router 3 0.035ms 1.88ms 0.035ms ** Add 1msec propagation delay (speed of light through fiber CESoPSN Pseudowire optic) for every 200km between substations MPLS PSN
19 IEC Standards for Energy Automation - status and outlook IEC 61850 - Communication networks and systems for power utility automation Scope and Objectives
. Interoperability (between devices and systems) . Free configuration (free allocation of functions to devices) . Long term stability (layered, object-model based design) . Extensibility (into new domains or even other IoT verticals)
Source: IEC TC57 . Btw., IEC 61850 has left the substation!
21 IEC 61850: Content & Work Items
. Currently, IEC 61850 comprises 25 standards + Technical Reports (TRs)!
. Beside the 3 main building blocks, IEC 61850 also contains: . Conformance testing requirements . Safety and environmental requirements . Technical reports for specific use cases
. Interoperability Testing in UCA (2015: Model, SCL, Ed2, PRP/HSR | 2017: WAN/MPLS)
. Current Work (important for Cisco): . TR IEC 61850-90-12 (WAN Engineering Guidelines) . TR IEC 61850-90-4 (Substation Engineering Guidelines) . IEC 61850-9-3 (Precision Time Protocol Profile for Power Utility Automation) . IPv6 Task Migration Guidelines
22 IEC 61850: The Protocol Stack
. Network Architecture for the Substation LAN is defined in IEC TR 61850-90-4 (VLANs, RSTP, PTP)
. It contains also PRP/HSR references to IEC 62439-3:2012 for seamless redundancy and recovery including Red Boxes
. Specifies Reference Topologies based on PRP/HSR
Note: Protocol Stack extended by XMPP profile
Source: IEC TC57 23 IEC 61850: Within the Substation Station and Process Bus
. The Station Bus connects entire substation; provides connectivity between central management and individual bays
Connects IEDs within a bay, connects bays, bays with the gateway/gateway router
Provides only soft real-time quality of service (except for bus bar protection) . The Process Bus connects the primary measurement and control equipment to the IEDs
Limited to a bay
Expected to provide hard real-time quality of service
Source: IEC TC57 24 IEC 61850: Flexible Communication Platform The 3+1 IEC 61850 mappings serve a broad variety of use cases
Message types and classes with strict performance requirements for critical use cases: . Protection . Control . SCADA
The new Routable Profile for GOOSE/SV based on IP MC provides scalability!
(New) XMPP Use Case Domains: Micro Grids, DER, Smart Customers, EV Charging Spots, Feeder Automation, etc.
25 Security in Power Automation Regulations and Standards in the Utility World
26 IEC 61850 Substation Automation - latest station bus design Substation Automation - Reference Model IEC 61850 Substation LANs
Control Center Control Center Wide Area Network Wide Area Network WAN
Station Level
Bay Level Substation Control Room Substation Protection & Control
Process Substation Primary Equipment Level Process Level IEC 61850 Substation Automation Main Design Topics
. High Availability Core . Topology Connectivity . Timing Design Topics
. Network Segmentation
…please do not forget QoS, Security and Management!
29 Substation Automation HA
Utility Private Substation Physical MPLS/IP WAN Security Perimeter (PSP) Substation Network NERC-CIP Electronic Substation Industrial Security Perimeter (ESP) MPLS Router Security Appliance
Private WiMax or LTE Legacy Serial, Sync/Async, to Field Area Network C37.94, E&M Distributed FAN Aggregation HMI HMI Controller Legacy RTU IEC 61850 Multiservice Bus Station Bus
Legacy Comm Bay PMU Teleprotection Protection Teleprotection RTU Processor Controller Relay Relay Relay PDC
IEC 61850 Process Bus
Breaker Physical Hardwired MU IED Security I/O Workforce Enablement PT CT Breaker Sensor PT CT Breaker PT CT Substation Automation HA IT type of applications and buying center Tree topology w/ RSTP more common REP also used with ring topologies
Utility Private Substation Physical MPLS/IP WAN Security Perimeter (PSP) Substation Network NERC-CIP Electronic Substation Industrial Security Perimeter (ESP) MPLS Router Security Appliance
Private WiMax or LTE Legacy Serial, Sync/Async, to Field Area Network C37.94, E&M Distributed FAN Aggregation HMI HMI Controller Legacy RTU IEC 61850 Multiservice Bus Station Bus
Legacy Comm Bay PMU Teleprotection Protection Teleprotection RTU Processor Controller Relay Relay Relay PDC
IEC 61850 Process Bus
Breaker Physical Hardwired MU IED Security I/O Workforce Enablement PT CT Breaker Sensor PT CT Breaker PT CT IEC 61850 Substation Automation (MMS, Goose, SV, 1588) Substation Automation HA Automation buying decision and mindset Ring topologies w/ REP more common in initial deployments PRP seeing strong adoption for “full” Station and Process Bus
Utility Private Substation Physical MPLS/IP WAN Security Perimeter (PSP) Substation Network NERC-CIP Electronic Substation Industrial Security Perimeter (ESP) MPLS Router Security Appliance
Private WiMax or LTE Legacy Serial, Sync/Async, to Field Area Network C37.94, E&M Distributed FAN Aggregation HMI HMI Controller Legacy RTU IEC 61850 Multiservice Bus Station Bus
Legacy Comm Bay PMU Teleprotection Protection Teleprotection RTU Processor Controller Relay Relay Relay PDC
IEC 61850 Process Bus
Breaker Physical Hardwired MU IED Security I/O Workforce Enablement PT CT Breaker Sensor PT CT Breaker PT CT Substation Automation HA Recovery time examples in IEC 61850-5 (Communication Requirements) => HA Protocol to be used
Use Cases Locale Network Recovery Time SCADA to IED, client-server Station bus 400 ms
IED to IED interlocking Station bus 4 ms
IED to IED reverse blocking Station bus 4 ms
Bus bar protection Station bus 0 ms
Sampled values Process Bus 0 ms
2 Substation Automation HA Recovery time examples in IEC 61850-5 (Communication Requirements) => HA Protocol to be used
Use Cases Locale Network Recovery Time SCADA to IED, client-server Station bus 400 ms
IED to IED interlocking• REP or RSTPStation can not bus meet HA 4 ms requirements for “full” Station Bus IED to IED reverse• Ciscoblocking does notStation currently bus support 4HSR ms • PRP is the only option… But also the Bus bar protection best technicalStation one! bus 0 ms
Sampled values Process Bus 0 ms
2 PRP Overview
• Parallel Redundancy Protocol: IEC 62439-3 Clause 4 • Two versions so far: PRP-0 (2010) and PRP-1 (2012) and they are not compatible • Two independent LANs must exist (any topology) • Two copies of each packet are delivered over these LANs
• Main PRP benefits: Zero packet loss when single LAN fail with support for any network topology • Main PRP limitation: Double of network components and cost
35 IEC 61850 Station Bus and PRP Platform Requirements
Forward Jumbo (PRP) Frames Cisco Catalyst, IE, CGS, etc Environment: IEC 61850-3 CGS 2520 and all IE switches PTP: 1588v2 Power Profile (and more…) IE5k, IE4k, IE 2kU, CGS 2520 Station Level Ports: GE Copper and Fiber IE5k, IE4k Bay Level Ports: FE Copper or Fiber IE4k, IE2kU, CGS 2520
36 IEC 61850 Station Bus and PRP Platform Requirements
Forward Jumbo (PRP) Frames Cisco Catalyst, IE, CGS, etc Environment: IEC 61850-3 CGS 2520 and all IE switches PTP: 1588v2 Power Profile (and more…) IE5k, IE4k, IE 2kU, CGS 2520 PRP RedBox Support (Station Level): IE5k, IE4k, or …IE2kU
37 PRP LAN A&B Topology (1/2) IE 5000 Update
Considerations Ring-based Topology Tree-based Topology (IE5000) More robust Fault Tolerance Less robust (fault is isolated to just the affected branch (faulty switch / link can affect the entire ring) (including maintenance) / switch) Availability Variable MTBF as the number of switches Fewer and fixed number of switches in the (service level) vary switching path results in a higher MTBF 50 ms to 250 ms (media, ring size, etc). 0ms 100 ms to 1s typical Convergence with PRP… 0ms with PRP… Less deterministic latency (traffic changing Usually lower latency (less hops). Remains Latency direction around the ring during failover) more constant even in large topologies PRP LAN A&B Topology (2/2) IE 5000 Update
Considerations Ring-based Topology Tree-based Topology (IE5000) Highly scalable – 20+ switches per ring Scalability – Number of validated for sub-50 ms failover. Larger Now very scalable with IE5k (specially nodes and distance topologies supported in a single ring or with with future horizontal stack features) nested rings Number of nodes on the ring determines Greater bandwidth per node (key in Scalability – Bandwidth available bandwidth between switches Process Bus) All inter-switch traffic contends for the ring All inter-switch traffic contends at limited QoS – Predictability / bandwidth. Traffic sent by the edge and typically fewer points in the Tree switches has to compete with similar class fairness topology of traffic at every hop on the ring
Fiber Investment Usually less fiber cables / length Usually more fiber cables / length PRP LAN A&B Topology (2/2) IE 5000 Update
Considerations Ring-based Topology Tree-based Topology (IE5000) Highly scalable – 20+ switches per ring Scalability – Number of validated for sub-50 ms failover. Larger Now very scalable with IE5k (specially nodes and distance• In PRPtopologies networks, supported with in a singleIE 5000 ring or atwith the withstation future horizontal stack features) level, tree-basednested topology rings offers the best Number of nodes on the ring determines Greater bandwidth per node (key in Scalability – Bandwidth technicalavailable solution bandwidth between switches Process Bus) • Ring-basedAll inter-switch topologies traffic contends may for stillthe ring be preferred All inter-switch traffic contends at limited QoS – Predictability / bandwidth. Traffic sent by the edge due to cost, installed base or mindsetand typically fewer points in the Tree switches has to compete with similar class fairness topology of traffic at every hop on the ring
Fiber Investment Usually less fiber cables / length Usually more fiber cables / length Power Utilities Timing Requirements …and typical deployment models
Dedicated IRIG-B Cables: • General Applications (<1msec) GPS Distance Limitations, Antenn Cost, Flexibility, etc • Sequence of Events a • Digital Fault Recorder (DFR) Distributed IRIG-B Controller • High Precision Timing (<10usec) Source • Synchrophasors (C37.118) • Sample Values (IEC 61850-9-2) Station Bus • Distributed DFR Events RTU DFR • IEC 61850-5-2003 (1usec to 1msec) IED PMU IED PMU • Class T1: Events = ±1msec • Class T5: Samples Values ±1usec Process Bus
41 Why IEEE 1588?
. IEC 61850 Edition 2 makes reference to IEEE 1588v2 Power Profile
. Precision Time Protocol (PTP) IEEE 1588v2 was developed with the following aims: Synchronization accuracy in the sub-microsecond range Minimum requirements of the processor performance and network bandwidth Low administration effort Use via Ethernet networks Specification as an international standard
Power Profile, as defined in IEEE C37.238: . Layer 2 (Ethernet) Multicast . 1usec over 16 hops . Peer-To-Peer Delay Measurements
42 Utility IEEE 1588 PTP Requirements …and new deployment model
Migration to IEEE 1588v2Dedicated Power IRIG Profile-B Cables & Distance Limitations
• IEC 61850-90-4 (Substation Engineering Guidelines) Transparent • IEEE C37.238 Clock (IEEE 1588 PTP in Power System Applications)
43 Why you don’t want to rely on GNSS only Global Navigation Satellite System (GNSS) – aka GPS, COMPASS, Galileo, …
• Reasons for using GPS • nearly available everywhere • A GPS disciplined oscillator can provide time accurate within 100ns
• Reasons for not using GPS • see statement on www.pnt.gov, from Nov 3rd ,2010 “GPS should not be used as the unique reference in any critical civilian system” • Reliability (very weak satellite signal) • Attacks (jamming and spoofing) • Cost of installation • Local Distribution (Splitters, Amplifiers, …) GPS Jammer Handheld
44 Current Synchronization Architecture Hybrid Mode: SyncE + End-to-End IEEE1588
PTP IEEE1588-2011 C37.238 Power Profile PTP Hybrid Mode 1EEE1588-2008 + SyncE
Packet Master GPS GPS Antenna • GPS: Primary I/P • 10Mhz/1PPS/ToD: Backup I/P SyncE Packet Source-1 Master-1 SyncE/ESMC SyncE/ESMC GPS (Frequency) (Frequency) Antenna 1588v2 BC (Freq, Phase/ToD) 1588v2 1588v2 Master 1588v2 Master 10Mhz (Freq, Phase/ToD) 1PPS 1588v2 ToD 1588v2 (Freq, Phase/ToD) (Freq, Phase/ToD) 10Mhz P2P Transparent BC BC Clock TC 10Mhz PRC MPLS/IP 1588v2 Network (Freq, Phase/ToD) 10Mhz IED/PMU 1PPS MPLS/IP Station/Process TC ToD TC Network 1588v2 SL Bus Master BC TC Hybrid Boundary Clock Hybrid Boundary Clock Packet Slave • SyncE :Freq • SyncE :Freq SyncE Master-2 Clock • 1588v2: Phase/ToD • 1588v2: Phase/ToD Source-2 SL SL
IED/PMU IED/PMU Multiservice Bus PTP Master CGS-2520 PTP Master
MPLS/IP MPLS/IP Station / Process Aggregation Bus Sub-CE ESP Core CGR-2010 Sub-PE CGS-2520 ASR-903, ME3600 45PRC Substation Aggregation Network Core Network IE 5000 Timing Features Value proposition game changer!
Shipping or Committed:
• NTP to 1588v2 PTP GMC
• IRIG-B Interface
• GPS to IEEE 1588v2 (Power Profile)
Other features In Radar:
• 1588 PTP redundancy in PRP (Annex A)
• 1588 Telecom to Power Profile
• Support for Galileo and other GNSS
46 Cisco Switching Portfolio for IEC 61850
Aggregation Access Best in Class Cisco IE 4000 Cisco IE 5000 Series • IEC 61850 in Station Bus is Cisco mature in EMEA Cisco IE 2000U CGS-2520
• Cisco has a leading
Industrial Switching portfolio . Layer 2 or 3 . Layer 2 and 3 (IP services) . Designed for all . Up to 8 PoE/PoE+ . Designed for all . Up to 12 PoE/PoE+ to address IEC 61850 (IP services) . 1RU industries . Dying Gasp industries . Dying Gasp . Small Form Factor . 2 GE combo uplinks . Layer 2 or 3 . TrustSec® SGT . Layer 2 or 3 . TrustSec® SGT . PRP . 8 PoE and 16 SFP or (IP services) HW ready (IP services) HW ready . IEEE 1588 PTP 24 copper . 4-port GE uplinks . MACsec . 4-port 10GE or . MACsec HW ready • Visit the whisper suite for & Power Profile . Power profile . Up to 20 ports GE . FNF HW ready GE uplinks . FNF hardware ready . IEEE 1588 PTP & . Time Sensitive . 24 ports GE . Time Sensitive . PoE/PoE+ (CGS2520) power profile Network (TSN) . IEEE 1588 PTP & Network (TSN) HW . PoE/PoE+ more good news… . Layer 2 NAT HW ready power profile ready . Layer 2 NAT • …and don’t forget security!
10 Gbps 1 Gbps 10/100 Mbps
47 Distributed Energy Resources - a new distribution grid model Field Area Network – Top Use Cases in EMEA
Use Case Use Case Title Distribution Automation (DA) – Grid Visibility and Control: 1 •Supervisory Control and Data Acquisition (SCADA) •Remote Asset Monitoring Advanced Metering Infrastructure (AMI) – Energy Efficiency: 2 •OPEX Reduction: Remote meter reading, Connect/Disconnect - Pre-payment, Demand Response •Customer Service: Power Outage / Restoration reporting, customer portals for usage data Distributed Energy Resources – Grid Efficiency, Renewable Resources: 3 •Integrated Volt / Var Control (IVVC) •Anti-Islanding; Peak shaving; Inject Energy surplus; Energy Storage Distribution Automation (DA) – Grid Reliability / Quality of Service: 4 •Fault Location, Isolation and Restoration (FLIR) or •Self Healing Feeder Network Remote Workforce Enablement – IT/OT Convergence: 5 •Multiple Devices, Physical Security, Secure Remote Access •Remote Experts and Applications, Emergency Voice 49 Cisco Multiservice (Secure) Field Area Network
AMI Operations FAN Operations DA Operations
Scada, DMS, Meter Data Collection ASR 1000 GIS, OMS & Management Public or Private WAN Backhaul
Substation (3G, WiMax, Fiber) WAN Tier
CGR 1000 IR 800
Ethernet NAN Networks Tier NAN RF Mesh, PLC or LoRa Work Force Enablement IR 809
AMI / HAN Transformer Distribution EV Charging Street Gas / Water Distributed SCADA, Direct Cellular Monitoring Automation Lighting Meters Resources Protection & Connected Assets Control 50 Cisco FAN Architecture
1. Flexible and Future Proof
2. Secure and Scalable
3. Lower TCO
4. New Business and Operational Models
51 Flexible, Enabling Multiple Applications and Devices
Proprietary protocols Standard protocols over Standards protocols IEC 61850 over Serial Serial over TCP/IP
MMS IEC 101, DNP3, Modbus, GOOSE/SV Vendor’s dependent DLMS/COSEM, etc IEC 104, DLMS/COSEM, Modbus/TCP, Web Svc, etc GOOSE/SV over IP/UDP future IEC 8-1 and 9-2 profiles
IP Interfaces Ethernet L2 switching Traffic tunneled over IP Raw Protocol Translation GW: • IEC 101 to IEC 104 Socket (TCP & UDP) Ethernet, Fiber • DNP3 to DNP3/IP Serial PPP/CHAP L2 over IP WAN and Copper (L2TPv3 or EoMPLS)
Secure & Reliable IP infrastructure VLAN; 802.1X; VRF; QoS; FlexVPN & DMVPN; FW; IPv4 & IPv6 dual stack; Multicast; EEM; IOX DER with L2 Goose over L3 IPSec WAN
IPSec L2 Goose L2TPv3 L2 Goose FlexVPN
Secondary 3G / 4G Substation Primary Substation IEC 61850 IED Cisco FAN Cisco Grid IEC 61850 IED Router Router
Ethernet Cellular Ethernet Cisco FAN Architecture
1. Flexible and Future Proof
2. Secure and Scalable
3. Lower TCO
4. New Business and Operational Models
54 IoT Cyber Security Principles
Access Control
• User and Device Identity PolicyManagement withOT/IT • Authentication, Authorization & Accounting Convergence & Ease of Use Availability and Safety Data Confidentiality and Data Privacy • Network Segmentation • Secure Connectivity Integrity Threat Detection and Mitigation • Security Zones • Intrusion Prevention; Application Visibility Confidentiality Device and Platform Integrity • Device Hardening and Secure Platform • Configuration Assurance Data Confidentiality and Integrity in the FAN Scalability Challenge
• Many thousands of field remote sites (maybe +100.000), spread over a country wide geography, that need to communicate over a public network (3G/4G most of the times) • Availability, Integrity and Confidentiality (IPSec VPN) are key requirements for this critical infrastructure
• How to achieve end to end segmentation and segregation with different SLA? • How to aggregate all these IPSec tunnels? • How to provision 100.000 sites and IPSec tunnels (IP, Certificates, AAA, etc)? • How to guarantee the Scalability and High Availability of the complete solution? • Needs to be Standards based with proven Interoperability!
• A Secure and Scalable IP Solution is one of the key value points from Cisco! Field Area Network Operation Center
CG-NMS: Network & Security Management: supports browser based clients, interface with ASR 1K, CGR 1K and End Points Metering & Data SCADA CG-NMS DB (Oracle) Stores all operational state, device configuration, network event alarm, performance metric, etc
Cisco FND AAA Server Meter Data SIEM OMSOMS DMSDMS GISDMS Management
NTP Appliance: acts as AAA Server: scalable, high-performance policy precision timing source system for authentication, user access, and administrator access; ECC for meters Directory Certificate Services Authority IPAM Active Directory(AD) & Certificate DHCP Authority (CA): for user & device identity ASR 1K management along with CA for certificate IPAM, DHCPv6 and DNS: IPv4/IPv6 address management Supports Cryptography: ECC allocation and naming: scale up to +10M keys for certificate-based authentication endpoints Public or Private IP WAN Firewall + IPS Appliance: primary IPv4 / IPv6 firewall for securing the head-end Adv. Scalable Routing infrastructure; optional use of IPS FlexVPN, IKEv2 module Application Visibility
DA devices (Ethernet / Serial) NAN (RF Mesh & PLC) Cisco FAN Architecture
1. Flexible and Future Proof
2. Secure and Scalable
3. Lower TCO
4. New Business and Operational Models
58 Field Network Director
. Scalable to Millions of Devices . via CoAP / CSMP
. Secure Zero Touch Deployment
. Inventory & Asset Visualization . via Google Maps & others
. Performance Management . Backhaul & access proactive management
. Fault and Outage Management . Collect & process alarm events from selected routers and endpoints
. Cyber Security Policy Compliance
. North Bound API . MDM, SIEM, SCADA, OMS, etc Ease of use to reduce TCO Wireless backhaul monitoring Cisco FAN Architecture
1. Flexible and Future Proof
2. Secure and Scalable
3. Lower TCO
4. New Business and Operational Models
61 Cisco IoT systems enables Applications to run closer to edge
VERTICAL SOLUTIONS APPLICATIONS ECOSYSTEM
Transportation City Oil and Gas Defense Manufacturing Utility Service Public Safety Provider
Application Enablement Security Management and Fog Services Automation
IoT Connectivity Why Fog Computing?
Traditional Computing Model IoT Computing Model (Terminal/Mainframe, Client-Server, Web)
Data Center/ Data Center/ Cloud Cloud
Efficient use of Assumes Limited Bandwidth, Variable Delay, and Intermittent Resources Connectivity
Assumes Infinite Resiliency BW, CPU & Storage. Fog 0 Delay Latency-Critical Assumes Limited Bandwidth, Variable Delay, and Intermittent Data Grows Faster Connectivity than Bandwidth Endpoint Device Cisco Industrial Routers IOX (IOS + Guest OS)
IOS (T/M Train) Guest OS
Hypervisor
IOT use cases: Linux Guest OS-enabled third-party . Asset Management applications independently from IOS . Application gateway / Protocol translation . Software in guest OS can be upgraded independently of Cisco IOS . Application data processing . Distributed control rd . Application layer security Cisco FND can manage OS and 3 party apps install/uninstall Cisco 809 Industrial Integrated Services Routers
Cellular MAIN GPS Cellular AUX
Dimensions (DxWxH) : 12.7x 15.9x 3.2 cm Temperature: -40C to +60C
Accelerometer and Gyroscope
One USB Type B Port
One RJ-45 RS232 Serial Port One RJ-45 RS232/RS485 Serial Port 9-54 VDC Power Input Digital Alarm Ports Two 10/100/1000 Base-T One USB 2.0 Type A port Long Range Backhaul LPWA Approach LPWA P Connecting Unconnectedthe PLC RF SmartGrid Mesh WiFi Other 2G 3G WiHart ISA 100 Industrial WiFi Priv P Pub/ Outdoor Outdoor WiFi Serial Cellular Backhaul SmartCity RF Mesh WiFi Priv Business Case at a Glance:at CaseBusiness Pub/ outdoor outdoor Long Range without Batteries! (@10mW = GSM 1W)(@10mWBatteries! = without Range Long Network and Cost Chipset Low Very bands frequency Unlicensed Technology, Open Location verticals, formultiple ecosystem Big LoRa • • • • WiFi RF Mesh Smart Buildings Smart Wireless Technologies Landscape Technologies Wireless –
LE -
B
Range
Range Range
Wearables
Short Medium Long IoT IoT Backhaul Conclusion / Key Takeaways Conclusions / Key Takeaways
• Cisco has mature IP solutions, validated with other Industry leaders, that answer to Power Utilities’ most demanding applications: o Current Differential Protection o IEC 61850 Substation Automation o Distributed Energy Resources in scale
• Cisco continues to innovate with new products (ex. IE 5000, IR809) and solutions (ex. LoRa, Data Analytics with IOX) in order to increase Cisco value proposition to Power Utilities
• We’re ready. Are you?
68 Call to Action
• Visit the World of Solutions. Look for IoT and Energy in particular
• Meet the Engineer
• Lunch and Learn Topics
• DevNet zone related sessions Complete Your Online Session Evaluation
• Please complete your online session evaluations after each session. Complete 4 session evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt.
• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations Thank you