BRKIOT-2111.Pdf

Home , IEC 60870, IEC 61850, IEC 62056, Rolling blackout

IoT in Substation and Energy Automation

BRKIOT-2111

Paulo Pereira, Consulting Systems Engineer Internet of Things, Europe Agenda

• Introduction • Cisco Architectural Approach • IEC 61850 Technical Overview • Fundamental Architectural Design Elements • Security Standards for Energy • Cisco IoT Portfolio for Energy Automation • Conclusion / Key Take Aways

• Many utilities looked this way in most of the 20th century. Regulation has changed this model considerably in most countries…

• Most of World Grid is AC (50Hz Alternating Current in EMEA = 20ms full cycle)

• Very Important to maintain frequency within very tight limits as rolling blackout may occur

• No significant energy storage in the grid => Power must be kept in balance (generation follows load)

Step-up Transformer 13.8 kV / Transmission voltage level

Transmission (220, 345, 500, 765 kV) Generation Station 13.8 kV Sub-transmission Switching Stations/ (161, 115, 69 kV) Transmission Substation Distribution Feeders 3-phase (7, 11, 33 kV) Distribution Substations Sub-transmission voltage level Generation Station 13.8 kV Service Transformers 7 kV to 120/240 V Secondary 120/240 V

• Monitor • Measure • Control • Automation • Protection

Electrical Grid Drivers Potential Network Impact (System Control Tier) Evolution from Centralized Generation to Distributed New locations requiring communications, use cases for Wide Area Generation and Storage Measurement and Control. Machine builders (Wind Towers). Industry and Security Compliance Strong, open standards based physical and cyber security: Video Cameras, Access Control, local storage; as well as Firewall, IPS/IDS, Encryption VPN, SIEM, Security Management.

TDM End of Life RFPs for TDM migration to IP/MPLS or MPLS TP with Utility specific requirements (ex. interface module) Aging infrastructure, inefficient assets (some 40+ Use cases for Condition Based Maintenance, substation years). Optimization and more efficient Infrastructures expansion and upgrades to IEC 61850 / Ethernet / IP New Generation Workforce / OPEX reductions Collaboration tools for remote expert support, reduced truck rolls, remote access into / out of the Substation Distributed Intelligence / FOG computing Opportunity to host distributed Utility applications on network (driven by more distributed nature of electrical grid) platforms (Ex: OSI Soft, Substation Gateway, Security)

Field Area NG WAN Field Workforce HV Substation Network Enablement

Load DMS MDMS Control SCADA DMS DRMS MDM Secondary CC Cisco Connected CG-NMS AMI HES Grid Security and Head-end CG-NMS Network Management

Substa on 2G/3G/LTE W GPRS, 3G, WIMAX / Prvt A substa on

N LTE LTE

Ethernet, Cisco Connected CGR1000 WiMAX Grid Router 1000 Cisco 1000 Series Connected Grid Router

Series

Cisco Connected e IEEE 802.15.4 sub-GHz RF Mesh

Grid Endpoint N

RF and PLC Mesh Protec on and N Neighborhood Area Network Control Networks Work Force Automa on

Residen al Metering

AMI Transformer Distribu on EV Charging Direct LoadO utdoor Gas / Water Distributed SCADA Protec on Direct Connect Metering / Monitoring Automa on Infrastructure Control Ligh ng Meters Genera on and Control AMI Meters HAN Gateway Network Business Cost Reduction, Operating Efficiency Workforce Productivity Operating Efficiency Outcome New Business Model

. Reduced energy theft . Upgrade legacy . Data Transfers in Seconds . Increase visibility and . Reduced downtime SCADA systems . Handles Multiple Wireless proactive maintenance . Zero touch deployment . Converged, Multi Laptops, Smartphones, Tablets . Scale network to support . Converged Multiservice services network Simultaneously growing number of devices Key Networks (cost) . Integration of TDM . Machine-to-Machine Capabilities . Rapid Fault Isolation and legacy services Communications for Background, . NERC/CIP Security and (uptime) . Follow more recent Next Step Tasks while Field Industry compliance . Scalability, Security and Industry standards Crews are Working . IEC 61850 Interoperability

The Cisco SA System covers the Substation Network, Wide Area Network, & relevant components and applications in the Control / Data Center.

Use Case SCADA (DNP3, Modbus, T101) serial tunneling with Raw Sockets

SCADA (DNP3, Modbus, T101) transport over E&M LMR

SCADA (DNP3-IP, Modbus-TCP & T104) IP transport

Wide Area Measurement Systems (WAMS) with C37.118.2

Monitoring, Wide Area Measurement Systems (WAMS) with IEC 61850-90-5 Control, Automation, IEC 61850 GOOSE messaging for Feeder Protection over Station Bus and IEC 61850 SV messaging with Merging Units over Process Bus Protection Traditional Teleprotection (Current Differential) with legacy interfaces

IEC 61850 Teleprotection (Current Differential) with Ethernet interfaces

System Integrity Protection Schemes (SIPS)

Wide Area Measurement Protection and Control (WAMPAC)

GridBlocks Application UseCase Architectures

VER Output Management

Renewable Generation

Wide Area Monitoring

Stabilization Actors/ Components Benefits Dynamic Line Rating

FISR / FDCL

Distribution Level Tele-Protection Actors/ Components Voltage Regulation Benefits

Advanced Meter Reading

Direct load Control

DG – Voltage Ride Actors/ Components Through Benefits

• Dedicated solution validation lab for substation automation

• Designed to support current and future real-world Power Utilities use cases

• Lab consists of complete end-to-end utility SA network: NOC, substations, DMZ, WAN

• End-to-end validation with RTU, Relays, IED, PMU etc (ex. Siemens and Alstom).

• Test validation results documented in SA Design and Implementation Guide

Proprietary protocols Standard protocols over Standards protocols IEC 61850 over Serial Serial over TCP/IP

IEC 60870-5-101 MMS Vendor’s dependent DNP3, Modbus, etc IEC 60870-5-104, DNP3/IP, GOOSE/SV Modbus/TCP, etc

IEC 60870-5-101 GOOSE/SV over IP/UDP DNP3 future IEC 8-1 and 9-2 profiles

IP Interfaces Protocol Translation Ethernet Layer-2 switching Traffic tunneled over IP Raw • IEC 60870-5-101 to IEC 60870- Socket (TCP and UDP) 5-104 Serial L2 over IP WAN • DNP3 to DNP3/IP Ethernet PPP/CHAP (L2TPv3 or EoMPLS)

Secure IP infrastructure (Data Integrity, Confidentiality and Privacy)

SCADA Server

RS232 or RS485 PSTN RTUs Infrastructure

Application communicates to COM ports

SCADA Server

Ethernet or Serial connection RS232 or RS485 IP RTUs Infrastructure

Raw Socket Scenarios

BRKIOT-2111 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public Teleprotection Migration to IP E&M, C37.94 E&M, C37.94 Preserving channel-bank E1/T1 E1/T1 CESoPSN or SAToP Pseudowire TPR Relay TPR Relay

Migrate from existing Migrate from existing Legacy to ASR-900 Legacy to ASR-900

E1/T1, Serial Direct Attachment from legacy relays E1/T1, Serial CESoPSN or SAToP Pseudowire TPR Relay TPR Relay

Direct Attachment from IEC 61850 relays Ethernet Ethernet EoMPLS Pseudowire TPR Relay TPR Relay

ESP ESP RTU RTU MPLS/IP MPLS/IP MPLS/IP DC DC CGS-2520 CGS-2520 Transport Transport Transport

DFR IED/PMU IED/PMU DFR IE-2000U Substation Substation IE-2000U Router Router Substation Substation Edge Network Core Network Substation Edge Network Substation

• The primary MPLS label switched 1.82 msec across 1-hop along primary path path traverses a direct link between the two ASR-903s 0.035 msec 1.75 msec 0.035 msec

Siemens Siemens • The backup MPLS label switched Siemens Siemens Serial CESoPSN Pseudowire Serial Relay-1 Relay-2 path traverses 10 ASR-903 routers Convert Convert ASR-903 ASR-903 • Latency delta between 1-hop and 512kbps 512kbps 512kbps 512kbps optical 8xDS0 8xDS0 optical 10-hops is only 130usec due to ASR-903 centralized architecture and Cisco low-latency ASIC 1.95 msec across 10-hop along backup path

0.035 msec 1.88 msec 0.035 msec Note: Latency numbers reflected here Siemens Siemens do not account for distances between Siemens Siemens Serial CESoPSN Pseudowire Serial Relay-1 Relay-2 substations. Add 1msec propagation Convert Convert delay (speed of light through fiber optic) ASR-903 ASR-903 for every 200km between substations 512kbps 512kbps 512kbps 512kbps optical 8xDS0 8xDS0 optical

Control Center Wide Area Wide Area Network WAN Network

Station Level

Bay Level

Substation Control Room Substation Protection & Control Process Level Substation Primary Equipment Process Level

Master Control Control Centre

•

HMI / Station • Equipment •

Protection Protection & Control & Control

Substation B Primary Equipment Substation A

Master Control Control Centre

•

HMI / Station • Equipment

Protection Protection & Control & Control

Substation B Primary Equipment Substation A

Number Title Published

3 General requirements IS 5 Communication requirements for functions and device models IS 7-1 Basic communication structure – Principles and models IS 7-2 Basic communication structure – Abstract communication service interface (ACSI) IS 7-3 Basic communication structure – Common data classes IS 7-4 Basic communication structure – Compatible logical node classes and data classes IS

8-1 Specific communication service mapping (SCSM) – Mappings to MMS (ISO/IEC 9506-1 and ISO/IEC 9506-2) and to IS ISO/IEC 8802-3 9-1 Specific communication service mapping (SCSM) – sampled values over serial unidirectional multidrop point to point link IS

9-2 Specific communication service mapping (SCSM) – sampled values over ISO/IEC 8802-3 IS

10 Conformance testing IS

90-1 Using IEC 61850 for the communication between substations TR

90-2 Using IEC 61850 for the communication between substations and control centers TR

90-4 Network engineering guidelines TR

90-5 Using IEC 61850 to transmit synchrophasor information according IEEE C37.118 TR

90-12 Wide Area Network Engineering Guidelines TR

Message types and classes with strict performance requirements for critical use cases:

. Protection . Control . SCADA

The new Routable Profile for GOOSE/SV based on IP MC provides scalability.

IP MC transport helps to address new domains and use cases such as Wide Area Monitoring and Protection as well as Demand Response.

Client Server Communication based on TCP/IP – Not as time critical • Typical SCADA application like control of switchgear or transmission of events (Reporting) • Store and retrieve sequence of events (Log) • Transfer of files

SCADA, GW IED Request Client Server Data Response Application Application

• • • • Short information; low probability of loss; a few milliseconds

Device Device Publish Publisher Subscriber GOOSE/SV Data via multicast Application Application

• Carry voltage and current samples • This traffic usually flows on the Process Bus but can also flow over the Station (Bus Bar) • High amount of data; a few milliseconds; loss of data needs to be detected

Protection Control

Multicast over Ethernet Binary Merging Unit Synchronisation, monitoring, inputs test, and configuration interfaces

Proprietary Links CT VT

IEC 61850 Profiles – Ed.2

SV Station Station Level Equipment GOOSE MMS Station Bus

Bay Bay Control Metering Bay Protection

Process Bus Instrumental Power Process Switchgear Transformer Transformer

• Title: “Use of IEC 61850 to transmit synchrophasor information according to IEEE C37.118”

• For communication it provides routable profiles for IEC 61850-8-1 GOOSE and IEC 61850-9-2 Sample Values packets

• IP-Multicast based on UDP as well as unicast transmission - IPv4 and IPv6 based profiles - Use of Internet Group Management Protocol, Version 3 (IGMPv3; RFC 3376) for multicast path determination - Specifies Explicit Congestion Notification (ECN) based on RFC 3168 - Quality of Service: Differentiated Services Code Point (DSCP) is used to provide IP priority tagging

• Security - Protocol security: information authentication and integrity (HMAC) are defined as mandatory and confidentiality as optional - An overall security model considers the security definitions in IEC 62351-6:2007 [2] to address end-to-end security. - Key Distribution based on GDOI (RFC 3547) introduces a perfect forward security mechanism

• Title: “Wide Area Network Engineering Guidelines”

• This Technical Report proposes guidelines for wide-area and real-time networks for various IEC 61850-based applications including teleprotection, power system monitoring (WASA, WAMS), operation SCADA, and condition monitoring and diagnosis (CMD)

• The Technical Report addresses substation-to-substation communication and substation to control center communication. Especially, the most critical aspects of IEC 61850 such as protection related data transmission via GOOSE and SVs, and the multicast data transfer of large volumes of sampled values (SV)

• Finally, the Technical Report also considers the high precision clock synchronization and “seamless” guaranteed transport of data across the network under failure conditions.

IoT increases Grid Reliability by truly combining Automation, Communications and Security Substation Automation Traffic Types Reference

Traffic Class Standard / Protocol Usage Transport Locale

MMS: Manufacturing IEC 61850-8.1 Supervisory control and • TCP/IP Unicast • Station Control to IEDs in Station Message real-time data access and Process bus Specification • Control Center to IEDs in Station and Process bus GOOSE: Generic IEC 61850-8.1 Distribution of a user • GOOSE: Ethernet Multicast • Between IEDs in Station bus and Object Oriented defined data sets - Status • R-GOOSE: UDP/IP Process bus Substation Event (breaker position, trip, Multicast • Control Center to IEDs in Station alarms, etc.) Analog bus (counter values, etc.) • Between IEDs in Station bus across substations SV: Sampled Values IEC 61850-9-2 Distribution of time sampled • SV: Ethernet Multicast • Between MU in Process bus and data - measurements, • R-SV: UDP/IP Multicast IEDs in Station bus status, I/O signals etc. • Between NIST and MU in Process bus PTP 1588 IEEE C37.238 Power Profile Time synchronization • Ethernet Multicast • Between 1588 Master and IED IEC 62349-3 Utility Profile slaves in Station and Process bus

SCADA DNP3/IP, Modbus-TCP, IEC Supervisory control and • TCP/IP Unicast • Control center to RTUs, Gateways 60870-5-104 data acquisition for grid in Station bus control Synchrophasor IEEE C37.118.2 Time synchronized • C37.118.2: TCP/IP Unicast • Control Center to IEDs in Station IEC 61850-90-5 sampled data for grid • R-SV: UDP/IP Multicast bus control

Traffic Class Characteristic Interval Frame Size Data Rate Tolerable Sensitivity Latency to PDV

MMS: Client-Server • Configurable reporting Variable, Large Supervisory control: 1-500ms N/A Manufacturing (unsolicited or periodic) 10kbps Message • Configurable polling Data access: Specification GOOSE: Generic Publisher-Subscriber • Event Driven 90-600 bytes 1-200 packets/sec 3-10ms LAN N/A Object Oriented (asynchronous and • Periodic heartbeats: 1- 300 bytes (typical) 4-20ms WAN Substation Event unsolicited) 60 sec • Periodic Analogs: 200ms SV: Sampled Values Publisher-Subscriber • Streaming: 80-256 9-2LE dataset: 126 • 4800 packets/sec @ 3-10ms LAN High (synchronous and samples/cycle – 763 bytes 80 samples/sec for unsolicited) 60Hz • 5-6Mbps / MU

PTP 1588 Master-Slave • Announce: 1 sec 66-86 bytes 3 packets/sec High • Sync: 1 sec • Delay Request: 1 sec SCADA Master-Slave • 2-4 poll/sec Variable 2-5 packets/sec 1sec N/A

Synchrophasor Publisher-Subscriber • Streaming: 80-256 90kbps (8 phasors, 3 20-100ms Medium (Streaming) samples/second analogs, 1 digital signal @ 60 packets/second.

. Topology

. Network Segmentation

. High Availability

. Timing

. Operations and Management

…but also QoS and Security!

Redundant Star Ring

Cisco Switch Catalyst 2955 Level HMI HMI

HMI

Cell/Area Zone

Device-level Topologies

Device Level

Considerations Ring-Based Topology Tree-Based Topology Fault Tolerance Less robust (faulty switch or link can affect the entire More robust (fault is isolated to just the affected ring) branch / switch / link) Availability (solution level) Variable MTBF as the number of switches in the ring Fewer and fixed number of switches in the vary switching path results in a higher MTBF

Convergence 50 ms to 250 ms (fiber, ring size, load balancing, etc) +100 ms typical

Latency Less deterministic latency (because of traffic changing Usually lower latency (less hops). Remains more direction around the ring during failover) constant even in large topologies

Considerations REP - Ring-Based Tree-Based Scalability – Number of Highly scalable – Up to 32 switches per ring validated May be less scalable, depending on specific Nodes and distance for sub-50 ms failover. Larger topologies can be available products (GE vs FE, Fiber vs Copper, supported in a single ring or with nested rings etc).

Scalability - Bandwidth Number of nodes on the ring determines available Greater bandwidth per node. bandwidth between switches

QoS – Predictability / All inter-switch traffic contends for the ring bandwidth. All inter-switch traffic contends at limited and fairness Traffic sent by the edge switches has to compete with typically fewer points in the Tree topology similar class of traffic at every hop on the ring

Fiber Investment Usually less fiber cables / length Usually more fiber cables / length Maintenance and Downtime required to add/ remove a switch to the ring No network downtime required to add a new leaf Serviceability (access) switch

. Topology

. Network Segmentation

. High Availability

. Timing

. Operations and Management

…but also QoS and Security!

. Two equipment categories managed by different groups

. Grid monitoring, protection, control and automation devices (RTUs, Relays, IEDs, etc.) - Managed by OT department

. Infrastructure support devices (Cameras, badge readers, Phones, PC, etc.) - Managed by IT/Telecom department

. Equipment inside ESP considered as Critical Cyber Assets

. Protect against attacks from outside the substation – using substation router / firewall at the substation edge

. Protect against access from other networks inside substation - Isolated LAN for Station & Process bus

. Three out of four IEC 61850 Ed.2 profiles are L2 Multicast traffic

. Multicast = Broadcast, in flat networks without VLANs or Multicast Filtering

. Multicast Flooding Issue:

. Substation have few 10s to over 1000 IEDs depending on size, voltage class, generation, solar/wind farms etc.

. MUs generate 5-6Mbps of streaming SV traffic @ 80samples/sec depending on frequency of operation

. GOOSE is event driven – 1 kbps in steady-state and about 1 Mbps during bursts.

. GOOSE/SV mapped to QoS high-priority queue (PQ) due to low-latency requirements

. Congestion and packet-drops may occur during grid events if traffic is not contained in domains

. IEDs have to examine all flooded GOOSE/SV even if they are not subscribing to them

. Application segmentation with VLANs

SCADA, Feeder protection, Busbar protection, WAMS, etc.

. Applications have domains of publishers and subscribers

. Knowledge of data flows is important for filtering and segregation of traffic

. The Station & Process bus could be physically and logically segmented

. Use multicast filtering to confine traffic within multicast domains and VLANs to segregate traffic

. Topology

. Network Segmentation

. High Availability

. Timing

. Operations and Management

…but also QoS and Security!

Wide Area Network Defined Physical Boundary (DBP) Substation Network Electronic Security Substation Perimeter (ESP) Router

Private WiMax or LTE Serial, C37.94, E&M to Field Area Network Distributed FAN Aggregation HMI HMI Controller Legacy RTU IEC 61850 Multiservice Bus Station Bus

Legacy Comm Bay PMU Teleprotection Protection Teleprotection RTU Processor Controller Relay Relay Relay PDC

IEC 61850 Process Bus

Breaker Physical Hardwired MU IED Security I/O Workforce Enablement PT CT Breaker Sensor PT CT Breaker PT CT

BRKIOT-2111 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public Substation Automation HA IT type of applications and buying decision  Tree topology w/ RSTP more common REP also an option when rings used

Wide Area Network Defined Physical Boundary (DBP) Substation Network Electronic Security Substation Perimeter (ESP) Router

Private WiMax or LTE Serial, C37.94, E&M to Field Area Network Distributed FAN Aggregation HMI HMI Controller Legacy RTU IEC 61850 Multiservice Bus Station Bus

Legacy Comm Bay PMU Teleprotection Protection Teleprotection RTU Processor Controller Relay Relay Relay PDC

IEC 61850 Process Bus

Breaker Physical Hardwired MU IED Security I/O Workforce Enablement PT CT Breaker Sensor PT CT Breaker PT CT

BRKIOT-2111 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public SA applications (ex. IEC 61850 GOOSE) Substation Automation HA Automation buying decision and mindset  Ring topologies w/ REP very common RSTP often used when standards are mandatory PRP seeing adoption and being pushed by IEC

Wide Area Network Defined Physical Boundary (DBP) Substation Network Electronic Security Substation Perimeter (ESP) Router

Private WiMax or LTE Serial, C37.94, E&M to Field Area Network Distributed FAN Aggregation HMI HMI Controller Legacy RTU IEC 61850 Multiservice Bus Station Bus

Legacy Comm Bay PMU Teleprotection Protection Teleprotection RTU Processor Controller Relay Relay Relay PDC

IEC 61850 Process Bus

Breaker Physical Hardwired MU IED Security I/O Workforce Enablement PT CT Breaker Sensor PT CT Breaker PT CT

BRKIOT-2111 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public SA applications (ex. IEC 61850 SV) Substation Automation HA Automation buying decision and mindset Critical network requirements  PRP or HSR required REP may also be used on top of PRP

Wide Area Network Defined Physical Boundary (DBP) Substation Network Electronic Security Substation Perimeter (ESP) Router

Private WiMax or LTE Serial, C37.94, E&M to Field Area Network Distributed FAN Aggregation HMI HMI Controller Legacy RTU IEC 61850 Multiservice Bus Station Bus

Legacy Comm Bay PMU Teleprotection Protection Teleprotection RTU Processor Controller Relay Relay Relay PDC

IEC 61850 Process Bus

Breaker Physical Hardwired MU IED Security I/O Workforce Enablement PT CT Breaker Sensor PT CT Breaker PT CT

Communicating Partners Locale Network Recovery Time SCADA to IED client-server Station bus 100 ms

IED to IED interlocking Station bus 4 ms

IED to IED reverse blocking Station bus 4 ms

Bus bar protection Station bus 0 ms

Sampled values Process Bus 0 ms

• Provides a fast and predictable L2 • Does not replace Spanning Tree for convergence (50 to 250ms) even in complex layer 2 networks (mesh, tree) large rings with high number of nodes • Cisco proprietary • Now supported on a large range of Cisco products, including all IoTG • Supported on Layer 2 Trunk Ports and switches and CGR 2010 ESM Etherchannel only

• Very easy to configure and troubleshoot

• Co-existence with Spanning Tree (TCN from REP to STP)

• Optimal bandwidth utilization (VLAN Load balancing)

BRKIOT-2111 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 50 Parallel Redundancy Protocol Parallel Redundancy Protocol Benefits Limitations

• Zero packet loss when single LAN fails • Double of network components and cost • Designed for mission critical applications

• Supports any network topology: tree, mesh, ring, etc

• Allows devices that are not PRP aware

• Transparent to upper layer protocols and applications (ARP, DHCP, TCP/IP, etc)

• Parallel Redundancy Protocol: IEC 62439-3 Clause 4 • Two versions so far: PRP-0 (2010) and PRP-1 (2012) and they are not compatible • Two independent LANs must exist (any topology) • Two copies of each packet are delivered over these LANs • Seamless switchover and recovery in case of single LAN failure

• SAN — Singly Attached Node, connected to only one LAN

• DANP — Double Attached Node implementing PRP, connected to both LANs

• RedBox — Redundancy Box, connected to both LANs, a special DANP, proxy of SANs connected to it

• VDAN —Virtual DAN, SAN connected to RedBox

Protocol Topology # of Nodes Typical Remark Convergence RSTP/ MSTP Any Max hop 255 50ms-6s Not well suited for big ring topology

MRP Ring 50 10-500ms Roadmap in IE 2000 & IE 4000 HSR Ring unlimited 0ms Might be limited by node table size, not supported yet PRP Any unlimited 0ms Duplicate LANs, might be limited by node table size REP (Cisco Proprietary) Ring unlimited 50-250ms Depends on # of switches, media type, load-balancing

Station Bus & Process Bus Substation CE-PE

• Resilient Ethernet Protocol (REP) • VRRP / HSRP

• Rapid STP • BFD triggered static routes

• Parallel Redundancy Protocol (PRP) • BFD triggered IGP fast convergence Multiservice Bus

• Resilient Ethernet Protocol (REP)

• Rapid STP

ESP ESP DC RTU MPLS/IP MPLS/IP DFR Aggregation Core Sub-CE CC-CE CGR-2010 ASR-1k, ISR, IED/PMU CGS-2520 Sub-PE Core ABR CC-PE CGR-2010 ASR-903, ME3600 ASR-9k, ASR-903 ASR-9k, ASR-903 Substation Substation Aggregation Network Core Network Control Center

. Topology

. Network Segmentation

. High Availability

. Timing

. Operations and Management

…but also QoS and Security!

Service Providers Industrial Solutions Financing and Trading …but mostly here

Audio/Video Smart Grid Science

• General Applications (<1msec) Dedicated IRIG-B Cables – Sequence of Events GPS & Distance Limitations Antenn – Digital Fault Recorder (DFR) a • High Precision Timing (<10usec) Distributed IRIG-B Controller – Synchrophasors (C37.118) Source – Sample Values (IEC 61850-9-2)

– Distributed DFR Events Station Bus

• IEC 61850-5-2003 RTU DFR – Class T1: Events = ±1msec IED PMU IED PMU – Class T2: Syncrocheck ±0.1msec – Class T3: Samples Values ±25usec – Class T4: Samples Values ±4usec Process Bus – Class T5: Samples Values ±1usec

• There have previously been different ways to synchronise distributed clocks through a network. The most common of these are the Network Time Protocol (NTP) and the simpler Simple Network Time Protocol (SNTP) derived from it.

• IEC 61850 Edition 2 makes reference to IEEE 1588v2 Power Profile

• Precision Time Protocol (PTP) described in IEEE 1588 was developed with the following aims: – Synchronisation accuracy in the sub-microsecond range – Minimum requirements of the processor performance and network bandwidth – Low administration effort – Use via Ethernet networks – Specification as an international standard

BRKIOT-2111 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public Multiple 1588 Profiles Power Profile, as defined in IEEE C37.238, is used with the following settings: . Multicast; One-Step and Two-Step . Layer 2 (Ethernet) . Peer-To-Peer Delay Measurements Telecom Profile, as defined in ITU G.8265.1, is used with the following settings: • Unicast; One-Step and Two-Step • Layer 3 (TCP/IP) IPv4/UDP • No Boundary Clocks or Transparent Clocks (TC) • End to end timing only

• Specifies up to 1 microsecond over 16 hops • 200ns for the grandmaster clock, 50ns per switch

• Reasons for using GPS – nearly available everywhere – A GPS disciplined oscillator can provide time accurate within 100ns

• Reasons for not using GPS – see statement on www.pnt.gov, from Nov 3rd ,2010 • GPS should not be used as the unique reference in any critical civilian system – Reliability (very weak satellite signal) – Attacks (jamming and spoofing) – Cost of installation – Local Distribution (Splitters, Amplifiers, …) GPS Jammer Handheld GPS …Global Positioning System; GNSS … Global Navigation Satellite System

PTP IEEE1588-2011 C37.238 Power Profile PTP Hybrid Mode 1EEE1588-2008 + SyncE

Packet Master GPS GPS Antenna • GPS: Primary I/P • 10Mhz/1PPS/ToD: Backup I/P SyncE Packet Source-1 Master-1 SyncE/ESMC SyncE/ESMC GPS Antenna (Frequency) BC (Frequency) 1588v2 1588v2 1588v2 (Freq, Phase/ToD) Master 1588v2 Master 10Mhz (Freq, Phase/ToD) 1PPS 1588v2 ToD 1588v2 (Freq, Phase/ToD) (Freq, Phase/ToD) 10Mhz P2P Transparent BC BC Clock TC 10Mhz PRC MPLS/IP 1588v2 (Freq, Phase/ToD) 10Mhz IED/PMU Network MPLS/IP 1PPS Station/Process TC ToD TC Network 1588v2 SL Bus Master BC BC

Hybrid Boundary Clock Hybrid Boundary Clock Packet Slave • SyncE :Freq • SyncE :Freq SyncE Master-2 Clock • 1588v2: Phase/ToD • 1588v2: Phase/ToD Source-2 SL SL

IED/PMU IED/PMU Multiservice Bus PTP Master CGS-2520 PTP Master

MPLS/IP MPLS/IP Station / Process Aggregation Bus Sub-CE ESP Core CGR-2010 Sub-PE CGS-2520 ASR-903, ME3600 PRC

. Topology

. Network Segmentation

. High Availability

. Timing

. Operations and Management

…but also QoS and Security!

. IEC 61850-3 and IEEE 1613 for Substation environment compliance

. 5 years warranty

. +20 years MTBF

. Free lifecycle SW upgrades

. Redundant DC and AC Power Supplies

. Designed for simple operations by Industrial / Energy Engineers

Easy to Use • “Zero-config” replacement – Simple switch replacement in case of a failure – No networking expertise required – IE SwapDrive ensures fast recovery • Files stored on the SwapDrive – IOS Image – (tar, html) – 2 sets – Config text – VLAN dat – Other devices configs

BRKIOT-2111 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public Integration with Legacy Operation Systems CGS 2520 External Alarms Alarm # 1: Remote Security- Building or Cabinet Door Open / Closed

Alarm # 2: 4 Dry Environmental- Contact High Building or Cabinet Inputs Room Temperature

Alarm # 3: Power- SNMP Trap UPS or DC System Outputs

Alarm # 4: Network Environmental- Alarm Operations Output Fire / Smoke Center

BRKIOT-2111 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public Increase Communications and Service Visibility Network Monitoring / Diagnostics End-to-end service connectivity Cisco products rich suite of Ethernet OAM Protocols: verification between substations – Connectivity Fault Management / 802.1ag Proactively monitor different WAN parameters – such as latency or packet loss – Layer 3 IP SLA / Layer 2 IP SLA

WAN

Distributed Distributed Controller Controller

RTU DFR IEC 61850 Station Bus IEC 61850 Station Bus RTU

IED IED IED IED IED IED IED Substation #1 Process Bus Process Bus Substation #2

. Centralized Discovery, Inventory and Configuration Management

. Customizable out-of-the-box Cisco best practices and validated design configuration templates

. Automated deployment with PnP and Configuration Templates

. Fault and performance monitoring

. Infrastructure lifecycle reports - EoX, Contract, PSIRT

. 3rd party device support – discover and monitor RFC 1213 compatible 3rd party devices

• Reduces OpEx and maximizes ROI through consolidation • Lifecycle management support for wired and wireless • Unified access management** and client tracking

Guidelines Requirements Implementation & Technical Solutions

Grid

3 3

(G,A)

- -

Product Product

3 3

Technical Technical

(G, A) (G,

Security

1: 1:

Vendor

2: 2:

Vehicle

62351

2 2

Internet Protocols for Protocols Internet

Systems Systems

/COSEM /COSEM Security

Security levels Security

Requirements for IACS IACS for Requirements

Whitepaper Whitepaper

IEC IEC

solution suppliers solution

the Smart Grid Smart the

security requirements and and requirements security

IEC 62443 IEC

IEC 62443IEC

IEC 62056 IEC end Security for IEC TC57 IEC forSecurity end

Communication I/F Communication

Development requirements Development

DLMS

security requirements IACS requirements security

IEC 62443 IEC

System System

NIST 7628 NIST

BDEW

IEC 15118 IEC

DIN SPEC 27009 SPEC DIN

Integrator

End

RFC 6272: RFC

BDEWWhitepaper NIST SP800 NIST

IEC 62443-2-1 EU Mandate M/490 SGIS M/490 Mandate EU

Requirements for an IACS security management system Report: Smart Grid Information Security Information Grid Smart Report:

Guidelines for Smart Grid Cyber Security Security Cyber Grid Smart forGuidelines NERC CIP v5 Guide to Industrial Control Control Industrial to Guide (US, CAN) IEC 27019

Operator Security Management for Process Control © 2013-2015 Cisco and/or its affiliates. All rights reserved. Maik G. Seewald, CISSP Cisco Confidential 74 IEC TC57 Architecture of Information Standards

Distributed Energy Resources (DE R)

Electric Vehicle Market System Back Office DER Generator

IEC 61850-90-7, 8, 9, 10, 15 Control Center A 8 5 6

2 Control Center B

9 3 1

DER Storage DMS 2

EMS 6 6 C Apps. Apps. C E E I I 0

2 IEC 61970 IEC 61968 4 - 7 0 - 5

1 Communication Bus

6 C E I IEC 61970

IEC 60870-6 SCADA TASE.2/ICC P 0 1 IEC 62351 4 -

7 Cybersecurity 0 ) - 3

2 5

P 4

0 8 0 N 0 1 1 1 -

D 5

1 6 ( 5 0 5 - C 8 0 C 7 1 1 1 C - E

8 - 8 I 6

5 0 S - 0 1 C 7

E S 6 Substations / Field Devices E

8 E I C 0 E E 6 I I

IEC 61850-

Turbine and 90-5 electric systems

Substation RTUs PMUs IEC 61850 Automation Syste ms

Hydro systems IEC 60870-5-103 IEC 61850

Hydroelectric/ Gas SS-SS

Turbine Power Plants Protection, Control, Meterin g IEC 6185 0

GOOSE, SV

IEC 61850

Switchgear, Transformers, Instrumental Transformer s

IEC 62351: Undertake the development of standards for security of the communication protocols defined by the IEC TC 57 and on end-to-end security issues.

IEC TC57 Communication Standards IEC 62351 Security Standards

IEC 62351 Part 1: Introduction

IEC 62351 Part 2: Glossary IEC 60870-6: TASE.2 (ICCP)

IEC 62351 Part 3: Profiles

o L y

including TCP/IP t M i d s IEC 60870-5-104 & DNP3 r l e X

u e ) s r t c a d C o e n f o B

A t s e - r y M n B IEC 62351 Part 4: Profiles e t

m l e i e

IEC 60870-5-101 & Serial t R r e o b c including MMS (

m u g y l e DNP3 R j e

c a o C : b

e g r n 8 : t a

s S a O 9 t

n : r l t M 7 o i a

1 a

r t F k C 1

IEC 61850 over MMS a

IEC 62351 Part 5: IEC 60870-5 P M r

t P a s 1 o

r & Derivatives y s P 5 1 a e

w e t 3 5 P 1 K c

e 2 3 5 c 1 6 2 N 3

5 A 6

IEC 61850 GOOSE & SV 2

3 C 6 2

IEC 62351 Part 6: IEC 61850 C E I 6 E C

Profiles I E C I E IEC 61970 & IEC 61968 CIM I

IEC 62351 Part10: Security Architecture Guidelines for TC57 Systems

Title Security for IEC 61850 Applies IEC 61850 MMS, IEC 61850 GOOSE, SV Typical Use Process-Bus (in Substation), also Substation-to-Substation and Substation to Case Control Center Specifies . Authentication based on symmetric keys (group based) is mandatory . Authentication based on asymmetric keys (digital signature) is optional . SNTP (RFC 2030) to be used

Used in No implementations known, but several IED vendors seem to consider this products because unprotected GOOSE/SV is an issue Comment . Some vendors tested digital signature on GOOSE/SV . Computational burden is considered as too heavyweight . Changes has been initiated (asymm.  symmetric)

CIP-002 CIP-003 CIP-004 CIP-005

CRITICAL CYBER ASSETS SECURITY MANAGEMENT CONTROLS PERSONNEL AND TRAINING ELECTRONIC SECURITY

1. Critical assets 1. Cyber security policy 1. Awareness 1. Electronic security perimeter 2. Critical cyber assets 2. Leadership 2. Training 2. Electronic access controls 3. Annual review 3. Exceptions 3. Personnel risk assessment 3. Monitoring electronic access 4. Annual approval 4. Information protection 4. Access 4. Cyber vulnerability assessment 5. Access control 5. Documentation 6. Change control

CIP-006 CIP-007 CIP-008 CIP-009

SYSTEMS SECURITY INCIDENT REPORTING AND PHYSICAL SECURITY RECOVERY PLANS FOR CCA MANAGEMENT RESPONSE PLANNING 1. Plan 1. Test procedures 1. Awareness 1. Electronic security perimeter 2. Physical access controls 2. Ports and services 2. Training 2. Electronic access controls 3. Monitoring physical access 3. Security patch management 3. Personnel risk assessment 3. Monitoring electronic access 4. Logging physical access 4. Malicious software prevention 4. Access 4. Cyber vulnerability assessment 5. Account management 5. Access log retention 5. Documentation 6. Security status monitoring 6. Maintenance & testing 7. Disposal or redeployment 8. Cyber vulnerability assessment 9. Documentation

Version 3 Requirements Cisco Solutions CIP 2 Identification and Documentation of Critical Cisco Prime Infrastructure Cyber Assets CIP 3 Security Management Controls Cisco Prime Infrastructure, Cisco ACS/ISE CIP 4 Personnel and Training Cisco ACS/ISE for Centralized User Management CIP 5 Electronic Security Perimeters Traffic Segmentation, CGR 2010 ZBFW & IDS, ACS/ISE CIP 6 Physical Security Cisco Physical Access Manager (CPAM), Cisco Video Surveillance Manager (VSM), Cisco IP Interoperability and Collaboration System (IPICS) CIP 7 System Security Management PSIRT, Security Intelligence Operations, IPS, Cisco Validated SIEM CIP 8 Incident Reporting and Response Planning Cisco Validated SIEM CIP 9 Recovery Plans Cisco Product HA Cisco Prime Infrastructure

Access Control

• User and Device Identity PolicyManagement withOT/IT • Authentication, Authorization & Accounting Convergence & Ease of Use Availability Data Confidentiality and Data Privacy and Safety • Network Segmentation • Secure Connectivity Integrity Threat Detection and Mitigation • Security Zones • Intrusion Prevention; Application Visibility Confidentiality Device and Platform Integrity • Device Hardening and Secure Platform • Configuration Assurance

Secure Remote Access

Electronic Security Perimeter

OS Hardening

Network Segmentation

Certificates & Strong Encrypt.

Network Security

RBAC & Central Policy Server

Manufacturing Mining Energy-Utility Oil and Gas Transportation City Defense SP/M2M

Substation Automation, Connected Factory, Intelligent Transportation, Smart Cities, Connected Pipeline

Video Manager & IP Cameras CGR 1000 E2E Security ESR 5900 Physical Access Manager IE 2000U 819H Architecture SW ESR 5921 IPICS CGS 2520 IR 829 IE 4000 AP 1552 Industrial CGR 2010 Security ASR 900 Appliance ESS 2020

Plant Network Field Network Cyber Security Embedded Network Physical Security

IoT Network Management and IoT Security

Fog Computing (Cisco IOx)

Data Center / Virtualization

BRKIOT-2111 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public Industrial Ethernet Products Overview All PIDs with 5 Year Warranty Aggregation All PIDs are IEC 61850-3 Compliant Access IE5000*

- Designed for all industries IE4000 - Best In Class Switch - L2 or L3 (IP Service) - 4 port 1/10G uplinks - 12 Gig SFP + 12 Gig PoE/PoE+ - IEEE1588 PTP (Power Profile) - L2 NAT IE3000 CGS2520 - PoE/PoE+ (IE3010) - GPS, IRIG-B, ToD ready - Designed for all industries - Din Rail - L2 or L3 (IP Service)

Features IE2000 IE2000U - 4 port Gig uplinks - Up to 20 ports Gig - L2 or L3 (IP Svcs) - L2 or L3 (IP Svcs) - 1 RU - PRP - Modular - IEEE1588 PTP - Din Rail - Up to 24 ports - 8 PoE + 16 SFP (Power Profile) - L2, basic L3 - L2, or L3 - Up to 24 ports or 24 Copper - L2 NAT - Small Form Factor - Small Form Factor - IEEE1588 PTP - IEEE1588 PTP - Up to 8 PoE/PoE+ - Din Rail - Din Rail - PoE/PoE+ (Power Profile) - Dying Gasp - IP30, IP67 - PRP - PoE/PoE+ - L2 NAT - IEEE1588 PTP - IEEE1588 PTP (Power Profile) - PoE/PoE+ - PoE/PoE+

FAN ASR 902/3

- Modular (6 slots) – 3RU CGR 2010 - Raw Sockets - ISSU - 128 Gbps, Low Latency - Ethernet, Serial, E1/T1, STM-1 - MPLS IP, MPLS TP, VPLS - PseudoWires CGR 1120 CGR 1240 - SyncE, IEEE 1588, - Modular (4 slots) - Raw Sockets - Protocol Translation - Security - MPLS L3 VPN

Features 819H IR 829* - 2Combo GE - Ethernet Modules - IP30 - Serial - Modular - xDSL - Raw Sockets - IP67 - Protocol Translation - Modular - Security - Raw Sockets - IP41 - IP54 - 6FE Copper - Protocol Translation - Raw Sockets - Raw Sockets - 2GE Fiber - Security - Protocol - Protocol Translation - WiFi - 4FE Copper Translation - Security - NAN modules - 2GE Fiber - Security - 4FE Copper - IOX - WiFi, PoE - 4FE Copper - 1GE Fiber - NAN modules - 1GE Copper - WiFi, PoE - IOX - WiFi - IOX - IOX *Committed Roadmap

• IEC 61850 is today the main standard world wide for Energy Automation • The best Energy Automation solutions take into consideration the knowledge from Automation, Communications and Security Engineers • Power Utilities must own the Communication Design even if still buying turn key solutions • Management and specially Security are very often not addressed from the start as part of the overall architecture, with potential impact in future operations and reliability • Cisco has best in class Communications and Security Solutions for Energy

• Visit the World of Solutions for – Cisco IoT Booth and Whisper Suites – Walk in Labs – Technical Solution Clinics • Meet the Engineer • Lunch time Table Topics • DevNet zone related labs and sessions • Recommended Reading: for reading material and further resources for this session, please visit www.pearson-books.com/CLMilan2015

• Please complete your online session evaluations after each session. Complete 4 session evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt.

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations