SCADA Network & Data Redundancy at Guelph Water
Ensuring Security of Supply for
Operations and Compliance
Graham Nasby Water SCADA & Security Specialist, Guelph Water Services
Derek Wong Principal Consultant, Symbiotic Networks 1 OWWA Conference –Apr 29 to May 2, 2018 –Niagara Falls, Ontario, Canada About the Speakers
Graham Nasby, P.Eng., PMP, CAP Water SCADA & Security Specialist City of Guelph Water Services
• 10 years in the consulting sector • Joined Guelph Water Services in 2015 • Co-chair of ISA112 SCADA Systems standards committee • Member of the OWWA Automation Committee • Contact: [email protected]
Derek Wong Principal Consultant, Symbiotic Networks Founder, Zabu
• 20+ years building and managing IT infrastructure • Extensive experience supporting Communication Service Providers • Active member in the information security community • Outspoken advocate of open source software • Contact: [email protected] 2 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 2 Presentation Outline
•SCADA Systems Review •Guelph SCADA Network Pre‐2017 •Datalogging Requirements: O.Reg. 170 • Impact of SCADA Network & Data‐Logging Outages
•SCADA network design goals •Guelph Water Solution: Fibre & Wireless with BGP •SCADA Data‐logging design goals •Guelph Water Solution: 3 layers of data‐logging
• Lessons Learned •Best Practices Moving Forward
3 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 3 City of Guelph Water Services
• Guelph, Ontario, Canada • 140,000 residents • 21 groundwater wells • 3 water towers
• 549 km of water mains • 49,000 service connections • 2,750 fire hydrants • 35 unmanned facilities • 46,000 m3/day [12 MGD]
4 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 4 Map of Guelph Water
•35 Unmanned sites
•Central Ops Centre is staffed 8am‐4pm Mon‐Fri •On‐call operators on evenings and weekends
•30‐40 minutes to drive across the city (with traffic) ARKELL SPRINGS
5 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 5 Water Supply Facilities
• Woods Station (Chlorine Building, UV Building, Booster Station)
•Arkell 1 & 7 •Downey Well •Arkell 6 •Verney Tower •Arkell 8 •Burkes Well •Arkell 14 •Arkell Well •Arkell 15 • Clair Tower •Diversion Chamber • Clair Booster •Water Street Well • Robertson Booster • Emma Well •Helmar Well •Park Well •University Well •Clythe Station •Dean Well •Paisley Station • Calico Well •Membro Well • Speedvale Tower •Carter Well •Scout Camp • Queensdale Well
All sites are linked together with SCADA Network for remote control, automatic control, monitoring and logging. 6 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 6 What is SCADA?
SCADA = Supervisory Control and Data Acquisition 7 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 7 Typical SCADA Architecture
8 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 8 Typical SCADA Network
9 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 9 Regulatory Requirements • O.Reg. 170 –Drinking Water Systems •Free Chlorine Residuals Must be Logged Minimum Every 5 minutes • Filter Plants: Turbidity must be logged every 15 minutes •Low chlorine alarms must be communicated promptly
•Most IT Systems only reach 90 ‐ 95% uptime (outages at night ok) • Very challenging to meet this uptime for a SCADA network •SCADA system must be online 24/7, 365 days/year •Less than 5 min downtime/year = 99.9995% uptime •SCADA system has to be 10,000 X more reliable than an IT system
10 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 10 From O.Reg. 170, Section 6.5:
11 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 11 “Nines” Availability
I.T.
SCADA
12 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 12 Guelph SCADA Network: Pre-2017
13 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 13 Impact of SCADA Network Outages •Data Logging – Meeting the 5 minute data‐logging requirement •Process Visibility – Ability of Operators to “see” the status of remote sites • Speed of Response – Ability to make adjustments remotely vs. Driving a Truck to Site •Automatic Control of Pumps based on Water Tower Levels – Requires PLC‐to‐PLC communications to work
•Can be very disruptive to Operations –Operator has to be on site to physically write‐down residuals every 5 min –Operator stationed at a Water Tower to call in pump starts/stops via radio
14 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 14 Goals of SCADA Network Upgrade • Maintain SCADA Network Connectivity at All Times •Be able to keep operating if a Fibre Cable is damaged/cut •Be able to keep operating during fibre provider maintenance
• Reduce reliance on store/forward data‐loggers during outages • Reduce the impact on Operations during a Fibre Outage • Reduce the compliance risk associated with SCADA outages
• Implement continuous network monitoring for SCADA network •Gather network statistics to monitor SCADA network performance
15 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 15 SCADA Network Technology Options •Existing –MPLS on Fibre (dedicated private network, via a Service Provider)
•Options to Supplement –Back‐up Fibre Connections –DSL –Ethernet‐over‐cable – Cellular –Private Cellular –Radios (unlicensed) –Radios (licensed)
•How do we activate the backup network link when it is needed? 16 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 16 Redundant SCADA Network Design •Hub Site –MPLS Primary & Secondary Fibre Optic Connections –Two different cable routings to the building(“diversity”) –HSRP = Hot Standby Router Protocol –Two Routers that use HSRP to create a floating “IP Address” that is shared between routers –UPS‐backed power, with Diesel Generator Backup
•Remote Sites –MPLS Primary Link: Fibre – Backup MPLS Link: Encrypted 3G/LTE Wireless on Private Cellular Network –One Router Used running BGP: Border Gateway Protocol –BGP is configured to switch to backup link in 45 seconds if Primary Fibre connection lost –BGP will automatically switch back to fibre when fibre connectivity is restored –UPS‐backed power
17 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 17 HSRP: Hot Standby Router Protocol • Technology developed by Cisco •Provides redundant Gateway using 2 identical fibre network links •Used for Main Site because two links are both fibre & identical •Does not do routing (could not be used for Remote Sites)
•Router One: xxx.xxx.0.1 •Router Two: xxx.xxx.0.2 •“Virtual Network Address” that shows up at: xxx.xxx.1.1 •Takes about 2 minutes to switch from Primary to Secondary •Does both “fail‐over” and “fail‐back” automatically
18 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 18 BGP: Border Gateway Protocol •BGP is the Protocol that runs the Internet –Used by Internet Service Providers to connect their networks together –For example, the connection between Bell and Rogers uses BGP
•We use BGP to transparently and automatically use fibre & backup wireless links •BGP switches from fibre to backup wireless within 45 seconds of a failure
• Border Gateway Protocol (BGP) is a standardized exterior gateway protocol designed to exchange routing and reachability information among autonomous systems (AS) on the Internet.The protocol is classified as a path vector protocol. The Border Gateway Protocol makes routing decisions based on paths, network policies, or rule‐sets configured by a network administrator and is involved in
making core routing decisions. (source: Wikipedia)
19 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 19 SCADA Network - Before
20 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 20 Upgraded Network
MAIN SITE
21 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 21 SCADA Network Monitoring Dashboard
22 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 22 SCADA Network Monitoring Graphs
23 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 23 SCADA Network Uptime Results • 2016 SCADA network uptime: 99.6% • 2017 SCADA network uptime (post upgrade): 99.995% + increasing
• Pre‐upgrade when fibre provider had to do network maintenance it was very disruptive and required sites to be shutdown/manned • Post‐upgrade when fibre provider has to do network maintenance, the network at that site automatically switches to backup wireless
• Fibre failures are no longer a major operational inconvenience •Network monitoring allows us to proactively fix problems
24 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 24 Back up Data-Logging •No technology or person is perfect… •So we have backup data‐logging to guard against failures/mistakes • Primary Logging – Facility PLC –SCADA server reads data in “real‐time” • Backup Data‐logging – “QuickPanel” store/forward data‐logger – Continuously logs into a buffer – Pushes data up to SCADA server –If no connectivity, stores timestamped data –When network connectivity restored, forwards logged data up to server • DNP3‐based Data‐loggers New Technology 25 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 25 Existing Data Logging Infrastructure
Historian
SCADA Network
QuickPanel Site PLC Data‐logger
Flowmeters, Analyzers, Level, Pressure 26 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 26 New Data Logging Infrastructure
Backup Main Historian Historian
SCADA Network
DNP3 Data‐logger QuickPanel Data‐logger Site PLC (store & forward) (store & forward)
Critical Instrumentation: Flow, Analyzers Flowmeters, Analyzers, Level, Pressure 27 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 27 Wiring the DNP3 Logger Into the Loop
28 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 28 DNP3 Protocol for Data-Logging • Automatic store/forward data logging is built into the protocol • Timestamping data at the device/instrument • Automatic Time synchronization •Both “Change on Value” and “Polling Interval” data gathering •Data Quality Flags
• Supports both Encryption and Authentication •Automatic Error checking of data packets • Prioritized Communications (Tags can be grouped in classes 0‐4) •Variety of networks supported: Ethernet, serial, ATM, cellular, etc. • Supports wide range of analog, digital and structured datatypes 29 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 29 New Data Logging Infrastructure
Backup Main Historian Historian
SCADA Network
DNP3 Data‐logger QuickPanel Data‐logger Site PLC (store & forward) (store & forward)
Critical Instrumentation: Flow, Analyzers Flowmeters, Analyzers, Level, Pressure 30 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 30 Accessing your Logged Data
31 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 31 Take Aways •A functioning SCADA system is critical for operations –SCADA Network allows for remote data‐logging & control –Data‐logging is needed to meet requirements of O.Reg. 170 –SCADA outages can be very disruptive to operations and have compliance risks •Uptime requirements for SCADA systems are very high •A typical SCADA system has to be 10,000X more reliable than an IT system •O.Reg. 170 requires us to log chlorine residual every 5 minutes (no exceptions!)
1. Consider using a redundant wire‐area SCADA network with automatic failover 2. Use redundant data‐logging to guard against equipment failures 3. Look at DNP3 protocol for doing back‐up store & forward data logging 4. Think about how your operations/compliance team will get access to the data 32 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 32 Any Questions?
* Not a High Performance SCADA System
Graham Nasby – graham.nasby@guelphca
Derek Wong – [email protected] 33 Backup SCADA Networks & Data‐logging OWWA 2018 –Niagara Falls, Ontario, Canada 33