Internet Outages: Not Just for Disasters Data networking with Amateur Radio © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 1 One upon a time, the general public only had this … • Fixed location voice communications • Talk to one person at a time (or maybe two) © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 2 So amateur radio brought this … • Portable voice communications • Shared frequency – Talk to multiple people at once – Don’t need to know which number to call • Multiple frequencies – Multiple conversations at once © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 3 Now, the general public has this … • Most of the same capabilities of an amateur radio HT • Of course, no repeaters … • … but heck, the package says they have a 22 mile range, right? © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 4 Plus, they have this … • Portable voice communications • Privacy (well, except for the NSA) • Plus: – Text – E‐mail – Web – Maps, GPS – Video chat – Broadcast radio – Broadcast television – Music – Movies – … © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 5 And this is how they use it … i.e. Phone Calls © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 6 Bottom line, the General Public is … • Capable of their own portable voice communications – Granted, with some limitations compared to amateur radio repeaters • Has access to and makes heavy use of more data services in the palm of their hand than were available 10 years ago on the desktop © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 7 So, when we offer to help at an event … © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 8 … perhaps we should think about … © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 9 … showing up with more than just this. • There are still many important uses for an individual with an HT – Shadow, rover, … • But we are capable of so much more! © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 10 What Else Can We Do? • Services that the general public can’t easily provide for themselves at an event, such as data networking – Too complicated – Too expensive – Too time consuming • Services that don’t rely on commercial infrastructure (power, Internet, …) – May not exist at the venue – If disrupted, would hamper the event • Backup for existing services in the event a disaster knocks out commercial infrastructure © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 11 BUT THE INTERNET NEVER GOES DOWN, RIGHT? © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 12 Natural and Man Made Causes • Natural causes • Man made causes – Earthquakes – Terrorism – Snow – Criminal activity – Rain/Flood – Negligence – Mud slide – Incompetence – Fire – Accident – … – Human error – Mechanical failure – … © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 13 Many Causes of Network Outages • Power failure – Generation, transmission, distribution, local facilities • Structural failure – Buildings, towers, conduits, … • Component failure – Hardware (e.g. relay contact), electronic (e.g. bad capacitor) • Link failure – Antennas, cables (e.g. water ingress, cable cut, …), congestion, signal fade • Software failure – Control‐plane failure – Configuration errors • Procedural errors – Miss a step; Perform step B before step A • … © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 14 Data Gathering Process • The Santa Clara County ARES/RACES data network has Internet service at four different locations, each from a different service provider • We monitor temperature, humidity, IP connectivity • Currently pinging all sites every 10 seconds, retry 5 times every 2 seconds. • Will see any outage of 21 seconds or longer – May see outages of 11 seconds or longer • Kept track of two categories – Flaps: <= 1 minute – Outages: > 1 minute © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 15 Internet Flaps at Four Access Points • July – September, 2014 • Disruptions lasting < 1 minute • Most likely due to rebooting a device or temporary signal fade • Not noticeable for e‐mail • Can be a real pain for interactive traffic (SSH, VoIP, Video) – Especially since they tend to come in bunches Total Flaps / Site % Flaps / Site 100% 4 3 80% 14 16 20 60% 50 40% 85 26 37 20% 22 0% Jul‐14 Aug‐14 Sep‐14 1‐WHD 2‐CPK 3‐MTV 7‐MLP 1‐WHD 2‐CPK 3‐MTV 7‐MLP © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 16 Internet Outages at Four Access Points • Jan‐Sep 2014 Totals: 46 Outages (> 1 min) totaling 57.6 hours # of Outages (> 1 min) Total Outage Time (hrs) 0.6 0.3 1 10 13.6 7 28 43.1 1‐WHD 2‐CPK 3‐MTV 7‐MLP 1‐WHD 2‐CPK 3‐MTV 7‐MLP © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 17 Internet Outages at Four Access Points • January through Sept 2014 • 46 outages lasting > 1 minute, totaling 57.6 hours • 15 outages > 5 minutes; 8 outages > 1 hour • Two longest: 10.4 hrs, 32.3 hrs # of Outages (> 1 min) Outage Time (hrs) 16 40 14 35 12 30 10 25 8 20 32.25 10 6 15 4 9 10 2 44 5 10.53 33 2 0 0 1 1 0 0.29 Jan Feb Mar Apr May Jun Jul Aug Sep Jan Feb Mar Apr May Jun Jul Aug Sep 1‐WHD 2‐CPK 3‐MTV 7‐MLP 1‐WHD 2‐CPK 3‐MTV 7‐MLP © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 18 Availability (% Uptime) • Availability = (Total time –Down Time) / Total Time • Service level agreements often refer to monthly availability or monthly downtime • How much downtime is acceptable for your application if it happens to occur during the event you are supporting? Availability %Downtime per Year Downtime per Month 90% (“one nine”) 36.5 days 72 hours 99% (“two nines”) 3.65 days 7.2 hours 99.5% 1.83 days 3.6 hours 99.9% (“three nines”) 8.76 hours 43.8 minutes 99.95% 4.38 hours 21.56 minutes 99.99% (“four nines”) 52.56 minutes 4.32 minutes 99.995% 26.28 minutes 2.16 minutes 99.999% (“five nines”) 5.26 minutes 25.9 seconds © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 19 Availability at Four Access Points Monthly Availability (% Uptime) at Four Access Points Month 1‐WHD 2‐CPK 3‐MTV 7‐MLP Jan 100.0000% 99.6841% 100.0000% Feb 100.0000% 99.9107% 100.0000% Legend: Mar 100.0000% 99.9073% 99.9610% >= 99.9999% Apr 100.0000% 99.8444% 100.0000% >= 99.999% >= 99.99% May 100.0000% 99.9960% 100.0000% 99.9906% >= 99.9% Jun 100.0000% 99.9958% 98.5375% 99.9819% >= 99% Jul 99.9597% 99.5013% 100.0000% 99.9812% < 99% Aug 100.0000% 99.9906% 100.0000% 99.9866% Sep 100.0000% 99.3083% 95.5208% 99.9722% © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 20 Conclusion –Is that Good or Bad? • For the most part, Internet service delivery is within the carrier’s service level agreement (SLA) – Even though we might like them to be better! • The real issues to consider are: – Is the service you’re going to offer needed, or just nice to have? • If it fails, will anyone care? – If it’s needed, and if your service depends on the Internet, then how long of an Internet service disruption is too long? • Consider eliminating dependence on the Internet by using amateur radio to deliver data services © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 21 So what can we offer that doesn’t depend on the Internet? POTENTIAL AMATEUR RADIO USES FOR DATA NETWORKING © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 22 Existing Service Offerings ‐ Disasters • Official message traffic flows: – CERT neighborhood Command Post to City Emergency Ops Center – City EOC to County EOC – Hospitals to MHOC – City fire stations to City EOC – County fire stations to county fire HQ, City/County EOC – Water districts, school districts, shelters, Red Cross, … – County EOC or MHOC to Region/State – … • Primary / Secondary – For some, amateur radio is the only alternative to the PSTN – For others, amateur radio is the alternative if private facilities (and radios) are damaged or evacuated • Take advantage of ARES/RACES/ACS experience – Practiced frequently; many lessons learned © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 23 Potential Service Offerings –Non‐Disasters • Parades – APRS tracking of first/last/key floats, rovers, shadows • Races – APRS tracking of SAG wagons, rovers – Statistics collection (bib#/time), supply requests from rest stops – Video monitoring of key locations, e.g. finish line • Festivals – APRS tracking of rovers, shadows – Telephony between volunteer workers, e.g. info booths – Video monitoring of key locations, e.g. entrances, parking lots • Other ideas …? © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 24 Example: Los Altos Festival of Lights APRS TRACKING © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 25 Typical APRS Connectivity to the Internet APRS iGate General Public Access http://aprs.fi Tracker GPS TNC Radio Receiver Radio Internet © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 26 Integrated APRS Trackers • Integrated: Radio / GPS / APRS TNC • Portable: for individuals – Yaesu VX‐8DR – Byonics Micro‐Track All‐In‐One Kenwood Yaesu Byonics TH-D72 VX-8DR Micro-Track AIO – Others … • Mobile: for vehicles – Kenwood TM‐D710G – Byonics Micro‐Trak Ready‐To‐Go – Others … Kenwood TM-D710G Byonics Micro-Track RTG © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 27 Portable APRS Tracking Solution for Events Trackers On Rovers Roll-Up J-Pole Public uses: @ http://aprs.fi Mast APRSdroid Trackers On Hams On Vehicles Internet Hotspot If Available Radio & TNC APRSISCE Net Control Station, Info Booth, etc.