Internet Outages: Not Just for Disasters Data Networking with Amateur Radio

Internet Outages: Not Just for Disasters Data networking with Amateur Radio

• Fixed location voice communications • Talk to one person at a time (or maybe two)

• Portable voice communications • Shared frequency – Talk to multiple people at once – Don’t need to know which number to call • Multiple frequencies – Multiple conversations at once

• Most of the same capabilities of an amateur radio HT

• Of course, no repeaters …

• … but heck, the package says they have a 22 mile range, right?

• Portable voice communications • Privacy (well, except for the NSA) • Plus: – Text – E‐mail – Web – Maps, GPS – Video chat – Broadcast radio – Broadcast television – Music – Movies – …

i.e. Phone Calls

• Capable of their own portable voice communications – Granted, with some limitations compared to amateur radio repeaters

• Has access to and makes heavy use of more data services in the palm of their hand than were available 10 years ago on the desktop

• There are still many important uses for an individual with an HT – Shadow, rover, …

• But we are capable of so much more!

• Services that the general public can’t easily provide for themselves at an event, such as data networking – Too complicated – Too expensive – Too time consuming

• Services that don’t rely on commercial infrastructure (power, Internet, …) – May not exist at the venue – If disrupted, would hamper the event

• Backup for existing services in the event a disaster knocks out commercial infrastructure

• Natural causes • Man made causes – Earthquakes – Terrorism – Snow – Criminal activity – Rain/Flood – Negligence – Mud slide – Incompetence – Fire – Accident – … – Human error – Mechanical failure – …

• Power failure – Generation, transmission, distribution, local facilities • Structural failure – Buildings, towers, conduits, … • Component failure – Hardware (e.g. relay contact), electronic (e.g. bad capacitor) • Link failure – Antennas, cables (e.g. water ingress, cable cut, …), congestion, signal fade • Software failure – Control‐plane failure – Configuration errors • Procedural errors – Miss a step; Perform step B before step A • …

• The Santa Clara County ARES/RACES data network has Internet service at four different locations, each from a different service provider • We monitor temperature, humidity, IP connectivity • Currently pinging all sites every 10 seconds, retry 5 times every 2 seconds. • Will see any outage of 21 seconds or longer – May see outages of 11 seconds or longer • Kept track of two categories – Flaps: <= 1 minute – Outages: > 1 minute

• July – September, 2014 • Disruptions lasting < 1 minute • Most likely due to rebooting a device or temporary signal fade • Not noticeable for e‐mail • Can be a real pain for interactive traffic (SSH, VoIP, Video) – Especially since they tend to come in bunches

Total Flaps / Site % Flaps / Site 100% 4 3 80% 14 16 20 60% 50 40% 85 26 37 20% 22

0% Jul‐14 Aug‐14 Sep‐14 1‐WHD 2‐CPK 3‐MTV 7‐MLP 1‐WHD 2‐CPK 3‐MTV 7‐MLP

• Jan‐Sep 2014 Totals: 46 Outages (> 1 min) totaling 57.6 hours

# of Outages (> 1 min) Total Outage Time (hrs) 0.6 0.3 1 10 13.6

7 28 43.1

1‐WHD 2‐CPK 3‐MTV 7‐MLP 1‐WHD 2‐CPK 3‐MTV 7‐MLP

• January through Sept 2014 • 46 outages lasting > 1 minute, totaling 57.6 hours • 15 outages > 5 minutes; 8 outages > 1 hour • Two longest: 10.4 hrs, 32.3 hrs

# of Outages (> 1 min) Outage Time (hrs) 16 40 14 35 12 30 10 25 8 20 32.25 10 6 15 4 9 10 2 44 5 10.53 33 2 0 0 1 1 0 0.29 Jan Feb Mar Apr May Jun Jul Aug Sep Jan Feb Mar Apr May Jun Jul Aug Sep 1‐WHD 2‐CPK 3‐MTV 7‐MLP 1‐WHD 2‐CPK 3‐MTV 7‐MLP

• Availability = (Total time –Down Time) / Total Time • Service level agreements often refer to monthly availability or monthly downtime • How much downtime is acceptable for your application if it happens to occur during the event you are supporting?

Availability %Downtime per Year Downtime per Month 90% (“one nine”) 36.5 days 72 hours 99% (“two nines”) 3.65 days 7.2 hours 99.5% 1.83 days 3.6 hours 99.9% (“three nines”) 8.76 hours 43.8 minutes 99.95% 4.38 hours 21.56 minutes 99.99% (“four nines”) 52.56 minutes 4.32 minutes 99.995% 26.28 minutes 2.16 minutes 99.999% (“five nines”) 5.26 minutes 25.9 seconds

Monthly Availability (% Uptime) at Four Access Points

Month 1‐WHD 2‐CPK 3‐MTV 7‐MLP Jan 100.0000% 99.6841% 100.0000% Feb 100.0000% 99.9107% 100.0000% Legend: Mar 100.0000% 99.9073% 99.9610% >= 99.9999% Apr 100.0000% 99.8444% 100.0000% >= 99.999% >= 99.99% May 100.0000% 99.9960% 100.0000% 99.9906% >= 99.9% Jun 100.0000% 99.9958% 98.5375% 99.9819% >= 99% Jul 99.9597% 99.5013% 100.0000% 99.9812% < 99% Aug 100.0000% 99.9906% 100.0000% 99.9866% Sep 100.0000% 99.3083% 95.5208% 99.9722%

• For the most part, Internet service delivery is within the carrier’s service level agreement (SLA) – Even though we might like them to be better!

• The real issues to consider are: – Is the service you’re going to offer needed, or just nice to have? • If it fails, will anyone care? – If it’s needed, and if your service depends on the Internet, then how long of an Internet service disruption is too long?

• Consider eliminating dependence on the Internet by using amateur radio to deliver data services

• Official message traffic flows: – CERT neighborhood Command Post to City Emergency Ops Center – City EOC to County EOC – Hospitals to MHOC – City fire stations to City EOC – County fire stations to county fire HQ, City/County EOC – Water districts, school districts, shelters, Red Cross, … – County EOC or MHOC to Region/State – … • Primary / Secondary – For some, amateur radio is the only alternative to the PSTN – For others, amateur radio is the alternative if private facilities (and radios) are damaged or evacuated • Take advantage of ARES/RACES/ACS experience – Practiced frequently; many lessons learned

• Parades – APRS tracking of first/last/key floats, rovers, shadows • Races – APRS tracking of SAG wagons, rovers – Statistics collection (bib#/time), supply requests from rest stops – Video monitoring of key locations, e.g. finish line • Festivals – APRS tracking of rovers, shadows – Telephony between volunteer workers, e.g. info booths – Video monitoring of key locations, e.g. entrances, parking lots • Other ideas …?

APRS iGate General Public Access http://aprs.fi

Tracker

GPS TNC Radio Receiver Radio

Internet

• Integrated: Radio / GPS / APRS TNC

• Portable: for individuals – Yaesu VX‐8DR – Byonics Micro‐Track All‐In‐One Kenwood Yaesu Byonics TH-D72 VX-8DR Micro-Track AIO – Others …

• Mobile: for vehicles – Kenwood TM‐D710G – Byonics Micro‐Trak Ready‐To‐Go – Others … Kenwood TM-D710G Byonics Micro-Track RTG © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 27 Portable APRS Tracking Solution for Events

Trackers On Rovers

Roll-Up J-Pole Public uses: @ http://aprs.fi Mast APRSdroid Trackers On Hams On Vehicles

Internet

Hotspot If Available Radio & TNC APRSISCE Net Control Station, Info Booth, etc. © Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 28 Example: Cupertino Citizens Corps BROADBAND DATA VOICE OVER IP

• Communications van for all Citizen Corps communications – ARES/RACES/ACS, CERT, … • Ubiquity 5.8 GHz connection to EOC • Rasberry Pi Asterisk PBX, VoIP phones – Communications between van and EOC

5.8 GHz Link to EOC Asterisk PBX

• Recent SCCo ARES/RACES County‐wide Exercise

• Provide easy‐to‐use phone services between event workers – Info booths, start/finish line, press office, first aid station, … • Monitor conditions at entrance, start/finish line, … • Optionally share external connections to PSTN

POTS or SIP Event VoIP Phones Mesh PSTN Network IP PBX If Available IP Cameras

• Line of sight can be a real problem for 802.11 networks – Example: Cupertino ARK Project

• Even more of a problem with ad hoc, temporary nets • Consider narrowband applications – Text messaging: keyboard‐to‐keyboard – E‐mail or packet messaging

• Each CERT Unit (neighborhood) gathers information – Uncontrolled fires – Hazards: gas, water, electrical, chemical – Extent of building damage – # of people injured, trapped, dead – Road obstructions or blockages – Other threats to life

Wes Freeman, KG6POV

Projected onto a screen in the Mountain View EOC

© Copyright 2014 Santa Clara County ARES®/RACES. All rights reserved. 40 For ad hoc, real‐world deployment … You don’t necessarily need a lot of bandwidth … But you need SOME bandwidth! COVERAGE IS KEY

• Iridium GO! • Latest product from Iridium satellite network (summer 2014) • Internet hotspot for up to 5 WiFi clients • Only 4.5” x 3.25” x 1.25” –very portable • 5.5 hours talk time, 15 hours standby time • Only needs view of the sky • But: – $850 for the unit itself – Need satellite data plan: $$$ – Iridium GO! application on each client – Satellite connection speed: 2400 bps – Iridium Mail and Web provides compression and light‐weight, lower‐

• Comprehensive coverage of Santa Clara County (and beyond) • Error free transmission • Easy to use interface • No monthly access fees • No additional equipment costs

Portable, Battery operated, Fixed Sites (typically) Deploy anywhere UPS, Generator

VHF/UHF (typically) No Internet Wide Coverage Required

Radio BBS BBS Bulletin TNC Board (Modem) System BBS GW Gateway PC Running Outpost and Other PacFORMS Networks

W4XSC

W3XSC W1XSC

W2XSC

• Comprehensive coverage of Santa Clara County and beyond • In-county locations can reach at least two BBS sites • Two-way interconnectivity with Internet e-mail • But no Internet required to reach anywhere in the county!

BBS Network Winlink AMPRnet

Internet EE-mail

2m 2m 2m 2m

W1XSC W2XSC W3XSC W4XSC Test 1.25m 1.25m 1.25m 1.25m

AX.25 Backbone (70cm)

BBS Network Winlink AMPRnet

Internet EE-mail

2m 2m 2m 2m

W1XSC W2XSC W3XSC W4XSC Test 1.25m 1.25m 1.25m 1.25m

AX.25 Backbone (70cm)

802.11/Microwave/Fiber Backbone

Next steps: 1. Move to high speed backbone as main BBS-to-BBS transport; 70cm network as backup 2. Add links to high-speed backbone to become fully redundant; 70cm can be repurposed

BBS Network BBS Network

AMPRnet AMPRnet

Internet EE-mailmail Internet EE-mailmail

2m Access 2m Access 1200 baud 1200 baud BBS … BBS … 1.25m Access 1.25m Access 1200 baud 1200 baud

70 cm Backbone 70cm Access 56k FEC Radios 56kbps

High Speed Backbone Redundant Redundancy High Speed Backbone

• Internet access connections: – The “last mile” is the most vulnerable part of the network – Failures occur for many, many reasons; not just disasters – Failures occur a significant number of times each month, and for a significant period of time each failure – How long of an outage is too long if it happens during your event? • Consider offering more advanced services: data/video – A guy with an HT is still useful, but we can do much more today – Both narrowband and broadband data networking is in regular use by radio amateurs in the area – We have great coverage and continuously improving speed, capability • Get involved – Experiment, build networks, write software, or just help out; have fun!