University Information Technology An Integrated Approach to Facilitating Network and Computational Research at the University of Utah Steve Corbató Deputy CIO, University Information Technology Adjunct Faculty, School of Computing Joe Breen Network Architect, Center for High Performance Computing University of Utah

GEC18 – GENI Engineering Conference Brooklyn, New York – October 28, 2013

Fourth node on the ARPANET – 1969

ARPANET (1969-1980) Univ. of Utah Computer Science Department (now School of Computing) University Information Technology Network research facilitation leverages highly collaborative environment • SoC Flux research group (Ricci, Eide, Van der Merwe) • Center for High Performance Computing (Breen) • University IT – CIO Office (VPIT Denna, Corbató), – Infrastructure (Elieson) – Security (Bowden) • VP for Research (VPR Parks, Furse) • Utah Education Network (Stewart, Quire) • Utah NSF EPSCoR (Teutonico, Crowl, Jones) • Internet2 (Vietzke, Wolff) University Information Technology Research infrastructure development • Downtown Data Center • Campus Network Upgrade and Science DMZ – NSF CISE CC-NIE • Metro SLC/Northern Utah Optical Network – NSF EPSCoR RII Cyber Connectivity – NTIA BTOP (UEN) • UEN GENI Spiral 3 • Network research/computational science – Apt: NSF MRI Instrument Development (Ricci)

 74,000+ sq ft2 former industrial building south of downtown SLC (~4 miles off-campus)  Designed for both enterprise & HPC (2.4 MW)  Co-location by research groups & partners  Low industrial electric power rates in Utah  Energy efficient design (low PUE)  In operation since spring 2012

5 Utah Education Network Wide Area Network

1.5 Mbs T1 45 Mbs - 100 Mbs Ethernet Richmond LIB

White Pine MS 45 Mbs DS3 N. Cache Center Park Valley Skyview HS Rich JHS (Box Elder SD) To USU Cache DO Cedar Ridge MS 155 Mbs OC3 Mt. Logan MS Cache HS Logan HS Spring Creek MS Logan S. Campus Logan DO S. Cache Center Gigabit Ethernet 1000 Mbs Mountain Crest HS Bear River HS USU To USU Bear River MS Willow Valley MS Fast Forward Charter School Alice C. Harris IS Grouse Creek Geomax 1000 Mbs (Box Elder SD) Rich HS Brigham City LIB Box Elder MS Geomax 1000 Mbs Backbone Box Elder HS Rich DO Box Elder DO Adele C. Young IS 10 Gbs Backbone

Central MS Highland MS Ogden DO Mound Fort MS Mount Ogden MS Ben Lomond HS WSU Snowcrest JHS Washington HS Ogden Tech Center See Salt Lake County Detail Ogden HS Weber HS Fremont HS Weber DO Orion JHS N. Ogden JHS MorganHS/MS Wahlquist JHS Bonneville HS TH Bell JHS WSU Davis Morgan DO S. Ogden JHS BDO Rocky Mtn JHS DATC Two Rivers HS Sand Ridge JHS Roy HS Dagget DO Roy JHS Davis DO N.Summit HS/DO Manila HS

Park City LIB S.Summit MS/DO EBC

Wendover HS Salt Lake DO Park City DO Wendover CO Grantsville HS Grantsville MS Tooele DO SLCC Granite DO Wasatch DO Neola CO USU Vernal Wasatch Co. LIB Altamont HS Tooele Valley HS Murray DO Wasatch MS Altamont CO Roosevelt MS Wasatch HS Uinta DO Roosevelt JHS La Point CO Vernal CO Jordan DO NUES Tabiona HS Tabiona CO Tooele MS Roosevelt LIB Ashley Valley HS Timpanogos SS UVSC Wasatch Duchesne DO Roosevelt CO Tooele HS Pleasant Grove LIB Vernal JHS Lone Peak HS J. Hancock School Roosevelt USU Roosevelt Uintah HS Timpanogos HS Union HS USU Vernal Alpine DO Mountainview HS Uintah River HS Duchesne CO UBATC Vernal Myton CO Orem HS Duchesne DO Lehi HS UBATC Ballard CO UVU Orem LIB Duchesne HS West JHS Duchesne LIB Myton/Conamore Thomsen School Provo DO Duchesne Jail Dugway HS Mountainland ATC

Alpine Life and Learning Ibapah EL (Tooele SD)

Nebo DO Springville HS Spanish Fork JHS Callao EL Tintic DO Spanish Fork HS (Tintic SD) Payson HS Spanish Fork MS

Tintic HS

Juab DO N. Sanpate MS N. Sanpete HS Juab HS Juab MS Wasatch Academy Helper LIB E. Carbon HS

N. Sanpete DO Carbon HS Carbon DO CEU Wellington LIB

Snow College Millard DO

West Desert HS Delta HS (Tintic SD) S. Sanpete DO Canyon View JHS Delta ATC Emery DO

Manti HS Eskdale School Ephraim MS (Millard SD) Emery HS

Gunnison HS

San Rafael JHS Garrison EL Fillmore MS (Millard SD) Fillmore HS

N. Sevier HS Green River HS N. Sevier MS CUES Snow College South Sevier DO Richfield Data Center Richfield HS Cedar Ridge HS Red Hills MS S. Sevier HS S. Sevier MS Grand HS Grand DO Moab Bald Mesa SouthEast ATC

Milford HS Wayne DO

Beaver DO Wayne HS Hanksville EL Wayne MS Beaver HS Minersville Piute DO/HS School

Antimony EL

Escalante Valley EL USU DCE

Monticello HS Parowan LIB Garfield DO Parowan HS Panguitch CO

Panguitch HS Escalante HS SUU Panguitch MS

Iron DO Canyon View HS Cedar City MS Bryce Valley HS SEDC Foothill HS Enterprise HS San Juan Media Center San Juan DO SWATC Cedar City HS Cross Hollows MS San Juan HS

Pine View HS Pine View MS Snow Canyon MS Post HS Snow Canyon HS Tonaquint MS Hurricane LIB Lava Ridge IS Valley HS Lake Powell EL Tuachan Charter School Hurricane IS Dixie HS Hurricane MS Montezuma Dixie MS Creek Higher Ed Whitehorse HS Desert Hills HS Hurricane HS Desert Hills IS Dixie Hurricane Ctr Fossil Ridge MS Kanab CO Santa Clara LIB Washington DO SWATC Kanab DCE 6 Washington Co. LIB Kane DO Kanab HS Monument Valley St. George Rehab Big Water EL Monument Valley HS Higher Ed Kanab MS Dixie College (Kane SD) Navajo Mountain HS  Collaboration of Utah Education Network (UEN) and University of Utah . Leverage UEN operational capability & statewide reach  Leverage public sector partner assets (fiber & conduit) wherever possible . UDOT (highway RoW) . Utah Transit Authority (UTA/TRAX light rail)  Work with wholesale oriented carriers (Zayo, Syringa)

7 SLC metro optical network

To Logan

215 15 Salt Palace Salt Lake SLC Weather Forecast 100 S. West Temple International Office Airport Utah Higher Education 2242 W. North Temple Kearns Building UEN EBC 60 S. 400 W. U of U Park Bldg. 136 S. Main St. 101 Wasatch Dr. 201 President’s Circle

NOAA Western Region 80 Headquarters 200 S. Downtown 125 S. State St. Salt Lake City

400 S.

.

t

S

n i

Level 3 PoP a M Internet2 & NLR Nodes University of 572 DeLong St. Utah Main 15 Campus 800 S. 900 S.

215 U of U Data Center 875 S. West Temple

Research@UEN: Salt Lake City Metro Optical Network U of U Campus Fiber UTA Light Rail Routes (proposed) CENIC/LLC Fiber IRU (through AFS) AFS Fiber IRU (proposed) Northern Utah Extension (proposed) 1 Mile

Carrier proprietary information included

8 www.it.utah.edu Campus Help Desk 801-581-4000 Optical network extensions to Logan

Idaho Utah

and Provo15 for USU, BYU and UVU 84 Research@UEN

Logan (Utah State University) Tremonton 30 Northern Utah Optical Network Extensions (amplifier, if necessary) Dark Fiber IRU Along UDOT Right-of- Way to Logan Dark Fiber IRU Along UDOT Right-of- Way to Provo (this proposal)

W 20 Miles/32 Kilometers

y

U

o

t

m

a

h

i

n

g Ogden (potential future add/drop node)

84 80 15 Wyoming Utah

University of Utah Level 3 PoP UofU Data Center 80 Salt Lake City Metro Fiber Ring Idaho Logan

Ogden Orem Wyoming (potential future add/drop node for UVU) Salt Lake City Provo (Brigham Young

University) Provo a

C d

o a

l

o v

r e

a N

15 d

o

Richfield 9 www.it.utah.edu Campus Help Desk 801-581-4000 Cedar City

St. George Arizona 100 Miles/161 Kilometers University Information Technology UEN GENI: Yr 1-2 Results

• 3-year award starting September 2011 • Deployed fiber and lambdas between EBC PoP and Level3 PoP • Deployed Switch in UEN EBC PoP • Deployed fiber and lambdas between EBC PoP and DDC PoP • Deployed Switch in UofU Downtown Data Center • Brought up 34 node InstaGENI rack in DDC with ProtoGENI group University Information Technology UEN GENI: GENI S3 Year 3

• Extend GENI capabilities to USU & BYU • Extend GENI capabilities to a Utah charter high school (Utah County Academy of Sciences) • Continue GENI collaboration with University of Idaho and IRON University Information Technology Utah CC-NIE award

• “Science Slices Converting Network Research Innovation into Enhanced Capability for Computational Science and Engineering at the University of Utah” • Collaboration of Flux research group & CHPC • Funded under Network Integration and Applied Innovation track – Oct 2013 start • Implementing ProtoGENI/GENI research outcomes in production science environment

University Information Technology

“What if we turned things upside down and placed the Science DMZ on top of SDN infrastructure?” – Prof. Rob Ricci, University of Utah

Implication: Make the Science DMZ a slice UTAH CC-NIE PROJECT University Information Technology A perspective – A “slice” is really a conceptual idea as well as an implementation –A virtual overlay network with unique security, performance and other characteristics University Information Technology Slices enable flexibility

• Science DMZ as a Slice • Network Research as a Slice(s) • Honors Student Housing as a Slice • Different slices – different contexts

University Information Technology Science DMZ as a slice – Science DMZ defines a separate egress at the border and a section of the network for a set of known Data Transfer Nodes (see http://fasterdata.es.net/science-dmz ) • Easy to define for HPC resources at data center • Easy to define for certain known research groups where we can reach with dedicated fiber. • What about ephemeral Data Transfer Nodes? • What about individual Research areas in campus connected to instruments that normally have network security constraints (e.g., PHI)? • ‘ University Information Technology Domain Science is not always constrained to the Data Center

– UofU needs deeper and dynamic penetration of the Science DMZ into campus network • Example #1: Huntsman Cancer Institute (HCI) genomic sequencers to CHPC computational resources • Example #2: Need dedicated path to pathology lab (ARUP) equipment which resides in a different domain on campus

University Information Technology How to balance these slices?

• Large bandwidth slice with other slices? Do we still egress on dedicated ports on a switch? (Both intra-and inter-campus) Is backbone overprovisioning adequate? • How to deal with security for sensitive flows? • How to monitor substrate and individual slices – what information do campus and regional operators need and how to expose? • What are the best mechanisms to implement security within slice?

 Utah Education Network (UEN)  Utah State University  Brigham Young University  Weber State University  Utah Valley University  Utah Transit Authority (UTA)  Utah Dept. of Transportation (UDOT)  Utah Dept. of Technical Services (DTS)  Governor’s Office of Economic Development (GOED)  EDCUtah  Salt Lake City Corporation.  Utah EPSCoR – iUTAH and CI-WATER  USTAR  Salt Palace Convention Center  Internet2  University of Montana  Idaho Regional Optical Network (IRON)  Funding provided by National Science Foundation, National Telecommunications and Information Administration (U.S. Dept. of Commerce) & University of Utah 20 Contact information

• Steve Corbató – [email protected]

• Joe Breen – [email protected]

www.it.utah.edu Campus Help Desk 801-581-4000