CS415 Human Computer Interaction

Lecture 12 – VR, AR, IoT, Pervasive (Ch. 20)

November 11, 2019  Sam Siewert Remaining Assignments

Assignment #5 – Explore HCI’s, Propose Group Project (Groups of 2, 3 or 4)

Assignment #6 – Proof-of-Concept or Evaluation with Final Report

Final Oral Exam – Presentation of Project

FINAL: 10:15-12:15pm, Sat, Dec 7, Sims&Games, Bldg 61 – Oral Presentation of Assignment #6 Report – With Updates for any Progress since

 Sam Siewert 2 Ubiquitous Computing Term Credited to Mark Weiser – “The Computer of the 21st Century”, Scientific American, 265(3):66-75, September 1991 – Vision Described Previously at Many HCI Research Organizations [e.g. Xerox PARC, MIT Media Lab, GIT, …]

Some Theorized Enabling Technologies (Book) – High speed wireless networks – Computer Vision – Perceptual Computing to Emulate Human Perception and Extend it – Projection, 3D, High Resolution Large Displays and Mobile Computing – Head tracking, Independent Display Eyeware, 3D Glasses – Large Scale Interactive Surfaces (Large Touch Screens) – Data Gloves and Body Suits

 Sam Siewert 3 Advanced HCI … Future AR/VR and IoT - Daily life integrated computing

Microsoft Research HCI Vision

Interesting HCI Innovations and Experiments (Examples) – MS Hololens, – , – jibo, – sixthsense, – VR, – NVIDIA AR/VR, PX, – Intel Real Sense, – Google Project Tango, – Interactive Film, – Gear VR  Sam Siewert 4 Ubiquitous Computing [“computing is made to appear anytime and everywhere”]

Pranav Mistry, Ted Talk on Interaction (We watched) John Barrett, IoT Ted Talk – Cork Inst. Of Technology – Embedded Systems Research Marco Annunziata, GE, IIoT Ted Talk (Watch …) Privacy in IoT World - Daniel Price TedX (Watch …)

Week 12 … IoT and re-visit to AR, VR, Quiz-3 Week 13 … Advanced Avionics and UAV/UAS Week 14 – Quiz-4, Review for Exam #2, A#6 Help Week 15 – Exam #2 Final Oral Exam - 12/14

 Sam Siewert 5 IoT and IIoT Reality and Vision Complex machines and systems self- report (digital twin)

Built-into devices and systems

Secure wireless connectivity

Quote from Aviation Week - Jan 2016: Internet of Aircraft Things Last year we saw the next generation of aircraft powered by the capabilities of Internet of Things (IoT) and Big Data hit scales that How much can a have never seen before. What does the future hold? pilot assess H&S At last year’s Paris Air Show for example, Bombardier showcased its C Series jetliner that carries Pratt & Whitney’s Geared Turbo Fan (GTF) from visual engine, which is fitted with 5,000 sensors that generate up to 10 GB of data per second. A single twin-engine aircraft with an average 12-hr. inspection? flight-time can produce up to 844 TB of data.

 Sam Siewert 6 Markets for Ubiquitous Computing Virtuous link - Games & Animation, HPC – Clusters (ILM, Pixar, Dreamworks, …) – GPU and GP-GPU (NVIDIA, ATI, Intel) – Serious Gaming (Education, Therapy, …)

Transportation (Industrials) Nov, 2018 Nov, 2019 – Move people, animals, goods and commodities – Modes - air, rail, road, water, cable, pipeline, space – UAS – aerial transportation at lower cost? Mobile, Internet – Self-Driving Cars Games Energy & Utilities (IIoT) – Smart GRID – generation and distribution – Alternative Energy Sources and storage Cyberphysical – Clean Nuclear and Coal (e.g. TeraPower) – Carbon sequestration Enterprise

Services (IT and Telecomm) Cyberphysical – Mobile and Wireless for IoT, IIoT (5th Generation) Cyberphysical

Enterprise Health Care (IoT) – Elder care (at home) – Therapeutics at home – Patient monitoring, Health and Fitness monitoring – Big data analytics – pathology, disease Cyberphysical cures, control and prevention  Sam Siewert 7 Definitions Ubiquitous Computing – Merge computational artifacts smoothly with world of physical artifacts (applications integrated with things) – Superimpose relevant information over top of the object – Objects in the physical world to manipulate virtual (graspable) – Context aware applications (Who, What, Where, When, Why) – Recommendation Engines, Adware, Personal Assistants – Vannevar Bush (Memex) – Often compared to Hypertext Today

VR () – Immersive – Desktop – Simulated World

AR () – Project Onto World – See World Through Mobile Screen / Glasses With Annotation – Requires Computer Vision (Object Recognition) for True AR  Sam Siewert 8 Internet of Things A new form of Ubiquitous Computing? – IPv6 - 128 bit address, 2128 , or about 3.4 x 1038 – Eddington_number - Theorized bound on Elements in Universe 256 79 – NEdd = 136×2 , or about 1.57×10

Build processing into industry and everyday life – IIoT – Industrial Internet of Things E.g. Honeywell – IIoT, Intel IIoT, GE IoT – IoT – Everyday and Workday Internet of Things E.g. Intel IoT, IBM IoT – Network infrastructure for IoT (Wireless) E.g. ATT IoT, Verizon IoT, Cisco IoT

Security Challenges (Cyberphysical) – Classic Risk, Benefit, Impact, Cost to Mitigate Design Trade-offs – Remote Monitoring and Proactive Maintenance

 Sam Siewert 9 IoT Processors - Sensors + Wireless Very Small, Wireless (BLE), Wired Ethernet, Built-in Sensors, Very Low-Power

E.g. Intel’s Quark, NXP, Texas Instruments TM4C, CC32XX

Arduino – Education, Makers and Quick Prototypes

Zephyr (Linux Foundation) – Micro / Nano kernel – Boards supported – Paper on Curie/Quark – Example Zephyr IoT Code

Simple Executives – E.g. Tiva with PDL

 Sam Siewert 10 NSF View of IoT Benefits and Risks of the “Internet of Things”

Embedded Computing – Cyberphysical Systems, Extension of Embedded Systems (Networked) https://ptolemy.berkeley.edu/projects/cps/

 Sam Siewert 11 Embedded Software and IoT

Simple Executive on bare metal - TI PDL with TIVA (Microprocessor class/lab - CEC 320/322)

Micro or Nano kernel purpose-built for IoT – Linux Foundation Zephyr – FreeRTOS – 8 More - Riot OS, Win10-IoT, VxWorks IIoT, Google Android Things, ARM Mbed, Apple Homekit, Nucleus, Green Hills

RTOS or OS on IoT device (CEC 450) – E.g. Raspberry Pi 3b+ (Drone Net Acoustic Tripwire Demonstration) Battery power (power bank) WiFi (802.11) Sensor interfaces (GPS, microphone, IMU) – General info on R-Pi 3b+ for IoT InformationWeek - 10 Raspberry Pi Projects For Learning IoT https://www.raspberrypi.org/blog/getting-started-with-iot/

 Sam Siewert 12 Zephry and Linux for IoT

Linux Foundation Real-Time IoT Nano / Micro-kernel

 Sam Siewert 13 What is IoT? Internet of Things (paper on Zephyr / Curie-Quark) – Similar to older idea of “Sensor Networks”, but high bandwidth Internet sensors and actuators – https://elinux.org/RPi_Hub (R Pi is large IoT Device) – Nodes MUST have at least local/wide area wireless interface BLE - Bluetooth Low Energy - mobile phone or tablet 802.11 - Wireless Ethernet 4G LTE or 5G Cellular

Security is a major challenge

Opportunity is smart devices and “digital twins” 4G LTE, 5G – E.g. Pilot walk-around with tablet for aircraft health and status – Telemetry to manufacturer for predictive maintenance – Simple configuration, provisioning, upgrades Zephyr board support (ARM NXP, Intel, …, TI TIVA?) Xbee, Zigbee, etc. BLE  Sam Siewert 14 More IoT Study

Digital Circuits - CEC220, CEC222

Embedded Systems - Micro class (CEC320) and lab (CEC322)

Undergraduate research with ICARUS (Siewert, Bruder, Andalibi) or Capstone IoT, or Dr. “A”, Dr. Post, etc.

Graduate programs – CU ESE program – MIT Sloan IoT business program – Stanford IoT Certificate program

 Sam Siewert 15 Required Software Basic Nano / Micro Kernel Network stack software – FreeRTOS – BLE network stack – Zephyr – 802.11 network stack – RTEMS – Cellular modem

App interfaces – AOS, iOS

L7 SCP Service SCP Client

L6 Semi-structured file Semi-structured file

L5 Sockets API Sockets API

L4 TCP TCP

L3 IP IP Router/Switch IP

L2 IEEE 802.3z - GigE 802.11abgn - Wireless

L1 Cat-6 UTP Cat-6 UTP

Internet

 Sam Siewert 16 Ubiquitous Computing Attribute Model IoT systems have more applications than CPS or ES? Relation to Cyberphysical Systems (CPS larger scope than ES?) Relation to Embedded Systems (IoT is a class of ES?) Ubiquitous Computing IoT (context aware, merged physical and virtual) Architecture

Class model or IoT (sensors, wireless, processing, digital twin) IoT a superset of ES+CPS? Might be too or Cyberphysical Systems (networked Constraining or Killer app M2M, HCI, secure) Oversimplified (making ES more E.g. ES with attributes prevalent) not found in a CPS or Embedded Systems IoT (i.e. not a proper subset)

 Sam Siewert 17 IoT Architecture - Class Model ES Less restrictive than attributes = {low power, superset/subset purpose built, specific model services, custom I/O}

ES as a base class CPS Transportation ES attributes = attributes CPS inherits from ES {ES’ + networked, {ES’ + durability, security, M2M, HCI} EMI/EMC} IoT inherits form CPS

Healthcare CPS IoT attributes = Goal is ubiquitous attributes = {CPS’ + therapeutics, computing {CPS’ + wireless, diagnostics, monitoring} digital twin}

 Sam Siewert 18 Smart Sensors - Machine Learning with IoT - Neuromorphic Architectures NVIDIA VOLTA GP-GPU Co-Proc NVIDIA Volta - Deep Neural Nets, GP-GPU

Google TPU - Tensor processing Unit Google TPU Co-Proc (Cloud)

Intel Nervana - NNP (Neural Net Processor)

Historical Intel Nervana Chip Neuromorphic - Intel ETANN and Ni1000

 Sam Siewert 19 Playing with Concepts Cost Reduction is Key to Become Ubiquitous (Killer App) MERL DiamondTouch (2003) - Book Example Computer Vision Chess, Automatic Duck Hunter Industrial HCI Research

MERL

Xerox PARC

Intel Perceptual Computing

MS Research

IBM HCI Research

 Sam Siewert 20 VR, AR Killer Applications Remains to Be Seen

Today – Scientific Visualization (VR) – Digital Cinema Animation (VR, AR) – Training Systems (Surgery, AR Overlay to Identify Objects) – Games and Entertainment (VR) – First Person Viewing and Sensing (Remote Presence), E.g. UAVs (AR)

Ubiquitous Use – Not Yet? – Cost and Convenience (Pranav Mistry Pendant – Good Start - $350, Easy to Wear) – Compelling Value (Wear Computer or Build into Things – IoT)

 Sam Siewert 21 New HCI is Risky

Does the HCI Define? Right Time to Innovate?

Classic Mistakes 1. Complexity - NRE, Capex, Opex 2. Scope control - Creeping Featurism Catalogue of Catastrophe 3. Expectations and Goals 4. Users and Evaluation 5. Social Acceptance 6. Micro-management vs. No management 7. Poor quality workmanship - SQA 8. Risk management – assessment, adjustment 9. Performance requirements – Interactive 10. Poorly planned transitions - Deployment

 Sam Siewert 22 Minute Paper

Beyond Today’s Vertical Applications Previously Enumerated…

VR, What is it Good for?

AR, What is it Good for?

IoT, What is Good for?

Will Ubiquitous Computing Envisioned by Mark Weiser be Realized?

 Sam Siewert 23 References Kranz, Matthias, Paul Holleis, and Albrecht Schmidt. "Embedded interaction: Interacting with the internet of things." IEEE internet computing 14.2 (2010): 46-53.

Sundmaeker, Harald, et al. "Vision and challenges for realising the Internet of Things." Cluster of European Research Projects on the Internet of Things, European Commision 3.3 (2010): 34-36.

Ray, Partha Pratim. "Internet of Things for Sports (IoTSport): An architectural framework for sports and recreational activity." Proceeding of IEEE EESCO, Vizag (2015): 79-83.

Fleisch, Elgar. "What is the internet of things? An economic perspective." Economics, Management & Financial Markets 5.2 (2010).

Motlagh, Naser Hossein, Tarik Taleb, and Osama Arouk. "Low-altitude unmanned aerial vehicles-based internet of things services: Comprehensive survey and future perspectives." IEEE Internet of Things Journal 3.6 (2016): 899-922.

Haller, Stephan. "The things in the internet of things." Poster at the (IoT 2010). Tokyo, Japan, November 5.8 (2010): 26-30.

Sadeghi, Ahmad-Reza, Christian Wachsmann, and Michael Waidner. "Security and privacy challenges in industrial internet of things." Design Automation Conference (DAC), 2015 52nd ACM/EDAC/IEEE. IEEE, 2015.

Stankovic, John A. "Research directions for the internet of things." IEEE Internet of Things Journal 1.1 (2014): 3-9.

 Sam Siewert 24 References Canedo, Arquimedes. "Industrial IoT lifecycle via digital twins." Proceedings of the Eleventh IEEE/ACM/IFIP International Conference on Hardware/Software Codesign and System Synthesis. ACM, 2016.

Tao, Fei, et al. "Digital twin-driven product design, manufacturing and service with big data." The International Journal of Advanced Manufacturing Technology 94.9-12 (2018): 3563-3576.

Alam, Kazi Masudul, and Abdulmotaleb El Saddik. "C2ps: A digital twin architecture reference model for the cloud-based cyber-physical systems." IEEE Access 5 (2017): 2050-2062.

 Sam Siewert 25