CS415 Human Computer Interaction Lecture 12 – Ubiquitous Computing VR, AR, IoT, Pervasive (Ch. 20) November 11, 2019 Sam Siewert Remaining Assignments Assignment #5 – Explore HCI’s, Propose Group Project (Groups of 2, 3 or 4) Assignment #6 – Proof-of-Concept or Evaluation with Final Report Final Oral Exam – Presentation of Project FINAL: 10:15-12:15pm, Sat, Dec 7, Sims&Games, Bldg 61 – Oral Presentation of Assignment #6 Report – With Updates for any Progress since Sam Siewert 2 Ubiquitous Computing Term Credited to Mark Weiser – “The Computer of the 21st Century”, Scientific American, 265(3):66-75, September 1991 – Vision Described Previously at Many HCI Research Organizations [e.g. Xerox PARC, MIT Media Lab, GIT, …] Some Theorized Enabling Technologies (Book) – High speed wireless networks – Computer Vision – Perceptual Computing to Emulate Human Perception and Extend it – Projection, 3D, High Resolution Large Displays and Mobile Computing – Head tracking, Independent Display Eyeware, 3D Glasses – Large Scale Interactive Surfaces (Large Touch Screens) – Data Gloves and Body Suits Sam Siewert 3 Advanced HCI … Future AR/VR and IoT - Daily life integrated computing Microsoft Research HCI Vision Interesting HCI Innovations and Experiments (Examples) – MS Hololens, – Magic Leap, – jibo, – sixthsense, – Oculus VR, – NVIDIA AR/VR, PX, – Intel Real Sense, – Google Project Tango, – Interactive Film, – Gear VR Sam Siewert 4 Ubiquitous Computing [“computing is made to appear anytime and everywhere”] Pranav Mistry, Ted Talk on Interaction (We watched) John Barrett, IoT Ted Talk – Cork Inst. Of Technology – Embedded Systems Research Marco Annunziata, GE, IIoT Ted Talk (Watch …) Privacy in IoT World - Daniel Price TedX (Watch …) Week 12 … IoT and re-visit to AR, VR, Quiz-3 Week 13 … Advanced Avionics and UAV/UAS Week 14 – Quiz-4, Review for Exam #2, A#6 Help Week 15 – Exam #2 Final Oral Exam - 12/14 Sam Siewert 5 IoT and IIoT Reality and Vision Complex machines and systems self- report (digital twin) Built-into devices and systems Secure wireless connectivity Quote from Aviation Week - Jan 2016: Internet of Aircraft Things Last year we saw the next generation of aircraft powered by the capabilities of Internet of Things (IoT) and Big Data hit scales that How much can a have never seen before. What does the future hold? pilot assess H&S At last year’s Paris Air Show for example, Bombardier showcased its C Series jetliner that carries Pratt & Whitney’s Geared Turbo Fan (GTF) from visual engine, which is fitted with 5,000 sensors that generate up to 10 GB of data per second. A single twin-engine aircraft with an average 12-hr. inspection? flight-time can produce up to 844 TB of data. Sam Siewert 6 Markets for Ubiquitous Computing Virtuous link - Games & Animation, HPC – Clusters (ILM, Pixar, Dreamworks, …) – GPU and GP-GPU (NVIDIA, ATI, Intel) – Serious Gaming (Education, Therapy, …) Transportation (Industrials) Nov, 2018 Nov, 2019 – Move people, animals, goods and commodities – Modes - air, rail, road, water, cable, pipeline, space – UAS – aerial transportation at lower cost? Mobile, Internet – Self-Driving Cars Games Energy & Utilities (IIoT) – Smart GRID – generation and distribution – Alternative Energy Sources and storage Cyberphysical – Clean Nuclear and Coal (e.g. TeraPower) – Carbon sequestration Enterprise Services (IT and Telecomm) Cyberphysical – Mobile and Wireless for IoT, IIoT (5th Generation) Cyberphysical Enterprise Health Care (IoT) – Elder care (at home) – Therapeutics at home – Patient monitoring, Health and Fitness monitoring – Big data analytics – pathology, disease Cyberphysical cures, control and prevention Sam Siewert 7 Definitions Ubiquitous Computing – Merge computational artifacts smoothly with world of physical artifacts (applications integrated with things) – Superimpose relevant information over top of the object – Objects in the physical world to manipulate virtual (graspable) – Context aware applications (Who, What, Where, When, Why) – Recommendation Engines, Adware, Personal Assistants – Vannevar Bush (Memex) – Often compared to Hypertext Today VR (Virtual Reality) – Immersive – Desktop – Simulated World AR (Augmented Reality) – Project Onto World – See World Through Mobile Screen / Glasses With Annotation – Requires Computer Vision (Object Recognition) for True AR Sam Siewert 8 Internet of Things A new form of Ubiquitous Computing? – IPv6 - 128 bit address, 2128 , or about 3.4 x 1038 – Eddington_number - Theorized bound on Elements in Universe 256 79 – NEdd = 136×2 , or about 1.57×10 Build processing into industry and everyday life – IIoT – Industrial Internet of Things E.g. Honeywell – IIoT, Intel IIoT, GE IoT – IoT – Everyday and Workday Internet of Things E.g. Intel IoT, IBM IoT – Network infrastructure for IoT (Wireless) E.g. ATT IoT, Verizon IoT, Cisco IoT Security Challenges (Cyberphysical) – Classic Risk, Benefit, Impact, Cost to Mitigate Design Trade-offs – Remote Monitoring and Proactive Maintenance Sam Siewert 9 IoT Processors - Sensors + Wireless Very Small, Wireless (BLE), Wired Ethernet, Built-in Sensors, Very Low-Power E.g. Intel’s Quark, NXP, Texas Instruments TM4C, CC32XX Arduino – Education, Makers and Quick Prototypes Zephyr (Linux Foundation) – Micro / Nano kernel – Boards supported – Paper on Curie/Quark – Example Zephyr IoT Code Simple Executives – E.g. Tiva with PDL Sam Siewert 10 NSF View of IoT Benefits and Risks of the “Internet of Things” Embedded Computing – Cyberphysical Systems, Extension of Embedded Systems (Networked) https://ptolemy.berkeley.edu/projects/cps/ Sam Siewert 11 Embedded Software and IoT Simple Executive on bare metal - TI PDL with TIVA (Microprocessor class/lab - CEC 320/322) Micro or Nano kernel purpose-built for IoT – Linux Foundation Zephyr – FreeRTOS – 8 More - Riot OS, Win10-IoT, VxWorks IIoT, Google Android Things, ARM Mbed, Apple Homekit, Nucleus, Green Hills RTOS or OS on IoT device (CEC 450) – E.g. Raspberry Pi 3b+ (Drone Net Acoustic Tripwire Demonstration) Battery power (power bank) WiFi (802.11) Sensor interfaces (GPS, microphone, IMU) – General info on R-Pi 3b+ for IoT InformationWeek - 10 Raspberry Pi Projects For Learning IoT https://www.raspberrypi.org/blog/getting-started-with-iot/ Sam Siewert 12 Zephry and Linux for IoT Linux Foundation Real-Time IoT Nano / Micro-kernel Sam Siewert 13 What is IoT? Internet of Things (paper on Zephyr / Curie-Quark) – Similar to older idea of “Sensor Networks”, but high bandwidth Internet sensors and actuators – https://elinux.org/RPi_Hub (R Pi is large IoT Device) – Nodes MUST have at least local/wide area wireless interface BLE - Bluetooth Low Energy - mobile phone or tablet 802.11 - Wireless Ethernet 4G LTE or 5G Cellular Security is a major challenge Opportunity is smart devices and “digital twins” 4G LTE, 5G – E.g. Pilot walk-around with tablet for aircraft health and status – Telemetry to manufacturer for predictive maintenance – Simple configuration, provisioning, upgrades Zephyr board support (ARM NXP, Intel, …, TI TIVA?) Xbee, Zigbee, etc. BLE Sam Siewert 14 More IoT Study Digital Circuits - CEC220, CEC222 Embedded Systems - Micro class (CEC320) and lab (CEC322) Undergraduate research with ICARUS (Siewert, Bruder, Andalibi) or Capstone IoT, or Dr. “A”, Dr. Post, etc. Graduate programs – CU ESE program – MIT Sloan IoT business program – Stanford IoT Certificate program Sam Siewert 15 Required Software Basic Nano / Micro Kernel Network stack software – FreeRTOS – BLE network stack – Zephyr – 802.11 network stack – RTEMS – Cellular modem App interfaces – AOS, iOS L7 SCP Service SCP Client L6 Semi-structured file Semi-structured file L5 Sockets API Sockets API L4 TCP TCP L3 IP IP Router/Switch IP L2 IEEE 802.3z - GigE 802.11abgn - Wireless L1 Cat-6 UTP Cat-6 UTP Internet Sam Siewert 16 Ubiquitous Computing Attribute Model IoT systems have more applications than CPS or ES? Relation to Cyberphysical Systems (CPS larger scope than ES?) Relation to Embedded Systems (IoT is a class of ES?) Ubiquitous Computing IoT (context aware, merged physical and virtual) Architecture Class model or IoT (sensors, wireless, processing, digital twin) IoT a superset of ES+CPS? Might be too or Cyberphysical Systems (networked Constraining or Killer app M2M, HCI, secure) Oversimplified (making ES more E.g. ES with attributes prevalent) not found in a CPS or Embedded Systems IoT (i.e. not a proper subset) Sam Siewert 17 IoT Architecture - Class Model ES Less restrictive than attributes = {low power, superset/subset purpose built, specific model services, custom I/O} ES as a base class CPS Transportation ES attributes = attributes CPS inherits from ES {ES’ + networked, {ES’ + durability, security, M2M, HCI} EMI/EMC} IoT inherits form CPS Healthcare CPS IoT attributes = Goal is ubiquitous attributes = {CPS’ + therapeutics, computing {CPS’ + wireless, diagnostics, monitoring} digital twin} Sam Siewert 18 Smart Sensors - Machine Learning with IoT - Neuromorphic Architectures NVIDIA VOLTA GP-GPU Co-Proc NVIDIA Volta - Deep Neural Nets, GP-GPU Google TPU - Tensor processing Unit Google TPU Co-Proc (Cloud) Intel Nervana - NNP (Neural Net Processor) Historical Intel Nervana Chip Neuromorphic - Intel ETANN and Ni1000 Sam Siewert 19 Playing with Concepts Cost Reduction is Key to Become Ubiquitous (Killer App) MERL DiamondTouch (2003) - Book Example Computer Vision Chess, Automatic Duck Hunter Industrial HCI Research MERL Xerox PARC Intel Perceptual Computing MS Research IBM HCI Research Sam Siewert 20 VR, AR Killer