Software > Blockchain > Augmented

Home , Betty Holberton, IEEE Rebooting Computing, Mixed reality, Ruth Teitelbaum, The Hillside Group

> Software > Blockchain > Augmented and Virtual Reality > Cyber-Physical Systems

SEPTEMBER 2019 www.computer.org Keep Your Career Options Open Upload Your Resume Today!

Whether your enjoy your current position or you are ready for change, the IEEE Computer Society Jobs Board is a valuable resource tool. Take advantage of these special resources for job seekers:

JOB ALERTS TEMPLATES

CAREER RESUMES VIEWED ADVICE BY TOP EMPLOYERS No matter your career WEBINARS level, the IEEE Computer Society Jobs Board keeps you connected to workplace trends and exciting new career prospects.

www.computer.org/jobs IEEE COMPUTER SOCIETY computer.org • +1 714 821 8380

STAFF

Editor Publications Portfolio Managers Cathy Martin Carrie Clark, Kimberly Sperka

Publications Operations Project Specialist Publisher Christine Anthony Robin Baldwin Publications Marketing Project Specialist Meghan O’Dell Senior Advertising Coordinator Debbie Sims Production & Design Carmen Flores-Garvey

Circulation: ComputingEdge (ISSN 2469-7087) is published monthly by the IEEE Computer Society. IEEE Headquarters, Three Park Avenue, 17th Floor, New York, NY 10016-5997; IEEE Computer Society Publications Office, 10662 Los Vaqueros Circle, Los Alamitos, CA 90720; voice +1 714 821 8380; fax +1 714 821 4010; IEEE Computer Society Headquarters, 2001 L Street NW, Suite 700, Washington, DC 20036. Postmaster: Send address changes to ComputingEdge-IEEE Membership Processing Dept., 445 Hoes Lane, Piscataway, NJ 08855. Periodicals Postage Paid at New York, New York, and at additional mailing offices. Printed in USA. Editorial: Unless otherwise stated, bylined articles, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in ComputingEdge does not necessarily constitute endorsement by the IEEE or the Computer Society. All submissions are subject to editing for style, clarity, and space. Reuse Rights and Reprint Permissions: Educational or personal use of this material is permitted without fee, provided such use: 1) is not made for profit; 2) includes this notice and a full citation to the original work on the first page of the copy; and 3) does not imply IEEE endorsement of any third-party products or services. Authors and their companies are permitted to post the accepted version of IEEE-copyrighted material on their own Web servers without permission, provided that the IEEE copyright notice and a full citation to the original work appear on the first screen of the posted copy. An accepted manuscript is a version which has been revised by the author to incorporate review suggestions, but not the published version with copy-editing, proofreading, and formatting added by IEEE. For more information, please go to: http://www.ieee.org/publications_standards/publications/rights/paperversionpolicy.html. Permission to reprint/republish this material for commercial, advertising, or promotional purposes or for creating new collective works for resale or redistribution must be obtained from IEEE by writing to the IEEE Intellectual Property Rights Office, 445 Hoes Lane, Piscataway, NJ 08854-4141 or [email protected]. Copyright © 2019 IEEE. All rights reserved. Abstracting and Library Use: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy for private use of patrons, provided the per- copy fee indicated in the code at the bottom of the first page is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923. Unsubscribe: If you no longer wish to receive this ComputingEdge mailing, please email IEEE Computer Society Customer Service at [email protected] and type “unsubscribe ComputingEdge” in your subject line. IEEE prohibits discrimination, harassment, and bullying. For more information, visit www.ieee.org/web/aboutus/whatis/policies/p9-26.html.

IEEE Computer Society Magazine Editors in Chief

Computer IEEE Security & Privacy Computing in Science David Alan Grier (Interim), David Nicol, University of Illinois & Engineering Djaghe LLC at Urbana-Champaign Jim X. Chen, George Mason University IEEE Micro IEEE Software IEEE Intelligent Systems Lizy Kurian John, University of Ipek Ozkaya, Software V.S. Subrahmanian, Dartmouth Texas, Austin Engineering Institute College IEEE MultiMedia IEEE Internet Computing IEEE Computer Graphics Shu-Ching Chen, Florida George Pallis, University of and Applications International University Cyprus Torsten Möller, University of Vienna IEEE Annals of the History IT Professional of Computing Irena Bojanova, NIST IEEE Pervasive Computing Marc Langheinrich, University of Gerardo Con Diaz, University of Lugano California, Davis

www.computer.org/computingedge 1 SEPTEMBER 2019 • VOLUME 5, NUMBER 9

THEME HERE 15 33 40 Flowcharting Compressing VR: Computer Templates Fitting Large Virtual Security as Environments within Civil Defense Limited Physical Space Software 8 The History of Software Engineering GRADY BOOCH 15 Flowcharting Templates PEGGY ALDRICH KIDWELL

Blockchain 19 Blockchain and the Economics of Food Safety NIR KSHETRI 23 Silver Bullet Talks with Nick Weaver GARY MCGRAW

Augmented and Virtual Reality 27 Co-creation and Risk-Taking—In Pursuit of New Technology for Human Augmentation: An Interview with Pranav Mistry MARC LANGHEINRICH AND NIGEL DAVIES 33 Compressing VR: Fitting Large Virtual Environments within Limited Physical Space KHRYSTYNA VASYLEVSKA AND HANNES KAUFMANN

Cyber-Physical Systems 40 Computer Security as Civil Defense MARILYN WOLF 45 Cyber-Physical Systems and Digital Twins in the Industrial Internet of Things CHRISTOS KOULAMAS AND ATHANASIOS KALOGERAS

Departments 4 Magazine Roundup 7 Editor’s Note: Software’s Evolution 45 72 Conference Calendar Cyber-Physical Systems and Digital Twins in the Industrial Internet of Things Subscribe to ComputingEdge for free at www.computer.org/computingedge. CS FOCUS

Magazine Roundup

Science & Engineering, an active hunting stability scheme is proposed for high-speed trains based on frame lateral vibration control. The stability of the vehicle is improved by exerting active control force on the front and rear frames of the bogie. First, a simplifi ed lateral vibration model of a single bogie is he IEEE Computer in static images and videos. established to the control sys- Society’s lineup of 12 However, videos contain addi- tem design. The feedback gain T peer-reviewed techni- tional temporal information and matrix is obtained according cal magazines covers cutting- convey a storyline. The authors to the linear quadratic optimal edge topics ranging from soft- of this article from the May control theory. Then, the mul- ware design and computer 2019 issue of Computer explore tibody dynamics model of the graphics to Internet comput- whether eye movement pat- vehicle is built using SIMPACK, ing and security, from scien- terns refl ect frame importance and the linear stability and tifi c applications and machine during video viewing and facili- straight running performance intelligence to visualization and tate video summarization. are analyzed under diff erent microchip design. Here are working confi gurations. Finally, highlights from recent issues. Computing in Science & the active control eff ects are ver- Engineering ifi ed by a scaled roller rig. Computer Simulation and IEEE Annals of the Human Eye Movements Experimental Study on History of Computing Reveal Video Frame the Active Stability of High- Importance Speed Trains “The Offi cial Response Is Human eye movements indicate In this article from the May/ Never Enough” important spatial information June 2019 issue of Computing in The Rockefeller Foundation

4 September 2019 Published by the IEEE Computer Society 2469-7087/19 © 2019 IEEE shipped two Apple II computers extended discussion of its under- IEEE Internet Computing with VisiCalc to the Tunisian Min- lying concepts, and review and istry of Agriculture to address a compare two previous case stud- Bots Acting Like Humans: grain shortage in the early 1980s. ies where promising results were Understanding and The foundation believed that Visi- obtained in terms of user comfort, Preventing Harm Calc would enable the speedy and engagement, and usability. They Bots are algorithmically driven complex analytical modeling nec- also discuss fi ndings of a novel entities that act like humans in con- essary to improve the management study conducted with geovisual- versations via Twitter, Facebook, and, consequently, the production ization experts, pointing to direc- chats, or Q&A sites. This article of grain resources. The foundation tions for improvement and future from the March/April 2019 issue also argued that VisiCalc would research. of IEEE Internet Computing stud- empower individuals in the Minis- ies how they can aff ect online con- try of Agriculture, improving their IEEE Intelligent Systems versations, provides a taxonomy own analytical thinking as they of harm that can be caused, and became more familiar with the Robust Authentication Using discusses how to prevent harm by modeling capabilities of the soft- Dorsal Hand Vein Images studying when abuses occur. ware. Even with the use of Visi- This article from the March/April Calc, Tunisia experienced violent 2019 issue of IEEE Intelligent Sys- IEEE Micro riots due to high bread prices after tems presents a robust dorsal the government’s removal of grain hand vein authentication system. A Hardware Accelerator for subsidies. This article from the A new method is proposed for the Tracing Garbage Collection January–March 2019 issue of IEEE region of interest extraction using Many workloads are written in gar- Annals of the History of Computing fi ngertips and fi nger valley key bage-collected languages and GC explores the narratives and uses points. Some new features and a consumes a signifi cant fraction of VisiCalc in the Tunisian Min- new classifi er are proposed based of resources for these workloads. istry of Agriculture in addressing on information set theory. Infor- The authors of this article from the this food crisis both before and mation set stems from a fuzzy May/June 2019 issue of IEEE Micro after the riots. set on representing the uncer- propose decreasing this overhead tainty in its attribute/information by moving GC into a small hard- IEEE Computer Graphics source values using the informa- ware accelerator that is located and Applications tion-theoretic entropy function. close to the memory controller and The new feature types include performs GC more effi ciently than Comfortable Immersive vein eff ective information, vein a CPU. They fi rst show a general Analytics with the energy feature, vein sigmoid fea- design of such a GC accelerator VirtualDesk Metaphor ture, Shannon transform feature, and describe how it can be inte- The VirtualDesk metaphor is an and composite transform feature. grated into both stop-the-world opportunity for more comfort- A classifi er called the improved and pause-free garbage collectors. able and effi cient immersive data Hanman classifi er is formulated They then demonstrate an end- exploration, using tangible inter- from training and test feature to-end RTL prototype, integrated action with the analyst’s physi- vectors using Frank t-norm and into a RocketChip RISC-V System- cal work desk and embodied the entropy function. The perfor- on-Chip (SoC) executing full Java manipulation of mid-air data rep- mance and robustness are evalu- benchmarks within JikesRVM resentations. In this article from ated on GPDS and BOSPHORUS running under Linux on FPGAs. the May/June 2019 issue of IEEE palm dorsal vein databases under The prototype performs the mark Computer Graphics and Applica- both the constrained and uncon- phase of a tracing GC at 4.2× the tions, the authors present an strained conditions. performance of an in-order CPU, www.computer.org/computingedge 5 MAGAZINE ROUNDUP

at just 18.5 percent of the area. By error in real-world uncontrolled systems are being used in sev- prototyping the design in a real experiments. eral developed countries to mini- system, they show that the acceler- mize the problems and limitations ator can be adopted without inva- IEEE Security & Privacy of the conventional paper-based sive changes to the SoC, and they approach. However, several devel- estimate its performance, area, Buddy’s Wearable Is Not Your oping countries like Pakistan have and energy. Buddy: Privacy Implications not advanced signifi cantly in adopt- of Pet Wearables ing the new healthcare standards IEEE MultiMedia As an increasingly prevalent class due to socioeconomic and techno- of consumer device, pet wearables logical constraints. Although there Multipoint Cooperative hold more privacy implications are some healthcare providers that Transmission for Virtual than might be initially apparent. are using their own EHRs, there is Reality in 5G New Radio Through analysis of privacy poli- no electronic repository of patients’ To meet the requirements of lower cies, the authors of this article electronic health data maintained latency and massive data trans- from the May/June 2019 issue of at the government level. In this arti- mission in virtual reality (VR) IEEE Security & Privacy show that cle from the May/June 2019 issue applications, a multipoint coop- more data is captured about own- of IT Professional, a cloud-based erative transmission mechanism ers than pets—and which data is architecture for the implementation is proposed for VR applications captured remains vague. of EHR for hospitals in Pakistan is over 5G New Radio. In particular, proposed. Adopting the proposed diff erent-quality coding levels are IEEE Software architecture will help improve utilized through the multipoint patient care, diagnostics, disease cooperative transmission to sup- Ethics Is a Software Design presentation, and round-the-clock port the immersive experience of Concern availability of electronic health users’ diff erent views. Read more The IEEE and Association for information. The development of in the January–March 2019 issue Computing Machinery (ACM) joint such a system will not only enable of IEEE MultiMedia. report “Software Engineering Code doctors and hospitals to exchange of Ethics” summarizes the respon- patient information but will also IEEE Pervasive Computing sibilities of software engineers as establish an electronic health data the following: Software engineers repository that subsequently can Area Occupancy Counting shall commit themselves to mak- be used for diverse purposes, such through Sparse Structural ing the analysis, specifi cation, as predictive diagnostics and per- Vibration Sensing design, development, testing, and sonalized medicine. This article from the January– maintenance of software a ben- March 2019 issue of IEEE Perva- efi cial and respected profession. sive Computing presents an indoor Read more in the May/June 2019 area occupancy counting system issue of IEEE Software. utilizing the ambient structural vibration induced by pedestrian IT Professional footsteps. The system achieves 99.55-percent accuracy in pedes- Cloud-Based Architecture to trian footsteps detection, 0.2 people Implement Electronic Health WWW.COMPUTER.ORG mean estimation error in pedes- Record (EHR) System in trian traffi c estimation, and 0.2 Pakistan /COMPUTINGEDGE area occupant activity estimation Electronic health record (EHR)

6 ComputingEdge September 2019 EDITOR’S NOTE

Software’s Evolution

e’ve come a long way from program- with Nick Weaver” delves into security problems ming mechanical computers using with blockchain technologies, including bugs in W punched cards. More than a century code. of developments in theory, hardware, program- Augmented-reality (AR) and virtual-reality (VR) ming languages, and methodologies has led to a systems are improving in part through software present in which software is integral to our daily innovations. IEEE Pervasive Computing’s “Co-cre- lives. This issue of ComputingEdge looks back on ation and Risk-Taking—In Pursuit of New Technol- the people, programs, and processes that carried ogy for Human Augmentation: An Interview with software engineering to where it is today. Pranav Mistry” discusses how software advances The author of IEEE Software’s “The History have enabled various new AR and VR products. of Software Engineering” details the evolution of IEEE Computer Graphics and Applications’ “Com- software engineering from its origins in the 19th pressing VR: Fitting Large Virtual Environments century to its modern form—and looks ahead to within Limited Physical Space” focuses on how to its future. “Flowcharting Templates,” from IEEE give users larger VR environments. Annals of the History of Computing, examines a Software is also integral to cyber-physical tool that early programmers used to draw diagrams systems (CPS). Two Computer articles on CPS containing inputs, outputs, operations, decisions, conclude this ComputingEdge issue. “Computer and connectors. Security as Civil Defense” calls for technology Software is the basis of many of today’s cut- and policy changes to protect against cyber-phys- ting-edge technologies, including blockchain. IT ical attacks. “Cyber-Physical Systems and Digital Professional’s “Blockchain and the Economics of Twins in the Industrial Internet of Things” intro- Food Safety” describes both the positive impact duces the concept of a digital twin, a virtual rep- blockchain could have on food supply chains and resentation that serves as the real-time digital the challenges involved in hiring blockchain devel- counterpart of a physical object or process and opers. IEEE Security & Privacy’s “Silver Bullet Talks addresses every instance for its total lifecycle.

The History of Software Engineering

Grady Booch

THE FIRST COMPUTERS were hu- 1963 or 1964 to distinguish her work time of Charles Babbage. However, man (and for the most part, women). from the hardware engineering taking the fundamentals of engineering The term “digital” didn’t enter cir- place in the nascent US space program. hold true, although, as we shall see, culation until around 1942, when each age discovers some new truth George Stibitz took the ideas from Software Engineering versus about engineering software. another George (Boole) and applied Computer Science them to electromechanical devices. It Grace Hopper suggested that pro- From the 19th to the 20th took another decade for John Tukey gramming is a practical art; Edsger Century: Human Computers to popularize the term “software.” Dijkstra called the art of program- Ada Lovelace was perhaps the fi rst What, then, of the term “software ming the art of organizing com- person to understand that program- engineering”? plexity; Donald Knuth referred to ming was a thing unto itself. Around programming as art because it pro- that same time, George Boole The Origins of the Term duced objects of beauty. I suspect brought a new way of thinking to the Many suggest it came from the 1968 that all of these observations are true, mathematicians and philosophers of NATO Conference on Software but what I like best is David Parnas’s the world, as expressed in his clas- Engineering, coined by Friedrich observation—much like Anthony sic book The Laws of Thought.2 At Bauer. Others have pointed to the 1966 Oettinger’s—that there is a distinc- the end of the 19th century, we saw letter by Anthony Oettinger in Com- tion between “computer science” and the fi rst human computers, such as munications of the ACM, wherein the other stuff that we do. This is not Annie Cannon, Henrietta Leavitt, he used the term “software engineer- unlike the distinction between chem- and others, the so-called “Harvard ing” to make the distinction between ical engineering and chemistry: both Computers” working for the astron- computer science and the building of are valid; both have their particular omer Edward Pickering. The way software-intensive systems.1 Even ear- sets of practices; both are very differ- these women organized their work lier, in the June 1965 issue of Comput- ent things. Software engineering is, was astonishingly similar to contem- ers and Automation, there appeared a in my experience, equally an art and porary agile development practices; classifi ed ad seeking a “systems soft- a science: it is the art of the practical. they too had a different way of think- ware engineer.” Engineering in all fi elds is all ing, very different for their time. All the data I have points to about the resolution of forces. In Around the start of the new cen- Margaret Hamilton as the person who civil engineering, one must consider tury, as computational problems be- fi rst coined the term. Having worked static and dynamic forces of a physi- gan to scale up and as mechanical on the SAGE (Semi-automatic Ground cal nature and of human nature. In aids to calculation became more reli- Environment) program, she became software engineering, one also must able and economical, the process of the lead developer for Skylab and balance cost, schedule, complexity, computing underwent further regi- Apollo while working at the Draper functionality, performance, reliabil- mentation. It was common to see Lab. According to an (unpublished) ity, and security, as well as legal and large rooms fi lled with human com- oral history, she began to use the term ethical forces. Computing technol- puters (again, mostly women), all “software engineering” sometime in ogy has certainly changed since the lined up in rows. Data would enter

one end; a computer would carry out George Boole’s ideas of binary logic from a machine’s hardware. This The History of Software one operation and then pass the re- to build the fi rst digital adder made led to one of the fi rst instances of sult to the next computer. This was of electromechanical relays. He abstraction in programming, the in effect the organic manifestation called this the K Model (the K rep- idea that one could devise a proof what today we’d call a pipeline resenting the kitchen table on which gramming language at a level closer Engineering architecture. he built it), and thus digital com- to human expression and further puting was born. The idea of build- from the machine’s hardware. Fur- From the Great Depression ing electromechanical mechanisms thermore, as Hopper realized, one Grady Booch to World War II: Birth of the for computation spread rapidly, and could use the computer itself to Electronic Computer it was not long thereafter that oth- translate those higher-order expres- Effi ciency and the reduction of costs ers realized that relays could be re- sions into machine language; the were then, as they are now, impor- placed by vacuum tubes, which were compiler was born. THE FIRST COMPUTERS were hu- 1963 or 1964 to distinguish her work time of Charles Babbage. However, tant to every industrial process. So, much, much faster. In the summer In the lamentations of World War man (and for the most part, women). from the hardware engineering taking the fundamentals of engineering we saw people such as Frederick of 1944, a serendipitous meeting II, the computing world split into The term “digital” didn’t enter cir- place in the nascent US space program. hold true, although, as we shall see, Taylor and Frank and Lillian between John von Neumann (who three pieces. In Germany, there was culation until around 1942, when each age discovers some new truth Gilbreth (of Cheaper by the Dozen3 George Stibitz took the ideas from Software Engineering versus about engineering software. fame) introduce time and motion another George (Boole) and applied Computer Science studies. The Gilbreths also promoted them to electromechanical devices. It Grace Hopper suggested that pro- From the 19th to the 20th the concept of process charts—the took another decade for John Tukey gramming is a practical art; Edsger Century: Human Computers direct predecessor of fl owcharts— Ada Lovelace was perhaps the to popularize the term “software.” Dijkstra called the art of program- Ada Lovelace was perhaps the fi rst to codify industrial processes. It did rst person to understand that What, then, of the term “software ming the art of organizing com- person to understand that program- not take long for these same ideas in engineering”? plexity; Donald Knuth referred to ming was a thing unto itself. Around manufacturing to jump over to the programming was a thing unto itself. programming as art because it pro- that same time, George Boole problems of computing. The Origins of the Term duced objects of beauty. I suspect brought a new way of thinking to the As the global Great Depres- Many suggest it came from the 1968 that all of these observations are true, mathematicians and philosophers of sion took hold, the Works Progress NATO Conference on Software but what I like best is David Parnas’s the world, as expressed in his clas- Administration was launched as Engineering, coined by Friedrich observation—much like Anthony sic book The Laws of Thought.2 At part of President Roosevelt’s New at the time was working on the Konrad Zuse. In a different time and Bauer. Others have pointed to the 1966 Oettinger’s—that there is a distinc- the end of the 19th century, we saw Deal. Gertrude Blanche was put in Manhattan Project) and Herman place, his work would have been the letter by Anthony Oettinger in Com- tion between “computer science” and the fi rst human computers, such as charge of the Mathematical Tables Goldstine (who was working at the center of gravity of modern comput- munications of the ACM, wherein the other stuff that we do. This is not Annie Cannon, Henrietta Leavitt, Project, the predecessor of today’s Ballistic Research Laboratory) led to ing, for he invented the fi rst high- he used the term “software engineer- unlike the distinction between chem- and others, the so-called “Harvard Handbook of Mathematical Func- their connection with John Mauchly order programming language as well ing” to make the distinction between ical engineering and chemistry: both Computers” working for the astron- tions. This was a work relief project (a professor at the Moore School of as the fi rst general-purpose stored computer science and the building of are valid; both have their particular omer Edward Pickering. The way that employed hundreds of out-of- Electrical Engineering). This caused computer. software-intensive systems.1 Even ear- sets of practices; both are very differ- these women organized their work work mathematicians and comput- ENIAC (Electronic Numerical In- In England, there was Bletchley lier, in the June 1965 issue of Comput- ent things. Software engineering is, was astonishingly similar to contem- ers (again, mostly women). Blanche’s tegrator and Computer) to come Park, where Alan Turing laid the ers and Automation, there appeared a in my experience, equally an art and porary agile development practices; work developed best practices for hu- into prominence and, more impor- theoretical foundations for modern classifi ed ad seeking a “systems soft- a science: it is the art of the practical. they too had a different way of think- man computing that were extremely tant, later yielded the First Draft computer science. However, it took ware engineer.” Engineering in all fi elds is all ing, very different for their time. sophisticated, including mechanisms of a Report on the EDVAC (Elec- an engineer—most notably Tommy All the data I have points to about the resolution of forces. In Around the start of the new cen- for error checking, which infl uenced tronic Discrete Variable Automatic Flowers—to turn those theories into Margaret Hamilton as the person who civil engineering, one must consider tury, as computational problems be- the way early punched-card comput- Computer).5 pragmatic solutions, and from this fi rst coined the term. Having worked static and dynamic forces of a physi- gan to scale up and as mechanical ing evolved. In 1940, Wallace Eckert And thus was born a new way of Colossus was born. Dorothy Du on the SAGE (Semi-automatic Ground cal nature and of human nature. In aids to calculation became more reli- published Punched Card Methods in thinking: the concept of a program- Boisson, a human computer, served Environment) program, she became software engineering, one also must able and economical, the process of Scienti c Computing,4 which turned mable, electronic computer with its as the primary operator of Colossus. the lead developer for Skylab and balance cost, schedule, complexity, computing underwent further regi- out to be, in a manner of speaking, instructions stored in memory. In her experience of leading a team Apollo while working at the Draper functionality, performance, reliabil- mentation. It was common to see the fi rst computing methodology or Grace Hopper, very much in of women who operated Colossus, Lab. According to an (unpublished) ity, and security, as well as legal and large rooms fi lled with human com- pattern language. the spirit of Ada Lovelace, then re- she codifi ed the ideas of workfl ow oral history, she began to use the term ethical forces. Computing technol- puters (again, mostly women), all As the winds of war were gather- discovered the idea that software that eventually were programmed “software engineering” sometime in ogy has certainly changed since the lined up in rows. Data would enter ing in Europe, George Stibitz applied could be a thing unto itself, distinct into the machine itself.

In the US, ENIAC, then later the Lyons Electronic Office (LEO). programming, and that work led EDVAC, dominated the scene. Ini- John Pinkerton, LEO’s chief engi- directly to the SAGE system. Con- tially, “programming” was carried neer, had the insight that software structed as a defense against the Soviet out by wiring up plugboards, a task could be treated as a component threat of sending nuclear-armed carried out by human computers unto itself. Realizing that many low- bombers over the Arctic, SAGE led to (yet again, mostly women), such as level programming tasks kept being a number of important innovations Kay Antonelli, Betty Snyder, Fran- written over and over again, he be- and issues, including ces Spence, Ruth Teitelbaum, and gan to bundle these common rou- Marlyn Wescoff. The way they or- tines into libraries, forming what • human–computer interfaces using ganized their work was reminiscent today we’d call an operating system CRT displays and light pens, of the Harvard Computers and thus, or framework, yet another rise in • the institutionalization of core in a manner of speaking, anticipated programming’s levels of abstraction. memory, and the structure of contemporary small Grace Hopper, Robert Bemer, • the problems associated with development teams focused on con- Jean Sammet, and others, influenced building very large soft- tinuous integration. by John Backus’s work, created ware systems in a distributed Cobol, another imperative language, environment. Post World War II: Rise of focused on the needs of businesses. Computing and Birth of With the introduction of IBM’s Software development was no lon- Software Engineering System/360, it was now possible to ger just a small part of bringing a The technical and economic forces write software for more than one spe- computer to life; it was increasingly that would shape modern software cific machine. IBM’s decision to un- a very expensive part, and certainly engineering further coalesced in the bundle software from hardware was the most important part. economic rise at the end of World a transformative event: now it was So there we were, in the second War II, where we began to see com- possible to develop software as a com- half of the 1960s, with the conflu- puting applied to problem domains ponent that had individual economic ence of three important events in the beyond the needs of conflict. Her- value. Around this time, organiza- history of software: man Goldstine built on the ideas tions such as SHARE emerged—a of the Gilbreths and, together with predecessor of today’s open source • the rise of commercial software John von Neumann, invented a no- software movement—giving a plat- as a product unto itself, tation that eventually morphed into form for third parties to write soft- • the complexities of defense what today we call flowcharts. Mau- ware for hardware they themselves systems such as SAGE, and rice Wilkes, David Wheeler, and didn’t control. In the UK, Dina • the rise of human-critical soft- Stanley Gill invented the concept of St. Johnson seized on the business op- ware as demanded by the US subroutines, thus again raising com- portunity and established England’s space program. puting’s levels of abstraction, and first software services business. This making manifest the pragmatics of made manifest the idea that one could This is the context in which Margaret algorithmic decomposition. John outsource software development to Hamilton coined the term “software Backus took Grace Hopper’s early teams with particular computing engineering” and in which NATO work and went further, yielding For- skills a company with specific domain declared that there was a “software tran, the high-level imperative lan- knowledge might not possess. crisis.” guage that would dominate scientific A sort of programming priest- computing for years to come. Rise of the Cold War: hood was the common form of The commercial world, now un- Coming of Age software development at the time, leashed at the end of global conflict, The rise of the Cold War between the and—in its time—it made a great turned to automatic aids to com- US and the Soviet Union generated deal of sense. In that era, the cost of puting: opportunities for growth another set of forces that pushed soft- a computer was greater than the cost quickly outran the cost and reliabil- ware engineering to come of age. Tom of its programmers, and as such, com- ity of human computers. The first Kilburn and his work with Whirlwind puters would be kept apart in a climate- computer put in commercial use was explored the possibilities of real-time controlled room. Much like the

10110 IEEE SOFTWAREComputingEdge | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE September 2019 ON COMPUTING ON COMPUTING

In the US, ENIAC, then later the Lyons Electronic Office (LEO). programming, and that work led pipelined methods of the punched- the ideas of coupling and cohesion artifacts and the processes of soft- EDVAC, dominated the scene. Ini- John Pinkerton, LEO’s chief engi- directly to the SAGE system. Con- card era, analysts would take re- applied as a mechanism for algorith- ware development. tially, “programming” was carried neer, had the insight that software structed as a defense against the Soviet quirements and pass them on to mic decomposition. Edsger Dijkstra This led to the fi rst generation of out by wiring up plugboards, a task could be treated as a component threat of sending nuclear-armed programmers, who would use their took a more formal approach, giving software engineering methodologies. carried out by human computers unto itself. Realizing that many low- bombers over the Arctic, SAGE led to fl owcharts to devise algorithms. us an important tool for software Doug Ross, Larry Constantine, Ed (yet again, mostly women), such as level programming tasks kept being a number of important innovations These programmers in turn would engineering: the idea of structured Yourdon, Tom DeMarco, Chris Gane, Kay Antonelli, Betty Snyder, Fran- written over and over again, he be- and issues, including pass their programs on to keypunch- programming. Trish Sarson, and Michael Jackson— ces Spence, Ruth Teitelbaum, and gan to bundle these common rou- ers. The resulting card decks would Around the same time, there was to name just a few—developed meth- Marlyn Wescoff. The way they or- tines into libraries, forming what • human–computer interfaces using be given to the computer operators important work by researchers such ods for structured analysis and design ganized their work was reminiscent today we’d call an operating system CRT displays and light pens, working in their sacred space. as Robert Floyd and Tony Hoare, that took over the fi eld. Adding the of the Harvard Computers and thus, or framework, yet another rise in • the institutionalization of core It wasn’t until the economics of who devised formal ways to express work by Michael Fagan (on software in a manner of speaking, anticipated programming’s levels of abstraction. memory, and computers changed with the rise of and reason about programs—a true inspections), James Martin (on infor- the structure of contemporary small Grace Hopper, Robert Bemer, • the problems associated with minicomputers and microcomput- attempt to connect computer science mation engineering), John Backus (on development teams focused on con- Jean Sammet, and others, influenced building very large soft- ers, together with the realization of and software engineering. Niklaus functional programming), and Leslie tinuous integration. by John Backus’s work, created ware systems in a distributed Christopher Strachey’s idea of time Cobol, another imperative language, environment. sharing, that this model of develop- Post World War II: Rise of focused on the needs of businesses. ment changed. This is also the con- Computing and Birth of With the introduction of IBM’s Software development was no lon- text in which the basic principles of Software Engineering System/360, it was now possible to ger just a small part of bringing a software project management came The introduction of personal The technical and economic forces write software for more than one spe- computer to life; it was increasingly alive, as Fred Brooks so profoundly computing changed software that would shape modern software cific machine. IBM’s decision to un- a very expensive part, and certainly described in The Mythical Man engineering further coalesced in the bundle software from hardware was the most important part. Month.6 Brooks made the important engineering. economic rise at the end of World a transformative event: now it was So there we were, in the second insight that software engineering War II, where we began to see com- possible to develop software as a com- half of the 1960s, with the conflu- was not just a technical process but puting applied to problem domains ponent that had individual economic ence of three important events in the also a very human process. beyond the needs of conflict. Her- value. Around this time, organiza- history of software: The economic rise after World Wirth invented Pascal, an effort to Lamport (on best practices for distrib- man Goldstine built on the ideas tions such as SHARE emerged—a War II, given a further boost by the explicitly support best practices in uted computing), software engineer- of the Gilbreths and, together with predecessor of today’s open source • the rise of commercial software Cold War, led inevitably to a coun- structured programming. Ole Dahl ing entered in its fi rst golden age. John von Neumann, invented a no- software movement—giving a plat- as a product unto itself, terculture shift, as wonderfully de- and Kristen Nygaard had the outra- tation that eventually morphed into form for third parties to write soft- • the complexities of defense scribed by John Markoff in What the geously wonderful idea that yielded The Eighties and Onward: what today we call flowcharts. Mau- ware for hardware they themselves systems such as SAGE, and Dormouse Said.7 The introduction the invention of Simula, a language Golden Age rice Wilkes, David Wheeler, and didn’t control. In the UK, Dina • the rise of human-critical soft- of personal computing not only was that was object-oriented rather than However, a sea change was com- Stanley Gill invented the concept of St. Johnson seized on the business op- ware as demanded by the US fueled by technical and social ad- algorithmic in nature. ing. Owing to the growing prob- subroutines, thus again raising com- portunity and established England’s space program. vances but also changed the nature Winston Royce then brought to lems of software quality, the rise of puting’s levels of abstraction, and first software services business. This of software engineering. Now, pro- us the idea of a formal software de- ultralarge software-intensive sys- making manifest the pragmatics of made manifest the idea that one could This is the context in which Margaret grammers were more expensive than velopment process. Although he is tems, the globalization of software, algorithmic decomposition. John outsource software development to Hamilton coined the term “software computers, and it was economically much criticized for what we today and the shift from programs to dis- Backus took Grace Hopper’s early teams with particular computing engineering” and in which NATO viable to put computers everywhere. call the waterfall process, his meth- tributed systems, new approaches work and went further, yielding For- skills a company with specific domain declared that there was a “software This led to Allen Newell speaking of odology was actually quite advanced: were needed. Ole Dahl and Kristen tran, the high-level imperative lan- knowledge might not possess. crisis.” the enchanted world that computing he spoke of iterative development, Nygaard’s ideas of object-oriented guage that would dominate scientific A sort of programming priest- made possible, as described in his the importance of prototyping, and programing gave rise to a completely computing for years to come. Rise of the Cold War: hood was the common form of wonderful essay “Fairytales.”8 the value of artifacts beyond source new class of programming lan- The commercial world, now un- Coming of Age software development at the time, code itself. Coupled with David guages: Smalltalk, C with Classes, leashed at the end of global conflict, The rise of the Cold War between the and—in its time—it made a great From the Sixties to the Parnas’s ideas of information hiding, Ada, and many others. Although turned to automatic aids to com- US and the Soviet Union generated deal of sense. In that era, the cost of Eighties: Maturation Barbara Liskov’s ideas of abstract data structured methods were useful, puting: opportunities for growth another set of forces that pushed soft- a computer was greater than the cost Software engineering was forced to types, and Peter Chen’s approaches to they were not altogether suffi cient quickly outran the cost and reliabil- ware engineering to come of age. Tom of its programmers, and as such, com- mature. Larry Constantine was per- entity–relationship modeling, all of for these new languages, and thus ity of human computers. The first Kilburn and his work with Whirlwind puters would be kept apart in a climate- haps the fi rst to introduce the con- a sudden the fi eld had a vibrant set was born the second golden age of computer put in commercial use was explored the possibilities of real-time controlled room. Much like the cept of modular programming, with of ideas whereby to expresses the software engineering.

110 IEEE SOFTWARE | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE www.computer.org/computingedge SEPTEMBER/OCTOBER 2018 | IEEE SOFTWARE 11111 ON COMPUTING

Ada—the US Department of influenced the creation of Java. Alan design patterns and applied them to Defense’s solution to the problem of Cooper’s Visual Basic invigorated organizational patterns. Mary Shaw the proliferation of programming the Windows platform. Brad Cox’s and David Garlan furthered these languages and the changing nature invention of Objective-C had a tre- concepts in their work on software of software itself—proved to be a mendous effect on NeXT and Apple. architecture styles. catalyst for this era. Some of the Furthermore, Cox’s ideas surround- Two other lasting developments structured-method pioneers pivoted. ing component-based engineering— of note took place in this era. First, James Martin and Ed Yourdon cel- another rise in software engineering’s Eric Raymond evolved an important ebrated object-oriented approaches; levels of abstraction—led directly to legal framework for open source, others brought completely new ideas Microsoft’s OLE (object linking and making it possible to scale the ideas to the field: Stephen Mellor, Peter embedding) and COM (Component first seen in the early days of com- Coad, and Rebecca Wirfs-Brock, Object Model), which were the pre- puting, with SHARE. Kiran Karnik, to name a few. The Booch Method decessors of today’s microservice working in India, established the grew out of this primordial soup of architecture. first outsourcing contracts between ideas, as did Jim Rumbaugh’s OMT General Electric and India, thus lay- (object-modeling technique) and Ivar The Nineties and the ing the foundation for a transfor- Jacobson’s Objectory. Sensing an Millennium: Era of Disruptions mative economic shift in software opportunity to bring the market to But another change was in the wind: development. some common best practices, the the Internet. Suddenly we had a very With the Internet well in place three of us united to produce what rich, as of yet unexplored, platform. and organizations beginning to em- became the Unified Modeling Lan- In it, distribution was the default, brace its possibilities, mobile de- guage (made an Object Management consumers were the new stakehold- vices hit the scene, and the world Group standard in 1987) and then ers, users were measured in the changed yet again. The foundation the Unified Process. billions, and participants in this eco- laid by Brad Cox for component- Other aspects of software engi- system were not necessarily reliable based engineering morphed into neering came into play—for example, or trustworthy. We were no longer service-based architectures, which building programs; we were building in turn morphed into microser- • Philippe Kruchten’s 411 View systems, often made of parts that we vice architectures, evolving as the Model of software architecture; no longer controlled. Web’s technical infrastructure grew • Barry Boehm’s work in software By this time, there existed a rela- in fits and starts. New program- economics, together with his tively stable and economically very ming languages came and went spiral model; vibrant software engineering com- (and still do), but only a handful • Vic Basili and his ideas on em- munity. Independent companies still dominate—for example, Java, pirical software engineering; existed to serve the needs of require- JavaScript, Python, C11, C#, PHP, • Capers Jones and software ments analysis, design, development, and Swift. Computing moved from metrics; testing, and configuration manage- the mainframe to the datacenter • Harlan Mills and clean-room ment. Continuous integration with to the cloud, but coupled with mi- software engineering; incremental and iterative develop- croservices, the Internet evolved to • Donald Knuth’s literate ment was becoming the norm. The become the de facto computing plat- programming; and Gang of Four—Eric Gamma, Rich- form. Company-specific ecosystems • Watts Humphrey and his ard Helm, Ralph Johnson, and John rose like walled cathedrals: Ama- Capability Maturity Model. Vlissides—gave us another bump zon, Google, Microsoft, Facebook, up in software engineering levels of Salesforce, IBM—really, every eco- Simultaneously, these software en- abstraction in the form of the de- nomically interesting company built gineering concepts influenced the sign pattern. Institutionalized by its own fortress. development of an entirely new gen- the Hillside Group in 1993, pat- This was now the age of the eration of programming languages. terns heavily influenced that genera- framework. Long gone were the re- Bjarne Stroustrup’s C with Classes tion of software development. Jim ligious battles over operating sys- grew up to become C11, which later Coplien took the ideas of software tems. Now, battles were fought

12112 IEEE SOFTWAREComputingEdge | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE September 2019 ON COMPUTING ON COMPUTING

Ada—the US Department of influenced the creation of Java. Alan design patterns and applied them to along the lines of the veritable Knowledge, fi rst released in 2004 We as an industry have not yet Defense’s solution to the problem of Cooper’s Visual Basic invigorated organizational patterns. Mary Shaw explosion of open source frame- and whose current version was re- built enough of these AI systems to the proliferation of programming the Windows platform. Brad Cox’s and David Garlan furthered these works: Bootstrap, jQuery, Apache, leased in 2014)9 and the Systems fully understand how they might languages and the changing nature invention of Objective-C had a tre- concepts in their work on software NodeJS, MongoDB, Brew, Cocoa, Engineering Body of Knowledge impact the software engineering of software itself—proved to be a mendous effect on NeXT and Apple. architecture styles. Caffe, Flutter—truly a dizzying, by INCOSE10 exist as an attempt process, as they most certainly will. catalyst for this era. Some of the Furthermore, Cox’s ideas surround- Two other lasting developments ever-growing collection. to codify software engineering best What is the best lifecycle for sys- structured-method pioneers pivoted. ing component-based engineering— of note took place in this era. First, Today, we no longer build just practices. tems whose components we teach, James Martin and Ed Yourdon cel- another rise in software engineering’s Eric Raymond evolved an important programs or monolithic systems; rather than program? How do we ebrated object-oriented approaches; levels of abstraction—led directly to legal framework for open source, we build apps that live on the edge The Decade Ahead: Big Data test them? Where does confi guration others brought completely new ideas Microsoft’s OLE (object linking and making it possible to scale the ideas and interact with these distributed and the New Season of AI management fi t in when data for to the field: Stephen Mellor, Peter embedding) and COM (Component first seen in the early days of com- systems. Agile methods—in vari- But software engineering is about to ground truth is perhaps more impor- Coad, and Rebecca Wirfs-Brock, Object Model), which were the pre- puting, with SHARE. Kiran Karnik, ous personality-led variations— undergo yet another change. tant than the neural network itself? to name a few. The Booch Method decessors of today’s microservice working in India, established the have fl owered and have become the The foundations of AI have How do we best architect systems grew out of this primordial soup of architecture. first outsourcing contracts between dominant method, in name if not been around for decades. Over the with parts whose operation we can- ideas, as did Jim Rumbaugh’s OMT General Electric and India, thus lay- necessarily perfectly in practice. decades, we’ve seen at least four not explain or fully trust? (object-modeling technique) and Ivar The Nineties and the ing the foundation for a transfor- Hirotaka Takeuchi and Ikujiro Non- seasons of AI, manifested by the ex- This will be the challenge of Jacobson’s Objectory. Sensing an Millennium: Era of Disruptions mative economic shift in software aka coined the term “Scrum” in 1986 treme rising and falling of fortunes. the next generation of women and opportunity to bring the market to But another change was in the wind: development. as an agile approach to product de- What we have now feels different. men who keep software engineer- some common best practices, the the Internet. Suddenly we had a very With the Internet well in place velopment. Later, Ken Schwaber and The growth of big data, the abun- ing vibrant. Add to this mix the three of us united to produce what rich, as of yet unexplored, platform. and organizations beginning to em- (independently) Jeff Sutherland and dance of raw computational power, growth of quantum computing, became the Unified Modeling Lan- In it, distribution was the default, brace its possibilities, mobile de- Jeff McKenna codifi ed those prin- guage (made an Object Management consumers were the new stakehold- vices hit the scene, and the world ciples in the domain of software de- Group standard in 1987) and then ers, users were measured in the changed yet again. The foundation velopment. Around that same time, the Unified Process. billions, and participants in this eco- laid by Brad Cox for component- Kent Beck introduced the concept of Other aspects of software engi- system were not necessarily reliable based engineering morphed into Extreme Programming, while Ralph How do we best architect systems neering came into play—for example, or trustworthy. We were no longer service-based architectures, which Johnson further developed the idea building programs; we were building in turn morphed into microser- of refactoring (which Martin Fowler with parts whose operation we cannot • Philippe Kruchten’s 411 View systems, often made of parts that we vice architectures, evolving as the further codifi ed in his book Refac- explain or fully trust? Model of software architecture; no longer controlled. Web’s technical infrastructure grew toring: Improving the Design of Ex- • Barry Boehm’s work in software By this time, there existed a rela- in fits and starts. New program- isting Code). In February 2001, 17 economics, together with his tively stable and economically very ming languages came and went agilists met in Snowbird, Utah, and spiral model; vibrant software engineering com- (and still do), but only a handful penned the Agile Manifesto. The ag- • Vic Basili and his ideas on em- munity. Independent companies still dominate—for example, Java, ile approach to software development and the presence of these walled augmented reality, virtual real- pirical software engineering; existed to serve the needs of require- JavaScript, Python, C11, C#, PHP, entered the mainstream. cathedrals have given rise to eco- ity, and the spread of computing to • Capers Jones and software ments analysis, design, development, and Swift. Computing moved from Software engineering had en- nomic forces that have made fi rst every human, every device, and metrics; testing, and configuration manage- the mainframe to the datacenter tered another golden age. Git and statistical approaches and now neu- every nook and cranny of the earth • Harlan Mills and clean-room ment. Continuous integration with to the cloud, but coupled with mi- GitHub emerged; Joel Spolsky gave ral networks viable. Most of these and beyond, and this makes for a software engineering; incremental and iterative develop- croservices, the Internet evolved to us Stack Overfl ow; Jeannette Wing modern advances have been in what tremendously exciting time to be in • Donald Knuth’s literate ment was becoming the norm. The become the de facto computing plat- introduced the idea of computa- I call “signal AI”: the use of neu- computing. programming; and Gang of Four—Eric Gamma, Rich- form. Company-specific ecosystems tional thinking; Andrew Shafer and ral networks and gradient descent In the history of computing, we • Watts Humphrey and his ard Helm, Ralph Johnson, and John rose like walled cathedrals: Ama- Patrick Debois brought us the idea of to do complex pattern matching in have seen the progression of sys- Capability Maturity Model. Vlissides—gave us another bump zon, Google, Microsoft, Facebook, DevOps; the full stack developer be- images, video, and audio signals. tems from mathematical, to sym- up in software engineering levels of Salesforce, IBM—really, every eco- came a thing; the Internet of Things The early outcomes are impressive, bolic, to what Yuval Harari calls Simultaneously, these software en- abstraction in the form of the de- nomically interesting company built appeared in every imaginable corner as evidenced in IBM’s Watson and “imagined realities.” Some soft- gineering concepts influenced the sign pattern. Institutionalized by its own fortress. of the world. Now, all of a sudden, Google’s AlphaGo. In many ways, ware is like building a doghouse: development of an entirely new gen- the Hillside Group in 1993, pat- This was now the age of the everyone could learn how to code we are just beginning to understand you just do it, without any blue- eration of programming languages. terns heavily influenced that genera- framework. Long gone were the re- (and many did). what is possible and where the lim- prints, and if you fail, you can Bjarne Stroustrup’s C with Classes tion of software development. Jim ligious battles over operating sys- Artifacts such as SWEBOK its of these connectionist models of always get another dog. Other soft- grew up to become C11, which later Coplien took the ideas of software tems. Now, battles were fought (Software Engineering Body of computation live. ware is like building a house or a

112 IEEE SOFTWARE | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE www.computer.org/computingedge SEPTEMBER/OCTOBER 2018 | IEEE SOFTWARE 11313 ON COMPUTING

6. F. Brooks, The Mythical Man Month: ABOUT THE AUTHOR Essays on Software Engineering, Addison-Wesley, 1975. 7. J. Markoff, What the Dormouse GRADY BOOCH is an IBM Fellow and one of UML’s original authors. Said, Penguin, 2005. Contact him at [email protected] and on Twitter as @grady_booch. 8. A. Newell, “Fairytails,” Carnegie Mellon Univ., 1976; http://repository .cmu.edu/cgi/viewcontent.cgi?article 3321&contextcompsci. 9. The Guide to the Software Engineer- ing Body of Knowledge, IEEE Com- puter Soc., 2004. 10. Guide to the Systems Engineering Body of Knowledge (SEBoK), Int’l high-rise: the economics are differ- keynote at the 2015 International Council Systems Eng., 2012. ent, the scale is different, and the Conference on Software Engineering cost of failure is higher. Much of in Florence, software is the invisible modern software engineering is like writing that whispers the stories of renovating a city: there is room for possibility to our hardware. This article originally appeared in radical innovation, but you are con- And you are the storytellers. IEEE Software, vol. 35, no. 5, 2018. strained by the past as well as the cultural, social, ethical, and moral Acknowledgments context of everyone else in the city. This essay is based on my ACM Learn-

One thing I do know. No mat- ing Webinar of the same title, broadcast July–September 2016 IEEE MultiMedia

ter the medium or the technology on 25 April 2018. A recording is available http://www.computer.org

or the domain, the fundamentals at https://www.youtube.com/watch?v 2016 july–september of sound software engineering will QUz10Z1AfLc. A more extensive bibliog- ❚

always apply: craft sound abstrac- raphy is available as a Web Extra at https:// Quality Modeling

tions; maintain a clear separation extras.computer.org/extra/mso2018050108s1 of concerns; strive for a balanced .pdf. distribution of responsibilities; seek Volume 23 Number 3 simplicity. The pendulum will con- References tinue to swing—symbolic to con- 1. A. Oettinger, “President’s Letter to mult-22-03-c1 Cover-1 July 12, 2016 4:40 PM nectionist to quantum models of the ACM Membership,” Comm. IEEE MultiMedia serves the computation; intentional architec- ACM, vol. 9, no. 8, 1966. community of scholars, ture or emergent architecture; edge 2. G. Boole, The Laws of Thought, developers, practitioners, or cloud computing—but the funda- Walton and Maberly, 1854. and students who are mentals will stand. 3. F.B. Gilbreth Jr. and E. Gilbreth interested in multiple Carey, Cheaper by the Dozen, media types and work in Thomas Y. Crowell, 1948. fields such as image and have named a few dozen women 4. W.J. Eckert, Punched Card Methods video processing, audio and men who have shaped soft- in Scienti c Computing, Thomas J. analysis, text retrieval, and I ware engineering, but please Watson Astronomical Computing data fusion. know that there are thousands more Bureau, Columbia Univ., 1940. who have made software engineering 5. J. von Neumann, First Draft of a Read It Today! what it is today, each by his or her Report on the EDVAC, US Army own unique contributions. And so it Ordinance Department and Univ. www.computer.org will be for the future of software en- Pennsylvania Moore School of Elec- /multimedia gineering. As I said in closing in my trical Eng., 1945.

14114 IEEE SOFTWAREComputingEdge | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE September 2019 41mahc01-kidwell-2893719.3d (Style 5) 24-06-2019 21:53

Anecdotes

Flowcharting Templates

Peggy Aldrich Kidwell National Museum of American History Smithsonian Institution

Editor: David Walden, [email protected]

& THE SMITHSONIAN HAS begun to post online Betty Holberton, who worked as a programmer on descriptions of objects in its collections. Figures the ENIAC, took the flowcharting techniques devel- and accounts are now available for roughly 2500 oped at Penn to the first American commercial objects from the math and computer collec- computer company, Eckert Mauchly. By 1949, tions—perhaps a quarter of the total. Our focus EMCC employee Grace Murray Hopper and her has been on small objects that are easily exam- colleagues were preparing flow charts for the ined. Hence I recently cataloged the two dozen company’s UNIVAC computer under Holberton’s flowcharting templates in the collections. This direction (see Figure 1). The following year, EMCC brief account is intended to encourage readers copyrighted a set of “Flow Chart Symbols.” to add their own reminiscences to the online The technique proved sufficiently successful records, creating a much richer resource. for computer companies to introduce and distrib- Early commercial computer manufacturers, ute small rectangular plastic templates with the most notably Remington Rand UNIVAC (previ- symbols used on flowcharts cut out of them. ously Eckert Mauchly Computer Company, later The company name also featured prominently. Remington Rand UNIVAC, now UNISYS) and IBM, The earliest flowcharting template in the Smithso- faced the challenge of teaching both potential cus- nian collections dates from about 1955 and was tomers and budding programmers about the logi- distributed, appropriately enough, by Remington cal structure of computer programs. Toward that Rand Univac, the corporate descendent of EMCC end, they used diagrams called flowcharts, which (see Figure 2). The symbols on it are not labeled had separate symbols to represent forms of input but are an expanded version of those used in and output, operations, decisions, connectors, earlier Eckert-Mauchly diagrams. The template is and directions of logical flow. Thomas Haigh, Mark shown in a short, undated, movie entitled Reming- Priestley, and Crispin Rope, in their book ENIAC in ton Rand Presents UNIVAC. It also appeared on the Action,1 have looked specifically at the diagrams cover of a company publication known as the used by those programming the pioneering ENIAC Programmer in March–April, 1956 (see Figure 3). computer at the University of Pennsylvania at Other computer manufacturers also soon about the time of World War II. As Nathan issued flowcharting. Examples in the Smithsonian Ensmenger has noted in 2016 in Information and collections come from the Electrodata Division of 2 Technology article on the history flowcharts, Burroughs, Burroughs itself, IBM, RCA, Honeywell, the Massachusetts firm of Sprague Electric, RCA, Digital Object Identifier 10.1109/MAHC.2019.2893719 Bunker-Ramo Corporation, the Bell System, and Date of current version 14 March 2019. Control Data Corporation. Use was not confined

January-March 2019 U.S. Government work not protected by U.S. copyright. 1058-6180 ß 2019 IEEE 2469-7087/19 © 2019 IEEE Published by the IEEE Computer Society September 2019 55 15 41mahc01-kidwell-2893719.3d (Style 5) 24-06-2019 21:53

Anecdotes

Figure 1. Flowchart drawn by Helen M. Diehl for engineer Herbert F. Mitchell at the Eckert-Mauchly Computer Corporation in September of 1949. Grace Murray Hopper Collection, 1944–1965, Archives Center, National Museum of American History. Smithsonian Image AC0324-0000042.

to American makers of mainframe computers. reached internationally to include one distributed For example, one IBM flowcharting template was by the British firm of ICL. designed for users of the IBM 402 and IBM 403 Once templates had become a standard tool electronic accounting machines. Somewhat later, of computer programming, they also were sold the Educational Services affiliate of minicomputer manufacturer Digital Equipment Corporation distributed flowcharting templates. The technique of flowcharting—and the design of templates—also

Figure 2 . Remington Rand UNIVAC Flowcharting Template, Figure 3. Cover of the Remington Rand publication about 1955, Gift of Joan P. Nichols. Mathematics Collections, The Programmer, vol. 3 #2, March–April, 1956. National Museum of American History. Smithsonian Image Grace Murray Hopper Collection. Smithsonian Image 2003-20240. AC0554-000002.

IEEE Annals of the History of Computing 16 56 ComputingEdge September 2019 41mahc01-kidwell-2893719.3d (Style 5) 24-06-2019 21:53 41mahc01-kidwell-2893719.3d (Style 5) 24-06-2019 21:53

Anecdotes

by makers of drawing instruments. Two exam- less clear how people actually used them—and ples in the Smithsonian collections were made in surviving objects are mute on the matter! the U.S., distributed by the German firm of Mars The objects are described online at the Staedler, and used in Canada. By the 1980s, such museum’s website (see http://americanhistory. templates also might be distributed as give- si.edu/collections/object-groups/flowcharting- aways by prospective employers. templates). These pages have space for com- This profusion of flow charts led to attempts at ments, and I urge those wishing to share their standardization. In the early 1960s, the American experiences with flow charts and flowcharting Standards Association established sectional com- templates to add their comments and memories. mittee X3 to develop standards for computers and information processing. The first standard & REFERENCES developed by the subcommittee on problem description and analysis concerned flowcharting 1. T. Haigh, M. Priestley, and C. Rope, ENIAC in Action: symbols. A proposal circulated in 1963 was app- Making and Remaking the Modern Computer, roved as ASA Standard X3.5-1965 (the 1965 version Cambridge, MA, USA: MIT Press, 2016. of the fifth standard developed by committee X3), 2. N. Ensmenger, ‘The Multiple Meanings of a Flowchart’ and was soon revised as X3.5-1966 and then as Info. Culture: A J. History, vol. 41, no 10, 2016, X3.5-1970. Templates sometimes refer to the stan- pp. 321–351. dard used in creating them, offering a clue as to the date on which they were designed. Peggy Aldrich Kidwell is Curator of Mathematics Surviving flowcharting templates well repre- at the Smithsonian Institution’s National Museum of sent the emergence of commercial computer American History. She has never used a flowcharting Figure 1. Flowchart drawn by Helen M. Diehl for engineer Herbert F. Mitchell at the Eckert-Mauchly Computer Corporation manufacturers, early attempts to provide train- template herself, and would welcome comments on in September of 1949. Grace Murray Hopper Collection, 1944–1965, Archives Center, National Museum of American ing for programmers, and efforts to develop those shown on the NMAH website. Contact her at History. Smithsonian Image AC0324-0000042. standards within information processing. It is [email protected]. to American makers of mainframe computers. reached internationally to include one distributed For example, one IBM flowcharting template was by the British firm of ICL. This article originally appeared in designed for users of the IBM 402 and IBM 403 Once templates had become a standard tool IEEE Annals of the History of Computing, vol. 41, no. 1, 2019. electronic accounting machines. Somewhat later, of computer programming, they also were sold IEEE COMPUTER GRAPHICS AND APPLICATIONS APPLICATIONS AND GRAPHICS COMPUTER IEEE IEEE COMPUTER GRAPHICS AND APPLICATIONS APPLICATIONS AND GRAPHICS COMPUTER IEEE IEEE COMPUTER GRAPHICS AND APPLICATIONS APPLICATIONS AND GRAPHICS COMPUTER IEEE the Educational Services affiliate of minicomputer APPLICATIONS AND GRAPHICS COMPUTER IEEE

November/December 2016 manufacturer Digital Equipment Corporation dis- July/August 2016 September/October 2016 January/February 2017 tributed ﬂowcharting templates. The technique of

ﬂowcharting—and the design of templates—also November/December 2016 September/October 2016 January/February 2017January/February

July/August 2016July/August Qualit Assessment and Defense Quality Assessment and Perception in Computer Graphics Computer in Perception and Assessment Quality Perception Applications Element Human Water, the Sky, and

in Computer Graphics Visualization Data Sports Defense Applications

VOLUME 36 NUMBER 4 NUMBER 36 VOLUME VOLUME 37 NUMBER 1 37 NUMBER VOLUME VOLUME 36 NUMBER 5 NUMBER 36 VOLUME VOLUME 36 NUMBER 6 NUMBER 36 VOLUME

c1.indd 1 12/14/16 12:21 PM

c1.indd 1 6/22/16 1:20 PM c1.indd 1 8/22/16 2:59 PM c1.indd 1 10/24/16 3:44 PM

CG& www.computer.org/cgaA Figure 2 . Remington Rand UNIVAC Flowcharting Template, Figure 3. Cover of the Remington Rand publication IEEE Computer Graphics and Applications bridges the theory and practice of computer graphics. Subscribe to CG&A and about 1955, Gift of Joan P. Nichols. Mathematics Collections, The Programmer, vol. 3 #2, March–April, 1956. • stay current on the latest tools and applications and gain National Museum of American History. Smithsonian Image Grace Murray Hopper Collection. Smithsonian Image invaluable practical and research knowledge, 2003-20240. AC0554-000002. • discover cutting-edge applications and learn more about the latest techniques, and • benefit fromCG&A ’s active and connected editorial board.

IEEE Annals of the History of Computing January-March 2019 57 56 www.computer.org/computingedge 17 PURPOSE: The IEEE Computer Society 2019 BOARD OF GOVERNORS MEETING is the world’s largest association 25 – 27 October: Hilton McLean Tysons Corner, McLean, VA of computing professionals and is EXECUTIVE STAFF the leading provider of technical information in the field. Executive Director: Melissa A. Russell; Director, Governance & MEMBERSHIP: Members receive the monthly magazine Associate Executive Director: Anne Marie Kelly; Director, Finance Computer, discounts, and opportunities to serve (all activities & Accounting: Sunny Hwang; Director, Information Technology & are led by volunteer members). Membership is open to all IEEE Services: Sumit Kacker; Director, Marketing & Sales: Michelle Tubb; members, affiliate society members, and others interested in the Director, Membership Development: Eric Berkowitz computer field. OMBUDSMAN: Email [email protected] COMPUTER SOCIETY OFFICES COMPUTER SOCIETY WEBSITE: www.computer.org Washington, D.C.: 2001 L St., Ste. 700, Washington, D.C. EXECUTIVE COMMITTEE 20036-4928; Phone: +1 202 371 0101; Fax: +1 202 728 9614; President: Cecilia Metra; President-Elect: Leila De Floriani; Past Email: [email protected] President: Hironori Kasahara; First VP: Forrest Shull; Second VP: Los Alamitos: 10662 Los Vaqueros Cir., Los Alamitos, CA 90720; Avi Mendelson; Secretary: David Lomet; Treasurer: Dimitrios Phone: +1 714 821 8380; Email: [email protected] Serpanos; VP, Member & Geographic Activities: Yervant Zorian; MEMBERSHIP & PUBLICATION ORDERS: Phone: +1 800 678 4333; VP, Professional & Educational Activities: Kunio Uchiyama; Fax: +1 714 821 4641; Email: [email protected] VP, Publications: Fabrizio Lombardi; VP, Standards Activities: IEEE BOARD OF DIRECTORS Riccardo Mariani; VP, Technical & Conference Activities: William President & CEO: Jose M.D. Moura D. Gropp; 2018–2019 IEEE Division V Director: John W. Walz; President-Elect: Toshio Fukuda 2019 IEEE Division V Director Elect: Thomas M. Conte; 2019–2020 Past President: James A. Jefferies IEEE Division VIII Director: Elizabeth L. Burd Secretary: Kathleen Kramer BOARD OF GOVERNORS Treasurer: Joseph V. Lillie Term Expiring 2019: Saurabh Bagchi, Gregory T. Byrd, David S. Ebert, Director & President, IEEE-USA: Thomas M. Coughlin; Director & Jill I. Gostin, William Gropp, Sumi Helal President, Standards Association: Robert S. Fish; Director & VP, Term Expiring 2020: Andy T. Chen, John D. Johnson, Sy-Yen Kuo, Educational Activities: Witold M. Kinsner; Director & VP, David Lomet, Dimitrios Serpanos, Forrest Shull, Hayato Yamana Membership and Geographic Activities: Francis B. Grosz, Jr.; Term Expiring 2021: M. Brian Blake, Fred Douglis, Director & VP, Publication Services & Products: Hulya Kirkici; Carlos E. Jimenez-Gomez, Ramalatha Marimuthu, Director & VP, Technical Activities: K.J. Ray Liu Erik Jan Marinissen, Kunio Uchiyama

revised 14 August 2019

ADVERTISER INFORMATION

Advertising Personnel Southwest, California: Mike Hughes Debbie Sims: Advertising Coordinator Email: [email protected] Email: [email protected] Phone: +1 805 529 6790 Phone: +1 714 816 2138 | Fax: +1 714 821 4010

Advertising Sales Representative (Classifieds & Jobs Board) Advertising Sales Representatives (display) Heather Buonadies Central, Northwest, Southeast, Far East: Email: [email protected] Eric Kincaid Phone: +1 201 887 1703 Email: [email protected] Phone: +1 214 673 3742 Fax: +1 888 886 8599 Advertising Sales Representative (Jobs Board)

Northeast, Midwest, Europe, Middle East: Marie Thompson David Schissler Email: [email protected] Email: [email protected] Phone: 714-813-5094 Phone: +1 508 394 4026 Fax: +1 508 394 1707 21mitp03-kshetri-2906761.3d (Style 5) 10-05-2019 21:53

IT Economics

Blockchain and the Economics of Food Safety

Nir Kshetri University of North Carolina at Greensboro

Abstract—Blockchain technology has a potential to address many of the food safety challenges facing the world today. Some of the most promising blockchain applications developed to data have been in the food supply chains.

& ADULTERATED, CONTAMINATED, MISLABELED, and using the currently available methods (https:// misbranded food products have imposed tremen- www.nzherald.co.nz/business/news/article.cfm? dous social and economic costs to the global c_id 3&objectid 12039985). ¼ ¼ economy. About 600 million people in the world Food safety is of particular concern in indus- become ill due to contaminated food every year. trialized countries where consumers are increas- Of those, about 420 000 die, which include 125 000 ingly demanding higher quality and safer food.2 children under the age of 5 years.1 According to a A study conducted at the household level in the study conducted by the World Bank, unsafe food U.S. found that inadequate quality of food prod- 3 products cost low- and middle-income economies ucts is one of the key sources of food insecurity. $110 billion in lost productivity and medical There is big hope that blockchain technology expenses annually (https://www.foodsafetynews. can address many of the food safety challenges 4 com/2018/10/unsafe-food-in-lmics-costs-110- facing the world today. Indeed, some of the billion-a-year-world-bank/). One estimate sug- most promising blockchain applications outside gested that 30%–40% of the food consumers eat finance are being developed to address various is either “adulterated or mislabeled” (http://www. concerns in the food supply chains (http:// connect.catalyst-inc.org/techwatch/arcnet). In a internetofthingsagenda.techtarget.com/blog/ survey, 39% of food manufacturers thought that IoT-Agenda/Blockchain-for-industrial- their products can be easily counterfeited, and enterprises-Hype-reality-obstacles-and-outlook). 40% viewed that food fraud is difficult to detect At the national level, there is a tremendous positive economic impact of safe and quality food products associated with better health out- Digital Object Identifier 10.1109/MITP.2019.2906761 comes of citizens. At the firm level, companies Date of current version 21 May 2019. in the food supply chains can use blockchain to

May/June 2019 Published by the IEEE Computer Society 1520-9202 ß 2019 IEEE 2469-7087/19 © 2019 IEEE Published by the IEEE Computer Society September 2019 63 19 21mitp03-kshetri-2906761.3d (Style 5) 10-05-2019 21:53

IT Economics

Table 1. Blockchain deployment in food supply chains: Some examples.

Food retailer Explanations Remarks

Blockchain enabled the tracking of pork products 2016: Trial-tested a blockchain-based solution to in a few minutes compared to many days taken in monitor pork products in China and produce the past. Details about the farm, factory, batch imported to the U.S. from Latin America (https:// number, storage temperature, and shipping can U.S. retailer Walmart classic.qz.com/perfect-company-2/1146289/the- be viewed on blockchain (http://www. worlds-biggest-retailer-wants-to-bring- foodsafetynews.com/2017/03/a-new-era-of-food- blockchains-to-the-food-business). transparency-with-wal-mart-center-in-china/#. WOB65mcVjIU). Announced a plan to track its own branded products in France, Spain, and Brazil. It also noted French retailer plans to expand to other countries by 2022 Signed an agreement with IBM to use the solution. Carrefour (https://hawthorncaller.com/ibms-food- blockchain-is-going-live-with-a-supermarket-giant- on-board/). 2017: Its blockchain system went live with inner Mongolia-based food supplier Kerchin as its first supply-chain partner (Kshetri and Loukojanova, Chinese e-commerce 2019).6 Kerchin collects and stores data in its Implemented blockchain in food supply chains company Jingdong supply chain by scanning barcodes of its system, mainly involving B2B e-commerce. (JD.com) products. The information is then entered onto blockchain. After that, any changes in data require a digital signature. Both parties are informed if there is any change and modification in the data.7 March 2019: Announced the launch of a By scanning a QR code on the product package, blockchain platform to trace seafood. The consumers would be able to access information U.S.-based Bumble Bee company teamed up with German technology related to the details of the supply chain such as Foods company SAP for the project (https:// products’ origins, the size of the catch, the point cointelegraph.com/news/north-american-seafood- of capture shipping history, and trade fishing firm-to-use-blockchain-tech-in-supply-chain). certification.

address problems related to inefficiency, opac- The French retailer Carrefour has been one of ity, and fraud. Blockchain is also being used by the early adopters of the IBM Food Trust (see some firms in the food industry to enhance repu- Table 1). The retailer announced in March 2019 tational value by demonstrating their ability to that it would launch blockchain-enabled QR- innovate.5 codes for some of its milk products. With a smartphone app, customers can scan the labels to learn details about milk products that they Some Blockchain Projects in Food buy. The labels provide relevant details such as Supply Chains the date and location of collection and packaging A number of firms in the food industry have of a milk package, the GPS coordinates of dairy started to incorporate blockchain in supply farm producing it, and how the cow was fed chains (see Table 1). In November 2018, IBM (https://thenextweb.com/hardfork/2019/03/06/ commercially launched its blockchain-based carrefour-blockchain-milk/). Food Trust. Companies of all sizes in the food Big food retailers are also forcing their supp- industry supply chain can join the network liers to adopt blockchain. In September 2018, for a subscription fee that ranges from $100 to Walmart announced that it would require its sup- $10 000 a month (https://hawthorncaller.com/ pliers of leafy green vegetables to upload their ibms-food-blockchain-is-going-live-with-a- data to the blockchain system by September 2019 supermarket-giant-on-board/). IBM Food Trust is (https://techcrunch.com/2018/09/24/walmart-is- being used by many large food companies such betting-on-the-blockchain-to-improve-food- as Nestle, Unilever, and Walmart. safety/).

IT Professional 20 64 ComputingEdge September 2019 21mitp03-kshetri-2906761.3d (Style 5) 10-05-2019 21:53 21mitp03-kshetri-2906761.3d (Style 5) 10-05-2019 21:53

IT Economics

Table 1. Blockchain deployment in food supply chains: Some examples. Firms in food supply chains are rapidly adopt- industry. The shortage is especially severe in ing blockchain systems. Examples from retailers developing economies.9 For instance, out of the Food retailer Explanations Remarks such as Carrefour indicate that blockchain can be country’s 2 million software developers, only Blockchain enabled the tracking of pork products used to provide access to rich and detailed infor- 5000 were estimated to have blockchain skills. 2016: Trial-tested a blockchain-based solution to in a few minutes compared to many days taken in mation about food products, which is likely to Some speculate that about 80% of these develop- monitor pork products in China and produce the past. Details about the farm, factory, batch reduce uncertainty about quality and ingredients. imported to the U.S. from Latin America (https:// number, storage temperature, and shipping can ers may pursue job opportunities outside the U.S. retailer Walmart 10 classic.qz.com/perfect-company-2/1146289/the- be viewed on blockchain (http://www. This will increase consumers’ confidence in food country. worlds-biggest-retailer-wants-to-bring- foodsafetynews.com/2017/03/a-new-era-of-food- products that they buy. Food companies, thus, can Due primarily to the high costs and limited blockchains-to-the-food-business). transparency-with-wal-mart-center-in-china/#. boost revenue and profits by using blockchain. availability of blockchain talents, currently, the WOB65mcVjIU). For food retailers, another key benefit of deployment of blockchain-based solutions is Announced a plan to track its own branded blockchain is its ability to effectively handle a cri- products in France, Spain, and Brazil. It also noted more justifiable and more realistic in high-value sis situation. To illustrate this argument, consider 4 French retailer plans to expand to other countries by 2022 food products than in cheaper products. For Signed an agreement with IBM to use the solution. Carrefour (https://hawthorncaller.com/ibms-food- the 2015 E.coli outbreak at Chipotle Mexican Grill instance, in 2018, JD.com announced a plan to blockchain-is-going-live-with-a-supermarket-giant- outlets. The crisis left 55 customers ill. There implement blockchain to track its meat supply on-board/). were many negative news stories about this food- chains. Customers would be able to monitor 2017: Its blockchain system went live with inner borne illness. Many Chipotle restaurants were their meat products. Initial focus would be on Mongolia-based food supplier Kerchin as its first shut down, and investigations took place. All supply-chain partner (Kshetri and Loukojanova, high-end beef from Australia (https://tinyurl. Chinese e-commerce 2019).6 Kerchin collects and stores data in its these led to a significant blow to the reputation Implemented blockchain in food supply chains com/y8kfyv75). Likewise, the French retailer Car- company Jingdong supply chain by scanning barcodes of its of the company. There was a dramatic reduction system, mainly involving B2B e-commerce. refour ‘s traceability project focused on its pre- (JD.com) products. The information is then entered onto in sales revenues. The company’s share price mium farm products (https://www.ledgerinsights. blockchain. After that, any changes in data require dropped by 42%. The roots of the problem lie a digital signature. Both parties are informed if com/oxfam-blockchain-cambodian-rice-farmers/). 7 partly in Chipotle’s reliance on multiple suppliers. there is any change and modification in the data. A related point is that only big firms in the Companies such as Chipotle cannot monitor their March 2019: Announced the launch of a By scanning a QR code on the product package, food supply chains are currently in a position suppliers in real time. It is, thus, impossible to blockchain platform to trace seafood. The consumers would be able to access information to implement blockchain-based solutions. For U.S.-based Bumble Bee company teamed up with German technology related to the details of the supply chain such as prevent food contaminations. It is also difficult to instance, JD’s SC partner Kerchin that has Foods company SAP for the project (https:// products’ origins, the size of the catch, the point contain a food crisis in a targeted way after it is cointelegraph.com/news/north-american-seafood- of capture shipping history, and trade fishing adopted blockchain had $300 million in revenue discovered (https://hbr.org/2017/03/global- firm-to-use-blockchain-tech-in-supply-chain). certification. in 2017.7 Most of the food products in developing supply-chains-are-about-to-get-better-thanks-to- blockchain). Chipotle’s value proposition is cen- economies such as Africa and China, on the tered on fresh and locally sourced ingredients. other hand, are produced by very small farms. address problems related to inefficiency, opac- The French retailer Carrefour has been one of Food supply chain systems based on nonblock- These farms lack access to technology or Inter- ity, and fraud. Blockchain is also being used by the early adopters of the IBM Food Trust (see chain methods are expensive and cumbersome. net connectivity. Adoption of blockchain sys- some firms in the food industry to enhance repu- Table 1). The retailer announced in March 2019 The process involves manual verification and tems can be unrealistic for these farms, at least tational value by demonstrating their ability to that it would launch blockchain-enabled QR- massive record keeping. Blockchain can reduce in the near future. 5 innovate. codes for some of its milk products. With a the workload and ensure traceability. smartphone app, customers can scan the labels Summary to learn details about milk products that they Some Key Challenges Blockchain systems can bring transparency Some Blockchain Projects in Food buy. The labels provide relevant details such as and accountability in food supply chains. Such Supply Chains While the various benefits of blockchain in the date and location of collection and packaging food supply chains cannot be disputed, it is also systems, thus, are likely to play a tremendously A number of firms in the food industry have of a milk package, the GPS coordinates of dairy important to look on the cost sides. The high important role in ensuring food safety. Global started to incorporate blockchain in supply farm producing it, and how the cow was fed cost of hiring blockchain developers leads to economic and health benefits of blockchain sys- chains (see Table 1). In November 2018, IBM (https://thenextweb.com/hardfork/2019/03/06/ adverse economics of blockchain deployment in tems’ deployment to trace food products are, commercially launched its blockchain-based carrefour-blockchain-milk/). this industry. For instance, according to the job thus, extremely high. Food Trust. Companies of all sizes in the food Big food retailers are also forcing their supp- data analytics firm Burning Glass Technologies, Firms in the food industry can significantly industry supply chain can join the network liers to adopt blockchain. In September 2018, the median annual salary for a fulltime block- enhance customer loyalty and sales growth by for a subscription fee that ranges from $100 to Walmart announced that it would require its sup- chain developer in the U.S. was $140 000 in 2018, using blockchain. For firms in the food industry, $10 000 a month (https://hawthorncaller.com/ pliers of leafy green vegetables to upload their compared to $105 000 for general software devel- it is also important to be able to handle crisis sit- ibms-food-blockchain-is-going-live-with-a- data to the blockchain system by September 2019 opers. Blockchain specialists are reported to uations in order to be profitable. Blockchain- supermarket-giant-on-board/). IBM Food Trust is (https://techcrunch.com/2018/09/24/walmart-is- charge as much as $250 per hour.8 based solutions can help deal with risk situa- being used by many large food companies such betting-on-the-blockchain-to-improve-food- Labor and skill shortages have been identi- tions involving crises and emergencies. For as Nestle, Unilever, and Walmart. safety/). fied as a key challenge in the blockchain instance, if contaminated food products are

IT Professional May/June 2019 64 www.computer.org/computingedge 65 21 21mitp03-kshetri-2906761.3d (Style 5) 10-05-2019 21:53

IT Economics

found, food retailers can easily identify the 4. N. Kshetri, “Blockchain systems are tracking food source and engage in strategic removals of safety and origins,” Nov. 21, 2018. [Online]. Available: affected products. They do not need to recall https://theconversation.com/blockchain-systems-are- the entire product line. tracking-food-safety-and-origins-106491 While the benefits of blockchain systems in 5. M. Higginson, M. Nadeau, and K. Rajgopal, food supply chains outweigh the costs on aver- “Blockchain’s Occam problem,” Jan. 2019. [Online]. age, such systems are currently out of reach for Available: https://www.mckinsey.com/industries/ most small firms in the food industry. The solu- financial-services/our-insights/blockchains-occam- tions already available hold the promise of problem?cid other-eml-alt-mip-mck&hlkid ¼ ¼ developing cheaper systems that are easier to f1ff7216a70e4041951d60293978a0ea& use and trust—for farmers, food processing hctky 2762145&hdpid 95e9bdfa-0709-4b4d-8252- ¼ ¼ plants, and customers alike. Over time, block- f401bcaac86d chain implementation costs in the food industry 6. N. Kshetri and E. Loukoianova, “Blockchain adoption are likely to reduce. This is likely to make block- in supply chain networks in Asia,” IEEE IT Profess., chain-based solutions more affordable to smaller vol. 21, no. 1, pp. 11–15, Jan./Feb. 2019. companies and accelerate its diffusion in food 7. E. Huang, “Blockchain could fix a key problem in supply chains. China’s food industry: The fear of food made in China,” 2017. [Online]. Available: https://qz.com/1031861/ blockchain-could-fix-a-key-problem-in-chinas-food- & REFERENCES industry-the-fear-of-food-made-in-china/ 1. World Health Org., “More than 23 million people in the 8. L. Mearian, “Blockchain moves into top spot for hottest WHO European region fall ill from unsafe food every job skills,” May 1, 2018. [Online]. Available: https:// year,” 2015. [Online]. Available: http://www.euro.who. www.computerworld.com/article/3235972/blockchain- int/en/health-topics/disease-prevention/food-safety/ moves-into-top-spot-for-hottest-job-skills.html news/news/2015/12/more-than-23-million-people-in- 9. N. Kshetri, “The Indian blockchain landscape: the-who-european-region-fall-ill-from-unsafe-food- Regulations and policy measures,” Asian Res. Policy, every-year vol. 9, no. 2, pp. 56–71, 2018. 2. J. Kinsey, “GATT and the economics of food safety,” 10. M. Agarwal, “Blockchain: India likely to see brain drain Food Policy, vol. 18, no. 2, pp. 163–176, Apr. 1993. as 80% developers may move abroad,” 2018. [Online]. Available: https://www.sciencedirect.com/ [Online]. Available: https://inc42.com/buzz/ science/article/abs/pii/0306919293900246 blockchain-india-likely-to-suffer-brain-drain-as-80- 3. P. Webb, J. Coates, E. Frongillo, B. Rogers, A. Swindale, developers-prepare-to-move-abroad/ and P. Bilinsky, “Measuring household food insecurity: Why it’s so important and yet so difficult to do,” J. Nutrition, Nir Kshetri is a Professor of management with the vol. 136, no. 5, pp. 1404S–1408S, 2006. [Online]. Bryan School of Business and Economics, University Available: https://academic.oup.com/jn/article/136/5/ of North Carolina at Greensboro. Contact him at 1404S/4670058 [email protected].

This article originally appeared in IT Professional, vol. 21, no. 3, 2019.

IT Professional 22 66 ComputingEdge September 2019 INTERVIEW Editor: Gary McGraw, [email protected]

occasionally publishing on it. What Silver Bullet Talks it comes down to is there’s actually three totally separate concepts. There is the concept of the crypto- with Nick Weaver currencies themselves. There is the concept of the public blockchains, and then there is the concept of the Gary McGraw | Synopsys private or permissions blockchains. Now let’s start with the latter. What is a private or permissions blockchain? Simply an append-only data structure with a limited num- Hear the full podcast and find show links, notes, and an online discussion at ber of authorized writers: aka, a git www.synopsys.com/silverbullet. archive. There is nothing fundamental in a private blockchain that hasn’t been understood in the field What are your views on ICSI tech for 20-plus years. It’s just it has a transfer into the world? buzzword that causes idiots to throw As a research lab, we like building money at the problem. If you see a things that work. For example, the private or permissions blockchain Bro Network Security Monitor was project, it means either one of two developed at ICSI, and that’s being things. Either it’s a delusional piece commercialized right now. Ten years of techno-utopianism, or somebody ago, there was the extensible open smart in IT knows that there are real router project, and there was a sig- problems with what data you store, or nificant attempt to tech transfer that. how you access it, data provenance, There are also systems that we’ve and all this other stuff, and has ban- ended up building that have monetiza- died around this buzzword because icholas Weaver is a staff re- tion models that don’t match industry, idiots up in management will now N searcher at the University of but are productized. The Netalyzr net- throw money at this person to solve California, Berkeley’s International work analysis tool that we originally the real, interesting, hard problem. Computer Science Institute (ICSI). wrote in Java in the web browser now He also teaches courses at Berkeley. runs on Android phones. We keep that That’s one of the three. What about Weaver joined ICSI in 2003 as a running because it pays us in research the other two? post-doc after earning a PhD in results. We are able to turn the ser- The public blockchains are a global computer science from Berkeley. vice into publications, and therefore data structure where the idea is His research focuses on network se- we have a monetization strategy. It there is no centralized point of curity, worms, botnets, and other couldn’t actually work out in the real trust, but anybody can append to it. Internet-scale attacks. He also works world, but works for us. And we end Now these systems are, let’s say, not on network measurement. up supporting a large number of users actually distributed as advertised. that way. The Bitcoin blockchain is actu- ICSI is a nonprofit computer science ally effectively controlled by only research center. How is it funded? That’s good stuff. You and I seem three entities, but in an attempt It’s almost entirely grant funded. to share the same skeptical stance to be distributed, there is this reli- As a researcher at ICSI, I’m very when it comes to cryptocurrencies gious notion that distributed trust project and grant focused, and this and blockchain. Can you briefly give is somehow good in and of itself. is why I am doing more lecturing us a synopsis of your recent Burn It The result is systems that are either at Berkeley, because as a lecturer, I with Fire webinar? grossly inefficient or insecure. don’t need to worry about research I’ve come to this after five-plus The biggest tool that’s used for grants. years of watching the field and these systems is what is called “proof

Cryptocurrencies do not work About Nick Weaver for legitimate purchases if you don’t believe in the cryptocurrency. But let ick Weaver is a staff researcher at Berkeley’s International us suppose you believe in the vision N Computer Science Institute (ICSI). He also teaches courses of the great Satoshi. Then you don’t at Berkeley. Weaver joined ICSI in 2003 as a post-doc after earn- want to use cryptocurrencies either, ing a PhD in computer science from the University of California, because they’re baked in with these Berkeley. His research focuses on network security, worms, bot- monetary policies that are designed nets, and other Internet-scale attacks. He also works on network to be deflationary. The first rule of a measurement. Weaver holds a BA in astrophysics and computer deflationary currency is never spend science. His thesis work was on FPGA architectures, but he’s your deflationary currency. focused on computer security since 2001. He lives in Berkeley. There is one aspect of cryptocurrency that I think people don’t understand, and it is this notion of of work.” And proof of work is best The only way to do censorship- tethers. Can you talk about that for described as “proof of waste.” The resistant transactions without a a second? idea is that for somebody to rewrite cryptocurrency is cash, and cash There is a way to make a cryptocur- the history, they have to do as much requires physical proximity and rency work. You have to have an useless work as was done to create math. One million in US dollars entity that takes dollars and gives the history in the first place. Now weighs 10 kilograms. That’s a con- you crypto dollars at par, and vice this is great if you do a lot of useless siderable amount of stuff to be versa, that will take the crypto dol- work, except then it’s inefficient. If lugging around. What a cryptocur- lars and return you dollars. This you make the system efficient so rency is, well, let’s do a direct to is called a “bank,” and these are you do not do a lot of useless work, peer-to-peer payment system so called “banknotes,” and it’s recreat- you run into the problem of not that there are no central intermedi- ing the 18th-century banking sys- actually having any real protection. aries, but let’s do it electronically. tem. This can work, but one of three For example, Bitcoin, since the This has been used quite practically things has to happen. One option proof of work is paid for by the for drug dealers, extortionists, fake is you have regulation and enforce newly minted coins, ends up using hitmen, and all sorts of things like money-laundering laws and every- as much power as New York City. that. But if I want to do any payment thing else, in which case you have It’s just an obscene waste of energy. that one of the central authorities a system that ends up being no At the same time, these distributed will process, the cryptocurrencies cheaper or no more expensive than public append-only ledgers only provably don’t work. Visa, or Venmo, or anything else. have been useful for cryptocurren- Let’s say I want to buy a couch What is the point? cies. Now it’s time to address the from Overstock.com using bitcoins. Option number two is you have elephant in the room; the notion of I have to turn my dollars into bit- what is known as a “wildcat bank.” the cryptocurrency itself. coins, because I don’t want to keep This is a bank that prints banknotes it in bitcoins because the price that are actually unbacked. And this Right? Back to one. Here we go. is jumping up and down. That is is a term from 18th-century banking. Cryptocurrencies don’t actually expensive. Transfer the bitcoin. That The third option is a Liberty work as currency. They are provably is relatively cheap right now, but it’s Reserve where you actually do back inferior and can never be superior been upwards of $30 in the past. up your reserves. You redeem your to the alternatives for real-world And then the recipient on the other digital banknotes, but you don’t fol- payments, unless you need what is side has to convert the bitcoins back low the money-laundering laws, in known as “censorship resistance.” If into dollars. You have these two which case you end up being a guest I want to transfer you $500 by Pay- mandatory currency conversion of the federal government for the Pal, or Venmo, or whatever, we have steps for any real-world transaction, next 15 to 20 years. these trusted intermediaries called and even Overstock, the one public At the same time, the money banks, and they make it relatively company that supposedly embraces that the average person had is tied cheap. However, there is a problem. cryptocurrency, only keeps a few up temporarily or forever when If I want to transfer $500 to you hundreds of thousands of dollars’ the Feds shut down the institu- for drugs or the like, these central worth of cryptocurrency, with the tion. Tether is a specific cryptocur- authorities don’t like it. rest converted to dollars. rency that promises to be backed

824 ComIEEE SecurityputingEdge & Privacy SeptemberJuly/August 2019 2018 by dollars; they promise that there that it’s no longer harm-limited to Except it’s not bug free. is this 1:1 ratio where you give a small population of self-selected Oh, it’s so amusingly not bug free. I them dollars, they give you tethers, believers. It is spilling out into the like to use three examples. The first is and vice versa. The problem is this regular public. the DAO, the Decentralized Auton- is almost certainly a wildcat bank Fortunately, I think the crypto- omous Organization. The idea is, because they managed to produce currency space can die with proper let’s create a self-voting mutual fund some 2 billion tethers in the space application of regulation because for how we can invest our crypto- of a few months, and they are tied of how the regulations already are, currency in other projects. Now that to a Bitcoin exchange that is other- but it’s become important for me there’s actually nothing to invest wise cut off from banking. It may to advocate for the need to clean was neither here nor there, but have been the direct reason why the up the space in that cryptocurren- around 10 percent of all Ethereum Bitcoin price shot up so much. cies don’t provide benefit to society. at the time ended up in this basi- Or they could be facilitating They don’t provide benefit to all of cally self-creating, self-perpetuating, criminal money laundering, in us who aren’t interested in com- not-quite-a-Ponzi Ponzi scheme. which case those behind tether mitting crimes, but they do enable This was all fine and good until are liable to be guests of the fed- these problems. I think it is impor- somebody noticed there was a eral government. This is, however, tant to speak out. Another thing is re entrancy bug that allowed them what actually enables most of the the amount of scams in the space is to say, “Hey DAO, I am an investor. Bitcoin exchanges. Very few of the just incredible. Give me all my money.” And in the cryptocurrency exchanges actu- Effectively every initial coin process repeat the thing as, “Hey ally are connected to the US bank- offering these days should be called DAO, give me all my money.” And ing system. You have Coinbase. You a scam, because it is an unregistered because there was a transfer then have Gemini, and you have Kraken security and wouldn’t even pass the update, and you could re-entrantly (which should actually be shut laugh test on Shark Tank. And we call this code, it basically sucked all down for other reasons of crimi- have got these people hyping smart the money out. nal activity, but that’s neither here contracts. Most of the cryptocur- The problem is, well, the money nor there). As for the rest of the rency community seems intent on that was stolen mostly belonged exchanges, you can’t actually trans- speed-running 500 years of eco- to the people who came up with fer money into and out of them. nomic history for choosing their Ethereum in the first place. They basi- These are where the hundreds and bad ideas, but smart contracts are cally did a code release that changed hundreds of different cryptocurren- actually a new bad idea. The idea it and undid history. Their notion that cies are actually traded on. behind a smart contract is that I code is law and there is no central Tether has become this de facto write a program that is not really a authorities and no way to undo things reserve currency. If you look at smart contract, it’s a finance bot, was revealed to be a transparent lie Bitcoin trading volume, most of it because if it’s a contract, you have when it’s their money on the line. is actually on tether-denominated this exception-handling mechanism exchanges and is not actually being called a judge in the legal system. Exactly. exchanged for dollars, but these If I can walk up to a smart con- So that’s number one. Number two notional cryptodollars that may or tract, say “Give me all your money,” is the Proof of Weak Hands explicit may not be backed up, may or may and it does, is that even theft? Well, Ponzi Scheme. Version 1.0 collected not be a criminal enterprise—the it would be theft in the real world several million bucks before one flow just seems to continue on. It’s because we believe in justifying bug locked it up so nobody could really actually surprised me that it’s things, and this exception-handling transfer any more money into it, lasted this long. mechanism of the judge and jury and another bug allowed somebody and all that. Smart contracts are to steal all the money in it. I think Yeah, it really is absolutely stun- instead—let’s take the idea of a con- they’re up to 3.0 now, which has yet ning this stuff. Thanks. That was tract that is standardized and written to have a fatal bug, but we’ll see how extremely helpful. I think a lot of in a formal way, it’s called “legalese,” long that lasts. people need to have their eyes and instead, rewrite it in a language Finally there is the Parity multi- opened on this stuff, and you’re one that is uglier than JavaScript and has sig wallet. One of the problems of of the main people doing that. all sorts of pitfalls for programmers, cryptocurrencies is you can’t actu- I feel I have an obligation to. I kept eliminate the exception-handling ally store your cryptocurrency on an looking at the field, and in the recent mechanism, and then require that Internet-connected computer because run up, I came to the conclusion the code be bug free. if somebody gets onto your computer, www.computer.org/securitywww.computer.org/computingedge 259 INTERVIEW This article originally appeared in IEEE Security & Privacy, vol. 16, no. 4, 2018.

they get your private key and steal guy who invented the programming They’re just bad at implementing it. all your money. We actually had this language in the first place. The prob- They don’t understand the costs happen to us in the early days of Bit- lem is these things are designed to involved in that, and they cannot seem coin, and if security researchers can’t be non-upgradeable, but there are to ever implement it that way anyway. use Bitcoin on an Internet-connected hacks that allow you to update them. computer, nobody can. The idea is, If your money is tied up in some- All right, so onto a very personal let’s make it a two-party check system. body else’s contract because their issue. You suffer from depression We will have three private keys, and contract is the service, you have a that’s treated by therapy and medi- you have to use two of them to trans- choice. Either that contract has to cation, and you talk about that so fer the currency. have been bug free when created, others can benefit from the good This gives you good controls not good, or that contract has to aspects of treatment and therapy. if you can theoretically maintain be upgradeable, in which case you Tell us a little bit about that. at least two of your cryptographic have to trust that they upgrade the I’ve basically had in my life multiple keys. Some systems, like Bitcoin, contract properly and don’t cause depression meltdowns, and therapy offer it as a primitive. For Ethereum, damage or work against you in the and drugs saved my life twice as a it was built as a smart contract on process. student. And both times, after about top of things. This was the Parity a year, I’d just go off the medication, multisig wallet, which collected You have a central authority again. and a couple of years later the same some hundreds of millions of dol- You have a central authority. For thing would happen again. Just after lars, including an ICO by the guy example, there was a bug discovered the third incident, I realized that I behind the Parity multisig wallet. in some of these smart contracts didn’t want to repeat that mistake. Until somebody noticed that there that run these ICOs, where some- So, when I’m teaching students, was a bug where you could go up to body was able to create, what was every semester I include in my first one of these wallets say, “Hey, wal- it, 200 billion new tokens? Well, the slide deck, the notion that yes, I’ve let. You belong to me. Hey, wallet. people in charge of that particular been there. I’ve done that. This is not Give me all your money,” and started smart contract were able to undo good. There is help available. Every cleaning these out. And the only the process, but that means also if semester at least one student has reason this wasn’t a $150 million they can destroy the hack-created proven that it’s been worthwhile and theft is somebody else noticed tokens, if you’re invested in them, they’ll come up to me afterward. that this was going on, stole all the they can destroy your tokens too if money first, and then gave it back they feel like it. Super important work. Last ques- to the victim once the victim had tion, what is your favorite fiction upgraded code. You have to trust them. book or your favorite fiction book This is the ultimate irony in all these you’re reading at the moment? Unbelievable. systems—their belief in this mantra Let’s just say I’m a huge fan of The Which gets better. Now there’s the that lack of trust and decentraliza- Laundry Files. upgraded wallet code. For efficiency, tion are good in and of themselves, everybody refers to the same wallet ignoring the huge advantages you contract, and there was a bug in this get with just even the slightest of he Silver Bullet Podcast with contract. Some random loser came smattering of centralized trust. Yet T Gary McGraw is cosponsored along and said, “Hey contract. You they end up building systems that by Synopsys and this magazine and belong to me now,” and the contract aren’t even decentralized. They is syndicated by SearchSecurity. said, “Okey-doke. Yeah, I do.” Okay, build things that are orders of mag- oh crap. This shouldn’t have hap- nitude less efficient than they could Gary McGraw is vice president pened. “Hey, contract. Kill yourself.” be, but which have central authori- of security technology at Syn- The contract committed suicide, and ties and aren’t distributed anyway. opsys. He’s the author of Soft- now $150 million worth of crypto- ware Security: Building Security In currency is locked up and effectively I think the real design decision was, (Addison-Wesley 2006) and eight inaccessible unless the central author- “I would like to have all the trust other books. McGraw received a ities, that aren’t supposed to exist, belong to me.” BA in philosophy from the Univer- change the code to unlock this. We’re No, the cryptocurrency community sity of Virginia and a dual PhD in not done yet. The pièce de résistance. truly believes in this idea of decen- computer science and cognitive sci- The lead programmer and shin- tralization; that you should have to ence from Indiana University. Con- ing light behind this fiasco is the trust nobody. tact him via garymcgraw.com.

1026 ComIEEE SecurityputingEdge & Privacy SeptemberJuly/August 2019 2018

INTERVIEW: AUGMENTING HUMANS

Co-creation and Risk- Taking—In Pursuit of New Technology for Human Augmentation

An Interview with Pranav Mistry

Marc Langheinrich Marc Langheinrich and Nigel Davies interview Pranav Università della Svizzera Italiana (USI) Mistry, Global Senior Vice President of Research at Samsung, about his views on the field of human Nigel Davies Lancaster University augmentation.

Creating new technology for augmenting humans represents a major challenge to both industry and academia. To explore how the field has developed over recent years, as well as discuss future trends, we sat down with Pranav Mistry, Global Senior Vice President of Research at Sam- sung and director of its “Think Tank Team,” an interdisciplinary group that aims to create Samsung’s “products of tomorrow.” Examples of Pranav’s work include visionary contributions such as the SixthSense device—a wearable gesture interface that is the subject of one of the most watched TED talks of all time—as well as commercial successes such as Samsung’s smartwatch, the Samsung Gear. The idea of augmenting humans has been at the center of Pranav’s work for many years, so we were excited when we got the chance to interview him for this special issue.

Looking back to 2008 when you started to work on SixthSense, how do you think the field of “augmenting humans” has changed? A couple of things have changed since the early days of research into human augmentation. First, at that time, academia and industry had very different ideas. In academia, many of us were thinking about the idea of augmenting humans, while in industry this wasn’t much of a topic at all. When I look at the field today, industry and academia are moving in the same direction, looking at the same problems—both are thinking much more about bringing human augmentation to the mass market.

IEEE Pervasive Computing Published by the IEEE Computer Society April–June 2018 44 1536-1268/18/$33.00 ©2018 IEEE 2469-7087/19 © 2019 IEEE Published by the IEEE Computer Society September 2019 27 IEEE PERVASIVE COMPUTING

The second thing that has happened since 2008 is that, because of the new form factors of computing, everyday users have started accepting the possibility of human augmentation. Of course, for many years, maybe even several hundred years, we’ve been “augmenting” ourselves with simple things, like reading glasses or a watch. I remember that during my school years I used to wear a Casio watch that allowed me to store my phone numbers on my wrist. Nowadays, not only geeks like me and you do this. “Regular” people also think that in the future “I might live like that” or “I might wear that kind of device,” or even “I might directly connect to computers using my brain.” This new trend of accepting human augmentation isn’t actually coming from either academia or industry; it’s coming from the media, from science fiction movies and stories. And that’s helpful for all of us, because unless some people accept the fact that it’s okay to add something to their bodies, to their cognition, there won’t be support for computers augmenting our memories, or our intelligence, or our reasoning. This new trend of Back in 2008, several companies and small startups were doing something similar to what we’re doing right now—creating this world of accepting human augmented reality (AR) and virtual reality (VR), and new kinds of augmentation technologies. But the investment industry wasn’t taking augmentation isn’t them seriously at all. Now, the situation is flipped upside-down—if someone comes up with a completely different augmentation technol- actually coming ogy, even if it has nothing to do with currently accepted technology, investors will support it, thinking there might be a market for it in the from either future. From a technologist’s perspective, acceptance of novel aug- academia or mentation technology as a potential big thing is a huge and positive change. industry; it’s coming

from the media, Google Glass attracted quite a lot of negative press in the past. Do you think this was because of something inherent in the technology from science fiction or would the reaction be different today? movies and stories. I would say that a fair debate is needed, at the individual as well as the societal level, on what’s right or wrong with any new technology. This debate is probably going to emerge as a result of trial and error. I don’t think a social scientist or psychologist can simply declare that people should or not should not accept the technology. Take, for example, self-driving cars: it seems that, perhaps due to their depiction in sci-fi movies, people already accept this technology. But is the technology safe? Is it for everyone? You can’t have an informed debate until there are enough self-driving cars on the road. I wouldn’t say Google Glass was a failed technology; it was a necessary intermediate step that allowed the public to understand how such technology works and could fit into their lives. Peo- ple often react negatively to a new technology but slowly come to accept it as they see its benefits. When we started research on Samsung VR and told users to insert their phone into the headset and put it in front of their eyes, their initial reaction was “Are you crazy? What are you talking about?” In time, people recognized its value for entertainment and other kinds of applications. And, of course, how a technology will play out in the market isn’t exactly what industry expects. VR is becoming less of a virtual reality experience and more of an “other reality” experience in which people want to be transported to another place, such as the other side of the world. Technology changes the perspective of both the user and the maker, and a period is needed to assess those changes.

Has technology in the human augmentation space evolved as you foresaw when you were doing your PhD?

April–June 2018 45 www.computer.org/pervasive

28 ComputingEdge September 2019 IEEE PERVASIVE COMPUTING AUGMENTING HUMANS

Some aspects of technology have seen much faster progress than I expected, while others have The second thing that has happened since 2008 is that, because of the new form factors of com- been much slower. In particular, I thought software development would be well behind hardware puting, everyday users have started accepting the possibility of human augmentation. Of course, development, but it has been the other way around because today there are a lot more experts on for many years, maybe even several hundred years, we’ve been “augmenting” ourselves with machine learning, AI, and computer vision algorithms. With respect to hardware miniaturization, simple things, like reading glasses or a watch. I remember that during my school years I used to there isn’t much difference between what we used nine years ago and what we’re using today. wear a Casio watch that allowed me to store my phone numbers on my wrist. Nowadays, not The Galaxy phone I have now is similar in size to the one I had in 2008. Hardware hasn’t only geeks like me and you do this. “Regular” people also think that in the future “I might live changed as drastically as I anticipated. like that” or “I might wear that kind of device,” or even “I might directly connect to computers using my brain.” I think there are a couple of reasons for this. One is that, when thinking about future hardware, we quickly forget about the fundamental limits of physics, which play a crucial role in the hard- This new trend of accepting human augmentation isn’t actually coming from either academia or ware field. Another is that the emergence of the cloud has enabled software development to ac- industry; it’s coming from the media, from science fiction movies and stories. And that’s helpful celerate. Although we didn’t know it a decade ago, the infrastructure needed for future for all of us, because unless some people accept the fact that it’s okay to add something to their applications was being prepared. bodies, to their cognition, there won’t be support for computers augmenting our memories, or our intelligence, or our reasoning. Right now, if you wanted to make a memory or visual augmentation system, hardware is the lim- This new trend of iting factor. We can explore different software solutions—a better UX or maybe a contextual in- Back in 2008, several companies and small startups were doing some- terface—but physical silicon is harder to change. You can’t do trial thing similar to what we’re doing right now—creating this world of accepting human and error with hardware as much as you can with software. augmented reality (AR) and virtual reality (VR), and new kinds of augmentation technologies. But the investment industry wasn’t taking augmentation isn’t them seriously at all. Now, the situation is flipped upside-down—if Would it be fair to say that it’s been a long time since we’ve seen someone comes up with a completely different augmentation technol- actually coming Right now, if you hardware in the human augmentation space that’s really been trans- ogy, even if it has nothing to do with currently accepted technology, formative? wanted to make a investors will support it, thinking there might be a market for it in the from either future. From a technologist’s perspective, acceptance of novel aug- academia or Yes, I think so. It takes time to develop hardware—there are many memory or visual mentation technology as a potential big thing is a huge and positive steps—but the good news is that pipelines are in place from the labs to change. industry; it’s coming end users. While hardware is still the limiting factor, we can bring in- augmentation novations to the market today much faster than we would have been from the media, able to, say, five years ago. Industry has accepted the first step in this system, hardware is Google Glass attracted quite a lot of negative press in the past. Do pipeline—that this technology has mass market potential—so we can you think this was because of something inherent in the technology from science fiction now develop a market strategy much more quickly. Also, it’s not just the limiting factor. or would the reaction be different today? about the computing or electronics industries anymore; the automotive movies and stories. industry is talking about more cameras and sensors, the fashion indus- I would say that a fair debate is needed, at the individual as well as the try is exploring new kinds of smart accessories and smart garments, societal level, on what’s right or wrong with any new technology. This and so on. And that’s what’s so exciting: the everyday world is start- debate is probably going to emerge as a result of trial and error. I don’t ing to get augmented. think a social scientist or psychologist can simply declare that people should or not should not accept the technology. Take, for example, self-driving cars: it seems that, perhaps due to their depiction in sci-fi So while the hardware side will continue to innovate, the rate of progress will be somewhat movies, people already accept this technology. But is the technology bounded compared to the software side? safe? Is it for everyone? You can’t have an informed debate until there are enough self-driving cars on the road. Yes, definitely. Even now, software improvements are defining what new hardware improvements are needed because the software side is much further advanced. For example, Apple, I wouldn’t say Google Glass was a failed technology; it was a necessary intermediate step that Google, Microsoft, and the like are building chips just for machine learning or computer vision; allowed the public to understand how such technology works and could fit into their lives. Peo- that wasn’t the case several years ago. The big industry players have already started thinking ple often react negatively to a new technology but slowly come to accept it as they see its bene- about the coming new world of human and digital augmentation and they want to be prepared. In fits. When we started research on Samsung VR and told users to insert their phone into the the early years of computing, hardware defined the software, which in turn defined the user ex- headset and put it in front of their eyes, their initial reaction was “Are you crazy? What are you perience. Today, popular culture like sci-fi movies and games is defining new user experiences, talking about?” In time, people recognized its value for entertainment and other kinds of applica- which we’re implementing in software; we then create custom hardware to enable this software- tions. And, of course, how a technology will play out in the market isn’t exactly what industry defined vision. expects. VR is becoming less of a virtual reality experience and more of an “other reality” experience in which people want to be transported to another place, such as the other side of the world. Technology changes the perspective of both the user and the maker, and a period is needed to assess those changes. Earlier you mentioned application areas for augmenting humans, such as memory or vision augmentation. Do you see a “killer app” in this space?

My thinking is that, as been the case with many new things, consumers will readily accept aug- Has technology in the human augmentation space evolved as you foresaw when you were do- mentation applications in a noncritical area like gaming. Industry, on the other hand, is always a ing your PhD? bit more skeptical about new technology, and hence about which particular area to explore. For example, Microsoft’s HoloLens or Google Glass are finding a home in niche B2B industry applications because it’s a much more controlled environment. Even at the device level, we can see April–June 2018 45 www.computer.org/pervasive

April–Junewww.computer.org/computingedge 2018 46 www.computer.org/pervasive29 IEEE PERVASIVE COMPUTING

this exploration at work. For example, my Galaxy phone has Bixby Voice and Bixby Vision, but my primary method of communicating with my phone is still the touchscreen because it under- stands what I want 100 percent of the time. Soon, however, I might not need to tap on the screen anymore, and that’s awesome. I believe this will happen with augmentation technology as well: new technology will coexist with traditional technology until users feel comfortable with it, and then they’ll make the switch. So I believe the proving grounds for human augmentation technology will be either noncritical areas such as entertainment or very controlled environments in critical areas such as business, where the technology does limited things but does them perfectly. Enterprise applications are especially promising for exploring augmentation technology because this world isn’t fully open. At the consumer level, the technology initially will mostly be for fun. Take AR: most people use it to augment photos on sites like Facebook to, say, put a cat on their head or to add virtual sunglasses or a smiley face.

So if we start out with “fun” things first, what do you think this will I believe the proving morph into? grounds for human When we started working on Gear VR, we thought it was all going to be about virtual experiences such as immersive gaming or exploring augmentation imaginary worlds. We didn’t know which particular aspect of the technology was going to be most important to users. We found out that technology will be what people wanted more than anything else was to use the device to “teleport” to another part of the real world. For us that revelation either noncritical opened up an entire ecosystem of new devices and 360-degree cameras to explore, and new research goals like the network capabilities to areas such as support streaming high-definition live video. Now when we introduce a technology we tell users that it can do this, this, and this, but we entertainment, or leave it up to them to decide what they would like to do the most and then follow their lead. very controlled Let me tell you an interesting story. A couple of years ago, I was in a environments in restaurant and at a table next to me was a group of young teenagers all using their smartphones’ heart rate monitor. Such an app is targeted at critical areas such older users, so out of curiosity I asked what they were doing, and they said they were playing a game in which the person with the highest as business. heart rate had to pay the bill. These teens had thus taken something created for a serious use case and repurposed it for something fun instead, and in so doing were extending the application spectrum of this kind of technology. As this story shows, it’s impossible to guess all the uses people will make of the things we create. At Samsung, we always say that we listen to our customers—we listen to what they want, we create a new technology, we listen to the response and change the technology, and so on until we get it right. This is essentially a customized process for designing the future. When Ford launched the Model T, there was one model and it was only available in black—everyone had the same car. That doesn’t work anymore: today people want devices that adapt to their needs. And that’s how it should be, since our needs are different. My father is an architect; his memory or visual augmentation needs are completely different from mine. Even if everyone eventually uses augmentation technologies like they use smartphones today, it’s unlikely they’ll be “augmented” the same way. I thus see a future with a proliferation of customized augmentation technologies. One person might have an app to help remember names or phone numbers, while another might use one to help translate Chinese to Japanese during trips to Japan. At the same time, creating a technology that can satisfy the diverse needs of millions of customers at an efficient production scale of billions of units is a huge challenge.

April–June30 2018 ComputingEdge 47 www.computer.org/pervasiveSeptember 2019 IEEE PERVASIVE COMPUTING AUGMENTING HUMANS

I should mention that a friend of mine in academia is interested in personalized artificial intelli- this exploration at work. For example, my Galaxy phone has Bixby Voice and Bixby Vision, but gence. His idea of AI isn’t a single “Watson” that serves everyone equally. Instead, your per- my primary method of communicating with my phone is still the touchscreen because it under- sonal assistant grows with you, has the same social and cultural experiences, and shapes itself in stands what I want 100 percent of the time. Soon, however, I might not need to tap on the screen line with your personality. I believe the same “one size doesn’t fit all” notion equally applies to anymore, and that’s awesome. I believe this will happen with augmentation technology as well: human augmentation. It also has interesting implications: virtual agents with their own personal- new technology will coexist with traditional technology until users feel comfortable with it, and ity might have problems communicating with each other or different users, leading to all sorts of then they’ll make the switch. misunderstandings! So I believe the proving grounds for human augmentation technology will be either noncritical areas such as entertainment or very controlled environments in critical areas such as business, where the technology does limited things but does them perfectly. Enterprise applications are Could you talk a little bit about your own current research, and that of Samsung as well, in especially promising for exploring augmentation technology because this world isn’t fully open. trying to realize this vision? At the consumer level, the technology initially will mostly be for fun. Take AR: most people use Samsung, at its core, used to be a hardware-heavy company: we understand silicon much better it to augment photos on sites like Facebook to, say, put a cat on their than anyone else in the world. And as I mentioned before, it takes time to grow silicon—it’s like head or to add virtual sunglasses or a smiley face. a tree; it doesn’t grow overnight. It takes a lot more time to develop a chip than to convert an algorithm into running software. If software doesn’t behave properly, you can change some lines of the code, but if hardware has an issue, you have to rebuild it. And then there are the physical So if we start out with “fun” things first, what do you think this will I believe the proving limits of the atoms compared to the bits. Samsung’s research focus is “being ready for the fu- morph into? grounds for human ture.” It sounds simple, but it’s hard because no one knows where we’re heading and so we have When we started working on Gear VR, we thought it was all going to to make some big bets in the R&D area. Some of my work focuses on thinking about the future be about virtual experiences such as immersive gaming or exploring augmentation and what its silicon requirements might be so that we don’t lag behind the software industry and imaginary worlds. We didn’t know which particular aspect of the tech- what consumers want. nology was going to be most important to users. We found out that technology will be Earlier in this interview I talked about how, as a first step, we have to “test the waters” in the real what people wanted more than anything else was to use the device to world—you can’t just skip straight to wide-scale deployment of augmentation technology. That “teleport” to another part of the real world. For us that revelation either noncritical step one is also my focus: to see if and how society accepts a new technology like Google Glass opened up an entire ecosystem of new devices and 360-degree cam- and what kinds of problems they have with it in terms of privacy, security, or just how it looks or eras to explore, and new research goals like the network capabilities to areas such as feels. So, personally, my research extends across the entire development spectrum because we support streaming high-definition live video. Now when we introduce have to take risks in this industry to move forward. We’re like a startup: notice that Samsung is a technology we tell users that it can do this, this, and this, but we entertainment, or among the first to try out new technologies on the market, from curved screens to VR to 360- leave it up to them to decide what they would like to do the most and degree cameras. then follow their lead. very controlled

Let me tell you an interesting story. A couple of years ago, I was in a environments in restaurant and at a table next to me was a group of young teenagers all Throughout our discussion has been this thread that software is moving very quickly—much using their smartphones’ heart rate monitor. Such an app is targeted at critical areas such more quickly than any of us predicted. Hardware moves more slowly, and because of that, and older users, so out of curiosity I asked what they were doing, and they because you want the silicon to be in place to support new software, you must maintain a long- said they were playing a game in which the person with the highest as business. term perspective. As part of that process, you’ve got to put stuff out there and see what sticks. heart rate had to pay the bill. These teens had thus taken something Is that a fair summary? created for a serious use case and repurposed it for something fun in- In one way, yes. However, the last point is really is about co-creating stead, and in so doing were extending the application spectrum of this the future with our users; it’s not just a matter of putting something on kind of technology. As this story shows, it’s impossible to guess all the market and seeing how it goes. No matter how good an initial the uses people will make of the things we create. product is, it isn’t perfect; it’s not “tomorrow” yet, but we’re almost At Samsung, we always say that we listen to our customers—we listen to what they want, we there—a minute to midnight. That’s what drives Samsung: let’s find There was a time create a new technology, we listen to the response and change the technology, and so on until we out what works and what doesn’t, and then let’s make it better. get it right. This is essentially a customized process for designing the future. When Ford when new ideas launched the Model T, there was one model and it was only available in black—everyone had the same car. That doesn’t work anymore: today people want devices that adapt to their needs. So it’s important to do co-creation and co-design early? about technology And that’s how it should be, since our needs are different. My father is an architect; his memory or visual augmentation needs are completely different from mine. Even if everyone eventually Absolutely. mostly originated in uses augmentation technologies like they use smartphones today, it’s unlikely they’ll be “augmented” the same way. academia, but now Finally, when it comes to taking risks, what would you like to see I thus see a future with a proliferation of customized augmentation technologies. One person from academia? the situation is might have an app to help remember names or phone numbers, while another might use one to help translate Chinese to Japanese during trips to Japan. At the same time, creating a technology At the start of this interview I said that academia and industry are almost reversed. that can satisfy the diverse needs of millions of customers at an efficient production scale of bil- moving in the same direction, and this is both good and bad. On one lions of units is a huge challenge. hand, industry and academia can benefit a lot by collaborating and sharing a common perspective. On the other hand, we might overlook something interesting that might have been.

April–June 2018 47 www.computer.org/pervasive April–Junewww.computer.org/computingedge 2018 48 www.computer.org/pervasive31 IEEE PERVASIVE COMPUTING

Academics have more freedom to explore; they should go wild and try out new things! We’re seeing fewer contributions from academics and more from industry. There was a time when new ideas about technology mostly originated in academia, but now the situation is almost reversed, with industry saying, “We have this technology—can you academics do something with it?” I feel academia should again become a little bit more dominant in telling industry where they see the future going. The key challenges academics can focus on, which is harder in industry, are those we discussed before: testing people’s experience, what works and what doesn’t; seeing, even in a small way, what users like and what they don’t; understanding what people expect the new world to be like. Ultimately, industry and academia must work together to solve the bigger societal problems, not just technical issues related to software and hardware. Because the main question about augmenting humans is not about what we can do but what we should do, which will ultimately de- termine whether the technology succeeds or fails. It’s this moral aspect in which academia needs to play a major role.

So it sounds like a call to arms for the world’s philosophers and ethnographers? I think so, because they’re critical to realizing this vision.

Thanks a lot, Pranav, for taking time out of your busy schedule to speak with us.

ABOUT THE AUTHORS Marc Lanheinrich is a professor in the Faculty of Informatics at Università della Svizzera Italiana (USI) in Lugano, Switzerland, where he heads the Research Group for Ubiquitous Computing, and Editor in Chief of IEEE Pervasive Computing. Contact him at langhein- [email protected]. Nigel Davies is a professor of computer science at Lancaster University and Editor in Chief Emeritus of IEEE Pervasive Computing. Contact him at [email protected].

This article originally appeared in IEEE Pervasive Computing, vol. 17, no. 2, 2018.

April–June32 2018 ComputingEdge 49 www.computer.org/pervasiveSeptember 2019 Editors: Frank Steinicke Spatial Interfaces and Wolfgang Stuerzlinger

Compressing VR: Fitting Large Virtual Environments within Limited Physical Space

Khrystyna Vasylevska and Hannes Kaufmann TU Wien

deally, a virtual reality (VR) system should All of them target the highest possible compression connect a real person to a computer simulated factors for any virtual space, and each has its own world, allowing the system to fully substitute bene ts and challenges. Ithe real world and its rules. Like the Holodeck featured on the TV series Star Trek, such a system Basic Reorientation should be able to provide an interactive, tangible The most basic approach is to stop users at the virtual world that the user can explore without re- boundary of the tracked space and ask them to strictions within a real room. One of the rst ways return to its center and continue from the same someone might attempt to explore such a world point in the VE.2 Rotation can also be instanta- would be to walk around. Nonetheless, as a result neously introduced based on the user’s position in of restricted physical workspaces and technologi- the real world.3 cal limitations, the free and unlimited exploration These basic approaches interrupt the VR experi- of an arbitrary large-scale virtual environment ence and thus might adversely impact important (VE) is not possible in practice. We could rely on characteristics of it, such as immersion and a sense walk-like gestures or use additional devices to al- of presence in the VE. More intricate methods of low users to travel through VEs, while their physi- redirection exercise unperceivable manipulation, cal locations do not change. However, real walking while the rendering and the user’s immersive ex- in VR provides important vestibular and proprio- perience remain intact. ceptive cues that positively impact higher mental processes and improve the illusion of reality.1 Sense Manipulation In this article, we provide an overview of the One class of techniques known as redirected walking existing approaches and techniques for enlarging employs sense or orientation manipulation.2 These the walkable virtual space. We speci cally focus methods build upon the principle that, during the on the methods that use spatial manipulation multisensory integration process, visual cues are for spatial compression, as it is one of the most usually weighted as more accurate and therefore promising, but underexplored methods for nonin- more important for orientation than other senses trusive user redirection in a limited physical space. such as proprioception. Redirected walking uses Researchers have developed several techniques to the concept of camera manipulations based on address the problem of free natural locomotion in gains. The user’s dynamic motions are scaled ac- VEs within an available real-world workspace. We cording to the de ned gains and then mapped to distinguish the following types of spatial compres- the translation and rotation of a virtual camera sion methods: within a VE. The user reacts to the changes in the virtual camera’s pose and adapts his/her motions ■ basic reorientation, accordingly, which in turn lets us keep the user ■ sense manipulation, within the real workspace.2 ■ rendering manipulation, and It is also possible to continuously apply the ad- ■ 3D scene manipulation. ditional rotation. A generalized version of this

2469-7087/19 © 2019 IEEE Published by the IEEE Computer Society September 2019 33 Published by the IEEE Computer Society 0272-1716/17/$33.00 © 2017 IEEE IEEE Computer Graphics and Applications 85 Spatial Interfaces

approach is called the circular algorithm,3 which the compression factor of VEs without the need mainly consists of two main types of manipula- to manipulate the users’ senses in an unnatu- tion and their combinations. The rst keeps users ral way. The core approach in scene manipula- on a small circular trajectory, allowing them to tion is to have different parts of a VE share the diverge in any direction. The other constantly re- same real workspace. To do so, some parts or directs the user to the center of a big circle when elements of a VE are relocated, overlapping in the user performs a rotation. The goal is to make the real-world space based on the users’ actions. the additional rotation imperceptible to the user. Most importantly, these changes occur without For example, it may be applied when the user is the users noticing. performing fast head motions trying to follow a One basic spatial manipulation approach in- fast-moving object. This approach is referred to as volves the use of deterrents. That is, during run- the distractor technique.4 time, objects are inserted into the VE that users Human sensitivity limits the extent to which we must avoid walking through, such as roadblocks, can apply manipulations in virtual spaces5 because which forces them to take an alternate route such manipulations of primary senses should re- within the environment.8 main unnoticeable to users to minimize the pos- Other approaches go further, changing the VE’s sible adverse effects. Hence, sense manipulation con guration more drastically while users explore still demands a considerably large real workspace. the virtual space and perform tasks. For instance, for users to continuously walk along a straight path in a VE with a curvature gain re- Change Blindness quires a squared workspace of almost 500 m2.5 Re- Change blindness is an entirely different approach search has shown however that the radius might to spatial compression wherein the system or be decreased by a factor of two if the curvature speci c task distracts users so they fail to notice gain is accompanied by translation gain.6 large changes in VEs.9 In the rst study,10 users In practice, redirection by sense manipulation were asked to perform a task that required they works well for moderately paced users who try to turn their backs to a door. While the users were follow the planned path, but it can fail in other distracted, the door’s location was moved to a dif- circumstances and scenarios. Therefore, sense ma- ferent wall in the virtual room (see Figure 1a). An nipulation is most suitable for outdoor open VEs interesting outcome of the study was that, after where the virtual path might be easily adjusted exploring the virtual building, the study partici- to t the real workspace. Nevertheless, the use of pants were able to draw a map of the environment sense manipulation requires ne-tuning and exten- despite substantial spatial manipulations. sive testing of each particular VE, and such testing A second study tested more signi cant scene should account for some unexpected user behavior. modi cations based on change blindnesss.11 In this second study, the entire wall containing the door Rendering Manipulation was moved several meters away from its original Qi Sun and his colleagues proposed a novel ren- position; this change signi cantly enlarged the dering approach to spatial compression.7 Their room in order to return users back to the real start- technique consists of a planar mapping of the con- ing point. Such an approach is most suitable for strained walking path with a custom reprojective environments that contain regular structures, al- rendering that is capable of wrapping an arbitrary though generalizing and expanding the approach to VE into any real-world workspace. The obvious arbitrary spatial arrangements is still problematic. bene t of this approach is its exibility. However, their method distorts the VE’s visuals and makes Impossible Spaces it dif cult for users to estimate the scale and exact Another method to compress VEs is the use of impos- shape of the environment. sible spaces.12 This approach increases the amount Because this technique alters the user’s percep- of walkable space by making separate rooms over- tion of the environment, it needs to be explored lap and partially share the real space with one further. Nevertheless, this approach could also be another. There are two possible implementations successfully applied to outdoor virtual scenes that of impossible spaces. One involves expanding the involve content that is less sensitive to distortions. space available in adjacent rooms by moving their shared wall and increasing the overlap (see Figure Scene Manipulation 1b). At the same time, the outer walls, doors, and Unlike the previous approaches, virtual scene ma- the connecting corridor do not change. The other nipulation has an enormous potential to increase implementation involves increasing the overlap

8634 September/OctoberComputingEdge 2017 September 2019 (a)

(b)

Figure 1. Spatial manipulations: (a) In the change blindness approach, the door is relocated in the virtual environment (VE) when the user is distracted by a task.10 (b) The impossible spaces approach lets us extend a room setup with 50 percent overlap. The wall between the rooms is relocated based on the users’ actions in order to enlarge the room they are about to visit using the overlap area.12 by bringing the two rooms closer to each other designed: a simple corridor; a U-shaped corridor, to minimize the space needed for them as well as with which we extended a simple corridor an ad- the length of their connecting corridor. A study on ditional 10 meters, detaching it from the rooms’ impossible spaces showed that when blind walk- perimeter; and a C-shaped corridor, which we ing between the identically placed objects in both extended with another 10 meters and four addi- rooms nonnaïve users failed to estimate the actual tional turns. distances between the rooms correctly. That result Although the simple length extension did some- suggests the use of impossible spaces ef ciently in- what decrease the users’ overlap perception, our creases the sizes of walkable virtual environments. results showed that it was not particularly ef cient We preformed a follow-up study for impossible in terms of the use of available space. However, spaces showing that, by changing the complex- the more complex C-shaped corridor substantially ity of the corridor, it is possible to increase the impacted the users’ spatial perception when com- amount of unperceived overlap.13 In this case, we pared with the simple and U-shaped corridors. The de ne the complexity by the corridor’s length and estimated distances between the rooms in this the number of corners in it. We used an expand- case suggested that the rooms were far apart from ing implementation of impossible spaces and ex- each other. Moreover, some of the participants plored whether the overlap perception depends on also stated that the rooms were not aligned. the corridor that connects the rooms. As in the In later work, we further delved into the cor- earlier study, we used blind walking as a measure. ridor-dependent effects on spatial perception by Figure 2 illustrates the three types of corridors we addressing the corridor con guration parameters

IEEE Computer Graphics and Applications 87 www.computer.org/computingedge 35 Spatial Interfaces

Figure 2. Virtual layouts with different corridors: (a) a simple short corridor, (b) a U-shaped corridor, and (c) a C-shaped corridor. The overlap was implemented by moving the wall between the rooms.

(a) (b) (c)

and geometry.14 Furthermore, we diverged from and tested this set separately. In this second set, the simple right-angled geometry. Instead, we used we eliminated the corners and straight parts of smooth curves and scrutinized their effect on spa- the corridors that could be used as landmarks tial perception. We used two rectangular rooms of or for directional hints. Our objective was to identical sizes that were aligned and overlapped by see whether users would still perceive the room 50 percent throughout the experiment and focused alignment and overlap in the same way and to only on corridor con guration. We hypothesized evaluate the potential use of curved paths for that the spatial perception in self-overlapping VEs spatial manipulations. might be inuenced by the following properties of In addition, we assumed that asymmetrical lay- the connecting corridor: outs might feel different when participants walked in alternating directions. Therefore, we had the ■ the number of corners, participants explore such layouts twice, in clock- ■ the sequence of corners, wise and counterclockwise directions. To measure ■ the positions of the corridor endpoints (doors) the participants’ spatial perception, we introduced relative to the overlap zone, and a new approach: interactive visual reconstruction ■ the path’s symmetry or asymmetry. using semitransparent representations of the rooms (see Figure 3c). We also explained to our partici- Based on these criteria, we created nine right- pants the possibility of the overlapping, adjacent, angled layouts, ve of which were symmetrical and completely detached rooms, challenging them and four asymmetrical. Figure 3a shows the right- to estimate the original room arrangement in each angled asymmetrical layout. We also created case separately. a second set of layouts where the right-angled The study results con rmed the importance of corridors were substituted with curved versions all the corridor parameters we have discussed here,

(a) (b) (c)

Figure 3. Experimental environment on the use of corridors in impossible spaces: (a) 3D models of symmetric and asymmetric right-angled layouts and (b) 3D models with curved corridors. (c) During task performance, participants were shown semitransparent representations of the rooms.

8836 September/OctoberComputingEdge 2017 September 2019 (a) (b) (c)

Figure 4. Flexible spaces: (a) a basic procedurally generated VE, (b) a user exploring the exible spaces in the VE, and (c) an elevator extension. the presence of distortions in the spatial percep- change the corridor’s direction should be taken tion, and differences in the perception of an asym- into account. metric layout depending on the walking direction. Our results also suggest that participants were still Flexible Spaces able to perceive the overlap area and room align- The exible spaces approach is one of the rst at- ment when they walked right-angled corridors. tempts to merge several techniques. Our approach The layout set with curved corridors provided an is based on the assumption that detailed spatial increased variation in estimated spatial arrange- knowledge might be useful for navigation but is ments and caused the participants to estimate not necessary for all environments, particularly larger distances between rooms compared with those that focus on information and content or the right-angled set. The results indicated that impression and experience. A perfect example of in many con gurations the participants believed such real-world settings is a large museum with there was space between the rooms. Unlike the signs that substitute the map of the building or right-angled layouts, some participants also asked the insides of a pyramid where loss of orientation whether the rooms had been rotated, which sug- is part of the experience. gests a perceived change of room orientation. The exible spaces algorithm also relies on Overall, the best results in both studies were the fact that cognitive maps are often distorted, achieved with the S-shaped corridor (see Figure sometimes to the degree that they cannot be rep- 3b), which reliably created a long distance between resented by images.15 These distortions originate the rooms. The S-shaped corridor was also the in the hierarchical structure of the cognitive maps most space ef cient because of the triple overlap and mental heuristics that help us to remember as it passed directly through the area where the information about the environment. Thus, human rooms overlap. perception gives us a way to create a new class of Earlier studies have con rmed distortions in information- and content-oriented environments spatial perception for larger real scenes, but to the that provide consistent connections between their best of our knowledge, our study is the rst to di- parts (prede ned bidirectional links between the rectly observe a similar effect for small-scale self- rooms) but that modify the details in between overlapping VEs. Based on the obtained results, with a changeable architecture. we suggest considering the parameters of the path Our algorithm creates a procedurally generated that connects different spaces when designing self-overlapping and self-reorganizing dynamic VE impossible VEs. If possible, loop-like paths should that automatically regenerates the environment be avoided as they might increase the perceived within the available workspace. In this approach, overlap. Meanwhile, the corridors that change the we united change blindness and impossible spaces, turning directions seem to be more realistic and taking them to the extreme by allowing constant decrease the overlap. The positions of doors rela- restructuring of the VE. Unlike previous work, our tive to the overlap also matters, and it is best to version of change blindness is task independent. position them as far from the overlap and each The exible spaces approach maintains the con- other as possible. The use of asymmetric corridors nections between the parts of the VE but does not also proved to be ef cient. However, the walk- repeat the layouts. The changes in the layout occur ing direction and placement of the elements that as soon as the user leaves a room or a corridor, and

IEEE Computer Graphics and Applications 89 www.computer.org/computingedge 37 Spatial Interfaces

they are occluded by the other elements of the VE. problems with large open spaces and support for Figures 4a and 4b show a procedurally generated a completely free exploration within a limited real layout for a VE with two rooms and a user exploring workspace. To complicate matters further, the var- it. (See earlier work for a detailed explanation of the ious types of VEs with real walking support are not exible spaces algorithm.16) universal and often require adaptation to specic The constantly changing nature of the algorithm real-world workspaces. prevents users from building up spatial knowledge Another challenge for VR systems with large and forces them to rely on other means for orien- workspaces is estimating how many people a work- tation. Following the museum metaphor, we in- space could t. Moreover, how do we support the troduced room-to-door color coding. For example, simultaneous free exploration of multiple users a red door always leads to a red room, making it within the same VR system? For that, we need fast, content independent. reliable, and smart path-prediction algorithms In our pilot study, we demonstrated that spatial that take the user’s behavioral specics into con- overlap could be efciently used in cases where it sideration and novel methods to effectively coun- is not necessary for users to learn the spatial ar- ter any unexpected user behavior. rangement. Our test participants perceived the VE At this stage, VR researchers and developers as something possible in the real world, which dem- should continue to explore and learn to exploit the onstrates the benets of spatial manipulations for limits of human vision, perception, and cognition efcient workspace usage. in close contact with psychologists. Unfortunately, Another advantage of the exible spaces algo- a large gap still exists between experimental psy- rithm is its versatility. It can be used in the originally chology that uses very simple setups and stimulus proposed version or to generate unique, single-use and the demands of the striving eld of VR. This layouts for each session. The algorithm supports an gap needs to be bridged in order to keep pace with unlimited number and different sizes and shapes VR technology. of rooms or other conned spaces, and it can easily Lastly, the spread of consumer hardware is - be adapted to different room designs. Unlike other nally opening up possibilities for studying human techniques, the exible spaces algorithm guarantees adaptation to VR over a large population of users, unlimited walking with successful redirection and but it raises concerns regarding the consequences undetectable spatial overlap of up to 100 percent. of a long-term VR exposure. Simultaneously, we In a case with a particularly dense spatial arrange- need to address the individual differences and sen- ment, it is possible to extend the environment to sitivity of various users. For example, some users different levels with portals, ying, or a haptic el- still suffer from cybersickness, which sense ma- evator simulation (see Figure 4c).17 nipulation might contribute to or help to counter. It is crucial for both research and industry to deter- Challenges mine what is causing these unpleasant symptoms Spatial manipulation still requires a rather large and learn how to control them. Whether users will real space to create a believable VE. At the same develop an increased tolerance to the factors caus- time, our experience with exible spaces and ing cybersickness after long-term exposure to VR is self-overlapping architectures suggests that users still an unanswered question. might consciously accept spatial manipulations. However, some users might also nd the concept of an unrealistic architecture to be disturbing. References Moreover, there might be an unexplored spectrum 1. F. Steinicke et al., Human Walking in Virtual of new rules and techniques that users might con- Environments, Springer, 2013. sciously accept. As a next step, we plan to evolve 2. S. Razzaque, “Redirected Walking,” PhD disserta- the exible spaces algorithm to accommodate tion, Univ. of North Carolina at Chapel Hill, 2005. curved geometry. That, in turn, might improve the 3. T. Field, S. Bay, and P. Vamplew, “Generalised compatibility with rotation and curvature gains. Algorithms for Redirected Walking in Virtual As for the existing methods, we consider combin- Environments,” Proc. Int’l Conf. Arti cial Intelligence ing multiple existing nonintrusive approaches for in Science and Technology (AISAT), 2004, pp. 58–63. real walking support into a single ultimate tech- 4. T.C. Peck, H. Fuchs, and M.C. Whitton, “Improved nique to be one of the hardest tasks in achieving Redirection with Distractors: A Large-Scale-Real- more efcient virtual space compression. Although Walking Locomotion Interface and Its Effect on some attempts have already been made, no perfect Navigation in Virtual Environments,” Proc. IEEE technique has been found yet. There are still open Virtual Reality Conf., 2010, pp. 35–38.

9038 September/OctoberComputingEdge 2017 September 2019 5. F. Steinicke et al., “Estimation of Detection in computer science from the National Technical Univer- Thresholds for Redirected Walking Techniques,” sity of Ukraine “Kyiv Polytechnic Institute.” Contact her IEEE Trans. Visualization and Computer Graphics, vol. at [email protected]. 16, no. 1, 2010, pp. 17–27. 6. T. Grechkin et al., “Revisiting Detection Thresholds Hannes Kaufmann is an associate professor in the Insti- for Redirected Walking?: Combining Translation and tute of Software Technology and Interactive Systems at TU Curvature Gains,” Proc. Symp. Applied Perception, Wien. His research interests include virtual and augmented 2016, pp. 113–120. reality, real-time ray-tracing for AR, wide-area tracking, 7. Q. Sun, L.-Y. Wei, and A. Kaufman, “Mapping motion capture, redirected walking and 3D user interfaces. Virtual and Physical Reality,” ACM Trans. Graphics, Kaufmann has a PhD in technical sciences from TU Wien. vol. 35, no. 4, 2016, article 64. Contact him at [email protected]. 8. T.C. Peck, H. Fuchs, and M.C. Whitton, “An Evaluation of Navigational Ability Comparing Redirected Free Contact department editors Frank Steinicke at frank Exploration with Distractors to Walking-in-Place and [email protected] and Wolfgang Stuerzlinger at w.s Joystick Locomotion Interfaces,” Proc. IEEE Virtual @sfu.ca. Reality Conf., 2011, pp. 55–62. 9. D.J. Simons and R.A. Rensink, “Change Blindness: Past, Present, and Future,” Trends Cognitive Sciences, Read your subscriptions through vol. 9, no. 1, 2005, pp. 16–20. the myCS publications portal at This article originallyhttp://mycs.computer.org. appeared in 10. E.A. Suma et al., “Leveraging Change Blindness for IEEE Computer Graphics and Applications, Redirection in Virtual Environments,” Proc. IEEE vol. 37, no. 5, 2017. Virtual Reality Conf., 2011, pp. 159–166. 11. E.A. Suma, D.M. Krum, and M. Bolas, “Redirection on Mixed Reality Walking Surfaces,” Proc. IEEE VR Workshop Perceptual Illusions in Virtual Environments, 2011, pp. 33–35. 12. E.A. Suma et al., “Impossible Spaces: Maximizing Natural Walking in Virtual Environments with Self- Overlapping Architecture,” IEEE Trans. Visualization and Computer Graphics, vol. 18, no. 4, 2012, pp. 555–564. 13. K. Vasylevska and H. Kaufmann, “In uence of Path Complexity on Spatial Overlap Perception in Virtual Environments,” Proc. 25th Int’l Conf. Arti cial Reality and Telexistence and 20th Eurographics Symp. Virtual stay Environments (ICAT-EGVE), 2015, pp. 159–166. on the 14. K. Vasylevska and H. Kaufmann, “Towards Efcient Cutting Edge Spatial Compression in Self-Overlapping Virtual Environments,” Proc. IEEE Symp. 3D User Interfaces of Artificial Intelligence (3DU), 2017, pp. 12–21. 15. I. Moar and G.H. Bower, “Inconsistency in Spatial

IEEE January/fEbruary 2016

Also in this issue: IEEE Intelligent Systems provides peer- Knowledge,” Memory and Cognition, vol. 11, no. 2, aI’s 10 to Watch 56

January/FEB real-Time Taxi Dispatching 68 from flu Trends to Cybersecurity 84 ruary IEEE

1983, pp. 107–113. 2016 PUTTING AI INTO PRACTICE reviewed, cutting-edge articles on the 16. K. Vasylevska et al., “Flexible Spaces?: Dynamic

Layout Generation for Innite Walking in Virtual Online Beh theory and applications of systems A vi OrAl AnA OrAl

Environments,” Proc. IEEE Symp. 3D User Interfaces lysis that perceive, reason, learn, and (3DU), 2013, pp. 39–42. VOL uME 31 nu uME

MBE act intelligently. 17. K. Vasylevska and H. Kaufmann, “In uence of Vertical r 1 www.computer.org/intelligent Navigation Metaphors on Presence,” Challenging IS-31-01-C1 Cover-1 January 11, 2016 6:06 PM Presence: Proc. 15th Int’l Conf. Presence, A. Felnhofer and O.D. Kothgassner, eds., 2014, pp. 205–212.

Khrystyna Vasylevska is a PhD candidate in the School of The #1 AI Magazine Informatics at TU Wien. Her research interests include im- www.computer.org/intelligent IEEE mersive VR, redirected walking, 3D user interface design, brain-computer interfaces, and AI. Vasylevska has an MS www.computer.org/computingedge IEEE Computer Graphics and Applications 3991 CYBER-PHYSICAL SYSTEMS

Computer Security as Civil Defense

Marilyn Wolf, Georgia Tech

Given the prevalence of computer systems, we must change our approaches by ensuring that civilians and companies can become responsible for much of their own cyberdefense.

t is time for us to treat computer and information se- › The Notpetya attack of 20173 targeted data and curity and safety as civil defense issues. I use the term system confi gurations at several companies and civil defense here in its classic sense: the protection of resulted in extensive interruptions of company civilians against military attack and natural disas- operations as well as a lengthy recovery process. Iters. Computer systems, both IT and cyberphysical (CPS), › A 2015 cyberphysical attack on Ukrainian elec- or the Internet of Things (IoT) can wreak widespread and tric power facilities resulted in a temporary long-lasting damage to civilian lives and property. Given loss of electrical service to more than 100,000 the huge attack surface presented by civilian systems, we customers.2 have no choice but to rely on civilians for a great deal of › A cyberattack took down the computer systems their own cyberdefense. Ensuring that civilians are pre- of the Erie County Medical Center for six weeks in pared for cyberattacks and mishaps will require changes 2017.1 Medical staff relied on paper documentation in our approaches to both technology and policy. during the outage. These serious examples of the damage that can be THE STAKES ARE HIGH caused by computational attacks may, in fact, not provide Let’s keep in mind the huge stakes involved by reviewing us with a suffi ciently bleak picture of worst-case damage. a few recent incidents. Reasonable people may be concerned that we could see › The Target retail chain was the victim of a large data much worse in the future at the hands of a capable and de- breach in 2013.5 The attackers gained access to 11 GB termined adversary. of data. As a result, Target sent notices to 110 million Embedded computers are now in an astonishing va- credit and debit card holders. riety of physical objects. Although computers have improved physical systems in many ways, these innovations also mean that we can no longer treat computer security and physical safety as separate topics. Safe and secure cy- Digital Object Identifier 10.1109/MC.2019.2891980 Date of publication: 11 March 2019 berphysical and IoT systems were the subjects of a special

40 September 2019 Published by the IEEE Computer Society 2469-7087/19 © 2019 IEEE 54 COMPUTER PUBLISHED BY THE IEEE COMPUTER SOCIETY 0018-9162/19©2019IEEE EDITOR DIMITRIOS SERPANOS CYBER-PHYSICAL SYSTEMS ISI/ATHENA and University of Patras; [email protected]

issue of Proceedings of the IEEE edited of equipment could further equipment to operation depend not only by Dimitrios Serpanos (this column’s increase these backlogs, as could on the computers but also on the eq - editor) and myself.9 Security clearly af- attacks on the facilities that uipment. Power-generating equipment fects safety; safety also infl uences our manufacture such equipment. may take several hours to come online. approach to computer security. Chemical plants may require hours or All nations need to be concerned DEFENDING AGAINST CIVIL days for a shutdown/restart cycle. More- about their cyber civil defense and CYBERTHREATS over, software for safety-critical systems readiness. Beyond nation-to-nation Broadly speaking, we can identify sev- is held to a high standard; fast updates Computer Security strife, nonstate actors could also carry eral goals of computer civil defense: to correct security-related bugs may not out attacks, the eff ects of which give 1) protect the integrity of data, 2) pro- be possible while also ensuring that the them a much broader reach. The 9/11 at- tect the timely transfer of data, and updates do not cause further problems. tacks showed that physical attacks with 3) protect physical equipment. These We need software-engineering methods as Civil Defense large eff ects can be planned and carried goals are challenging in themselves. that result in fewer bug-fi x distributions. out by small groups;6 we should be sim- Computer civil defense is made even Design techniques for graceful deg- Marilyn Wolf, Georgia Tech ilarly concerned about the potential for harder because of the wide variation radation have received extensive at- large-scale computational attacks car- in equipment and confi guration and tention over many decades. However, ried out by relatively small groups from computer system operators’ relative these methods are applied primarily in Given the prevalence of computer systems, well within their own safe havens. lack of expertise. certain types of high-reliability systems. we must change our approaches by CYBER AND CYPERPHYSICAL ensuring that civilians and companies can THREATS Several types of threats are posed by Security clearly aﬀ ects safety; safety also inﬂ uences become responsible for much of their own cyber and cyberphysical attacks. › Disruptions of service can aff ect our approach to computer security. cyberdefense. both information systems and physical systems. The lines t is time for us to treat computer and information se- › The Notpetya attack of 20173 targeted data and between IT and CPS/ the IoT are We can identify technical steps, Attacks that disrupt operations on IT curity and safety as civil defense issues. I use the term system confi gurations at several companies and often blurry. As one example, ranging from known best practices systems suggest that more types of civil defense here in its classic sense: the protection of resulted in extensive interruptions of company IT failures at three U. S. air- to research topics, that can reduce cy- systems should be designed to pro- civilians against military attack and natural disas- operations as well as a lengthy recovery process. lines caused fl ight delays and berthreats. Some of these methods vide some functionality in the face of Iters. Computer systems, both IT and cyberphysical (CPS), › A 2015 cyberphysical attack on Ukrainian elec- cancelations.4,7,8 should be practiced by manufacturers. failures to other parts of the system. or the Internet of Things (IoT) can wreak widespread and tric power facilities resulted in a temporary › Identity theft enables follow-on Root-of-trust design, which ensures Defense-in-depth methods are not con- long-lasting damage to civilian lives and property. Given loss of electrical service to more than 100,000 crimes. Beyond credit card that critical software can be traced back sistently applied. The Target attack, for the huge attack surface presented by civilian systems, we customers.2 fraud, attackers could use stolen to a trusted source, is employed in prac- example, came through a cybersecurity have no choice but to rely on civilians for a great deal of › A cyberattack took down the computer systems credentials for improper access tice but not universally. Root-of-trust weakness of a refrigeration contractor. their own cyberdefense. Ensuring that civilians are pre- of the Erie County Medical Center for six weeks in to facilities or data. design uses a combination of hardware System design should also take into ac- pared for cyberattacks and mishaps will require changes 2017.1 Medical staff relied on paper documentation › Cyberphysical attacks can dam- and software methods: digital signa- count the time required to recover from in our approaches to both technology and policy. during the outage. age equipment. The Ukrainian tures for software are checked, access attacks. The six weeks required to re- These serious examples of the damage that can be power grid attack targeted power privileges for trusted versus nontrusted cover from the attack at the Erie County THE STAKES ARE HIGH caused by computational attacks may, in fact, not provide control devices but operated non- software are enforced, and digital sig- Medical Center is not an isolated exam- Let’s keep in mind the huge stakes involved by reviewing us with a suffi ciently bleak picture of worst-case damage. destructively, allowing workers natures may be applied at several levels ple. Long recovery times amplify the a few recent incidents. Reasonable people may be concerned that we could see to manually reset the equipment. of deployment. damage caused by an attack. › The Target retail chain was the victim of a large data much worse in the future at the hands of a capable and de- A variation of the attack could More controversial is a move to- Cyberphysical systems are sensi- breach in 2013.5 The attackers gained access to 11 GB termined adversary. have resulted in permanent ward lessened reliance on software tive not just to data values but also to of data. As a result, Target sent notices to 110 million Embedded computers are now in an astonishing va- damage. Industrial equipment updates. This is one example of physi- timing—we can disrupt many control credit and debit card holders. riety of physical objects. Although computers have im- often has replacement lead times cal safety infl uencing our approach to systems merely by delaying critical proved physical systems in many ways, these innovations measured in weeks or months, computer security. Updating control- data without changing a single bit of also mean that we can no longer treat computer security resulting in extended outages. lers for physical systems is diffi cult for information. Research has developed and physical safety as separate topics. Safe and secure cy- A large-scale attack damaging several reasons. Shutting down equip- some architectures that preserve tim- Digital Object Identifier 10.1109/MC.2019.2891980 Date of publication: 11 March 2019 berphysical and IoT systems were the subjects of a special an unusually large amount ment for updates and then returning the ing properties. Timing resilience—the

detection of timing problems and re- could be independent nonprofits, sup- .gov/_layouts/ntsb.aviation sponses to preserve system function— ported by local or state government, or /index.aspx) and railroad accidents deserves more study. IT-oriented ap- national government organizations. (https://www.ntsb.gov/investigations proaches to CPS and IoT security tend to We should expect that these organi- /AccidentReports/Pages/railroad. treat these systems as collections of in- zations will cooperate to provide ser- aspx) that serve as examples of inci- put–output devices to reduce the safety vice—national or international orga- dent reporting and analysis. In some and security problems of traditional IT nizations may provide expertise on cases, safety recommendations or approaches. In fact, cyberphysical and specialized topics that smaller organi- maintenance alerts may be made as IoT systems perform distributed re- zations cannot afford. a result of accidents. al-time computations that require new An important role of cybercivil de- In contrast, the National Vulnerabil- security and safety methodologies. fense organizations can be to dissem- ity Database (NVD) maintained by the Some quasi-technical factors also inate useful information and provide U.S. National Institute for Standards contribute to cybersecurity threats. training. The 21st century equivalents and Technology (https://nvd.nist Some IT personnel have received rela- to pamphlets on bomb shelter construc- .gov/) concentrates on code. The NVD defines vulnerability as “a weakness in the computational logic (e.g., code) found in software and hardware com- A code-centric view of security minimizes the ponents that, when exploited, results in a negative impact to confidentiality, importance of system architecture and procedures integrity, or availability” (https://nvd followed by personnel. .nist.gov/vuln). A code-centric view of security minimizes the importance of system architecture and procedures tively little formal training in IT after tion could provide useful information followed by personnel. promotion from technician or sup- to individuals and companies on how Mandatory standards may be ap- port roles. Training in cybersecurity to prepare for cyberattacks. The cyber propriate in some cases—for example, is relatively new and may not have equivalent of duck-and-cover drills aircraft certification takes into ac - reached all practitioners. Personnel could educate citizens on the nature count some aspects of cybersecurity with training and experience in cy- of threats and appropriate responses and software safety. Mandates may berphysical or IoT security are even to unexpected events. Consider, for ex- help to overcome manufacturer iner- harder to find. ample, an attack on automobiles that tia, with air bag regulations providing Computer people pride themselves interferes with their operation while a classic example. on the generality of computers. The re- on the road—a little preparation and sult is that we see a huge variation in practice could drivers them how to re- REGULATIONS deployment configurations for devices act to minimize risk. AND STANDARDS and networks. Such variation makes Governments should consider en- We need to be sure that export regu- security holes more likely and secu- couraging or requiring reporting. Cy- lations do not unnecessarily restrict rity properties harder to monitor. The berattacks are not always reported by technologies that promote cybersecu- use of more typical configurations for companies because of concerns about rity and safety. Some technologies will devices and networks would help to re- bad publicity or reliability. In contrast, always be closely guarded. Regulators duce problems and simplify fixes when accidents in several domains, such as should take into account both risks problems are identified. transportation, are required by law to be and rewards when designing protec- reported. Information gleaned from at- tions. Global supply chains mean that POLICIES TO RAISE tacks can be used to learn about attack- export controls have a broad reach AWARENESS ers’ methods and develop responses. that may keep important technologies Policy will need to reinforce our un- Reporting systems can be designed to from being adopted. Also, Internet at- derstanding of risks and how we can protect confidential data while provid- tacks can be conducted by devices that best prepare ourselves. Organizations ing useful public knowledge—patent lit- have never entered the country. can help to educate the citizenry and igation regularly uses protection orders Regulators need to treat cyberse- encourage cybersecurity efforts. Such for confidential data while conducting curity and safety as top-of-the-list organizations will need to operate lo- the main business of the case in public. concerns. Electric power utilities put a cally and provide a personal touch—ad The U.S. National Transportation great deal of effort into traditional re- campaigns won’t cause enough people Board keeps public databases of avi- liability in case of storms and natural to change their ways. Organizations ation accidents (https://www.ntsb disasters; regulators require utilities

42 ComputingEdge September 2019 56 COMPUTER WWW.COMPUTER.ORG/COMPUTER CYBER-PHYSICAL SYSTEMS

detection of timing problems and re- could be independent nonprofits, sup- .gov/_layouts/ntsb.aviation to be prepared for such events and im- 3. A. Greenberg, “The untold story glitch causes cancellations, delays sponses to preserve system function— ported by local or state government, or /index.aspx) and railroad accidents pose fines for certain types of power of Notpetya, the most devastating for third day,” The Washington Post, deserves more study. IT-oriented ap- national government organizations. (https://www.ntsb.gov/investigations outages. Cyberthreats are arguably a cyberattack in history,” Wired, July 22, 2016. [Online]. Available: proaches to CPS and IoT security tend to We should expect that these organi- /AccidentReports/Pages/railroad. lower priority at some utilities because Aug. 22, 2018. [Online]. Available: https://www.washingtonpost.com treat these systems as collections of in- zations will cooperate to provide ser- aspx) that serve as examples of inci- their regulators do not place high im- https://www.wired.com/story /news/dr-gridlock/wp/2016/07/21 put–output devices to reduce the safety vice—national or international orga- dent reporting and analysis. In some portance on such threats. Cyberattacks /notpetya-cyberattack-ukrain /long-lines-for-southwest-a and security problems of traditional IT nizations may provide expertise on cases, safety recommendations or have been much less frequent than, e-russia-code-crashed-the-world/ irlines-passengers-at-area-airports/ approaches. In fact, cyberphysical and specialized topics that smaller organi- maintenance alerts may be made as for example, weather-caused outages. 4. A. Halsey III, “Delta computers crash, 8. E. Ortiz, J. Shamlian, and T. IoT systems perform distributed re- zations cannot afford. a result of accidents. Unfortunately, the consequences of causing delays and cancellations. Costello, “United Airlines al-time computations that require new An important role of cybercivil de- In contrast, the National Vulnerabil- a cyberattack could be huge and long Experts say it shouldn’t have flights no longer grounded, delays security and safety methodologies. fense organizations can be to dissem- ity Database (NVD) maintained by the lasting. Regulators need to find ways happened,” The Washington Post, remain,” NBC News, July 8, 2015. Some quasi-technical factors also inate useful information and provide U.S. National Institute for Standards to encourage utilities of all types—for Aug. 8, 2016. [Online]. Available: [Online]. Available: http:// contribute to cybersecurity threats. training. The 21st century equivalents and Technology (https://nvd.nist example, electric, natural gas, water, https://www.washingtonpost www.nbcnews.com/business Some IT personnel have received rela- to pamphlets on bomb shelter construc- .gov/) concentrates on code. The NVD sewage, and transportation—to plan .com/local/trafficandcommut- /travel/united-airlines-passengers- defines vulnerability as “a weakness for these new threats. ing/delta-airlines-compute say-flights-grounded-nation- in the computational logic (e.g., code) Voluntary standards have proven r-systems-crash-causing-flight- wide-n388536 found in software and hardware com- useful in other domains. The Energy delays-and-cancellations/2016 9. M. Wolf and D. Serpanos, “Safety A code-centric view of security minimizes the ponents that, when exploited, results Star ratings used in the United States /08/08/7d5e8fa0-5d72-11e6-af8e- and Security in Cyber-Physical in a negative impact to confidentiality, were created by the federal govern- 54aa2e849447_story.html?utm_ Systems and Internet-of-Things importance of system architecture and procedures integrity, or availability” (https://nvd ment and are voluntary. A wide range term=.cb218d77d206 systems,” Proc. IEEE, vol. 106, no. followed by personnel. .nist.gov/vuln). A code-centric view of of consumer products advertise their 5. M. Kassner, “Anatomy of the Target 1, pp. 9–20, Jan. 2018. doi: 10.1109/ security minimizes the importance of Energy Star ratings. Manufacturers can data breach: Missed opportunities JPROC.2017.2781198. system architecture and procedures employ voluntary systems to advertise and lessons learned,” ZDNet, tively little formal training in IT after tion could provide useful information followed by personnel. their security capabilities and allow Feb. 2, 2015. [Online]. Available: promotion from technician or sup- to individuals and companies on how Mandatory standards may be ap- consumers to vote with their wallets. https://www.zdnet.com/article MARILYN WOLF is the Rhesa “Ray” port roles. Training in cybersecurity to prepare for cyberattacks. The cyber propriate in some cases—for example, /anatomy-of-the-target-data-breach- S. Farmer, Jr., distinguished chair is relatively new and may not have equivalent of duck-and-cover drills aircraft certification takes into ac - missed-opportunities-and- in Embedded Computing Systems reached all practitioners. Personnel could educate citizens on the nature count some aspects of cybersecurity yberthreats to our data and lessons-learned/ and Georgia Research Alliance Eminent Scholar at the Georgia In- with training and experience in cy- of threats and appropriate responses and software safety. Mandates may our physical world are real, 6. National Commission on Terrorist stitute of Technology. She is a Fel- berphysical or IoT security are even to unexpected events. Consider, for ex- help to overcome manufacturer iner- and they will not go away. The Attacks Upon the United States, “The low of the IEEE and ACM. Contact harder to find. ample, an attack on automobiles that tia, with air bag regulations providing Cpervasive adoption of computer tech- 9/11 report,” July 22, 2004. [Online]. her at [email protected]. Computer people pride themselves interferes with their operation while a classic example. nology has given us huge benefits but Available: https://www.9-11commission on the generality of computers. The re- on the road—a little preparation and also new types of risk. A civil defense .gov/report/ sult is that we see a huge variation in practice could drivers them how to re- REGULATIONS approach to cybersecurity and safety 7. J. W. Moyer, D. Hedgpeth, and F. Sid- deployment configurations for devices act to minimize risk. AND STANDARDS can help the citizenry protect itself diqui, “Southwest Airlines computer This article originally appeared in and networks. Such variation makes Governments should consider en- We need to be sure that export regu- against attacks and effectively respond Computer, vol. 52, no. 1, 2019. security holes more likely and secu- couraging or requiring reporting. Cy- lations do not unnecessarily restrict to the inevitable attempts by bad actors rity properties harder to monitor. The berattacks are not always reported by technologies that promote cybersecu- to interfere with daily life. use of more typical configurations for companies because of concerns about rity and safety. Some technologies will devices and networks would help to re- bad publicity or reliability. In contrast, always be closely guarded. Regulators REFERENCES duce problems and simplify fixes when accidents in several domains, such as should take into account both risks 1. CBS News, “Inside the New York hos- problems are identified. transportation, are required by law to be and rewards when designing protec- pital hackers took down for 6 weeks,” reported. Information gleaned from at- tions. Global supply chains mean that Aug. 18, 2017. [Online]. Available: POLICIES TO RAISE tacks can be used to learn about attack- export controls have a broad reach https://www.cbsnews.com AWARENESS ers’ methods and develop responses. that may keep important technologies /news/cbsn-on-assignment- Policy will need to reinforce our un- Reporting systems can be designed to from being adopted. Also, Internet at- hackers-targeting-medical-industry- derstanding of risks and how we can protect confidential data while provid- tacks can be conducted by devices that hospitals/ best prepare ourselves. Organizations ing useful public knowledge—patent lit- have never entered the country. 2. D. Goodin, “First known hack- can help to educate the citizenry and igation regularly uses protection orders Regulators need to treat cyberse- er-caused power outage signals trou- encourage cybersecurity efforts. Such for confidential data while conducting curity and safety as top-of-the-list bling escalation,” ArsTechnia, Jan. organizations will need to operate lo- the main business of the case in public. concerns. Electric power utilities put a 14, 2016. [Online]. Available: http:// cally and provide a personal touch—ad The U.S. National Transportation great deal of effort into traditional re- arstechnica.com/security/2016/01 campaigns won’t cause enough people Board keeps public databases of avi- liability in case of storms and natural /first-known-hacker-caused-power- to change their ways. Organizations ation accidents (https://www.ntsb disasters; regulators require utilities outage-signals-troubling-escalation/

www.computer.org/computingedge 43 56 COMPUTER WWW.COMPUTER.ORG/COMPUTER JANUARY 2019 57 Education Awards Nominations Call for Award Nominations Deadline: 1 October 2019

Taylor L. Booth Education Award Computer Science and Engineering A bronze medal and $5,000 honorarium Undergraduate Teaching Award are awarded for an outstanding record A plaque, certifi cate, and a honorarium of in computer science and engineering $2,000 is awarded to recognize outstanding education. The individual must meet two contributions to undergraduate education or more of the following criteria in the through both teaching and service and computer science and engineering fi eld: for helping to maintain interest, increasing ■ Achieving recognition as a teacher the visibility of the society, and making of renown. a statement about the importance of undergraduate education. ■ Writing an infl uential text. ■ Leading, inspiring, or providing The award nomination requires a minimum signifi cant education content during the of three endorsements. creation of a curriculum in the fi eld. Read more award details at bit.ly/cs-eu ■ Inspiring others to a career in computer science and engineering education. Robert R. Kessler Two endorsements are required for an The University of Utah award nomination. 2019 Award Recipient Read more information at bit.ly/taylor-booth

Susan H. Rodger Duke University Nomination 2019 Award Recipient Deadline Submit your nomination by 1 October 2019 Contact us at [email protected] EDITOR DIMITRIOS SERPANOS ISI/ATHENA and University of Patras; CYBER-PHYSICAL SYSTEMS [email protected]

Cyber-Physical Systems and Digital Twins in the Industrial Internet of Things

Christos Koulamas and Athanasios Kalogeras, CPSs lie at the cross section of Industrial Systems Institute/ATHENA the physical and digital worlds. Integrating physical processes and A digital twin is a virtual representation that computer systems is the main challenge presented by them, as serves as the real-time digital counterpart of a the computational cyber part continuously senses the state of the physical object or process and addresses every physical system and applies decisions and actions for its control. instance for its total life cycle. CPSs present a wide range of applications in different sectors, including manufacturing, energy, perational models and other virtual represen- health care, consumer services, and monitoring of criti- tations of cyber-physical systems (CPSs) are a cal infrastructures. common industrial engineering practice to- CPSs are mostly networked systems characterized by day. The evolution of Internet of Things (IoT) distribution of functions and often wireless connectivity Oand AI technologies enables complex interactions of such between intelligent physical devices, providing sensing virtual representations for the total lifetime of system in- and actuating as well as control capabilities. Real-time stances under the digital twin (DT) concept, which poses behavior is a critical challenge, as the continuous moni- a number of challenges for its seamless integration in the toring and control of the physical world has to be ascer- modern industrial environment. tained. They represent complex, flexible, and adaptive systems, whose constituent elements are characterized by increased autonomy and intelligence. Cybersecurity Digital Object Identifier 10.1109/MC.2018.2876181 Date of publication: 15 January 2019 mechanisms need to be integrated in a holistic approach

toward detection of attacks, resilience, Still, the DT concept attempts to mate- relevant modeling engineering in de- and privacy concerns. rialize a bidirectional integration of the sign, simulation, and testing: the per- The Industrial IoT (IIoT) is an en - digital and physical worlds, intercon- manent connection between the real abling technology of CPSs, providing necting physical things with their digi- and the virtual part for the total life the networking infrastructure for phys- tal counterparts while also bringing to cycle of a specific system instance. This ical objects to sense, communicate, and the physical world changes to its DT. connection means that information interact. The spread of the IIoT has led to an Gartner included DT as top strate- exchange between the system instance explosion of data and information. With gic technology trend number four for and its DT counterpart (that is, sensing 21 billion connected things by 2020,1 the 2018 (among the top 10 trends that will and often also actuating infrastruc- IIoT market is estimated to add $14.2 tril- contribute to the intelligent digital ture) can be part of a cyber-physical sys- lion to the global economy by 2030.2 mesh, that is, the integration of things, tem on its own, depending on the type Manufacturing, connected logistics and services, content, and people). of information exchange—whether a transportation, and energy and utilities One direct utilization of a DT is in real-time data flow or some systematic represent the three largest markets for the field of asset management. Being data collection that is integrated offline the IIoT. Digitization of these markets the cyber twin of a physical thing and in the behavior of the DT. creates a plethora of data and new oppor- having access to real-time information tunities for companies to extract knowl- regarding the physical thing as well INTEGRATION CHALLENGES edge out of these data. as to related historical data, the DT The need for a bidirectional life cycle - can help optimize physical asset per- extended integration between the THE DT CONCEPT formance through efficient predictive physical world and its DT mandates a The increasing availability and ubiq- and preventive maintenance opera- relevant supporting reference architec- uity of real-time operational data, as tions, thus reducing overall mainte- ture. Different initiatives deal with the well as the boost of AI implementation nance costs and downtime. IIoT providing relevant reference archi- capabilities in learning and reasoning, Furthermore, the DT can simulate tectures, the most important of which represent drivers toward realizing a vi- the behavior of the physical thing that are Industry 4.0, the Industrial Internet sion of physical products or processes it is twinned with—or of an associated Consortium (IIC), and Society 5.0. having accompanying virtual repre- process—and can thus contribute sig- The Reference Architecture Model sentations that evolve throughout their nificantly to performance optimiza- for Industry 4.0 (RAMI 4.0) is a three- entire life cycle. Such virtual represen- tion. It can act as a tool for predictive dimensional model along three axes tations, or DTs, represent real-time dig- analysis, predicting the performance of (hierarchy, architecture, and product ital counterparts of physical objects. the physical thing or its associated pro- life cycle), with IT security and data pri- There is not any unique, globally access. If this example is enlarged in the vacy as enablers (see the right side of cepted, and common closed definition scope of a digital enterprise, then the Figure 1).3 The Industrial Internet Ref- of the DT concept; however, there are overall process, production system, or erence Architecture (IIRA) of the IIC4 certain aspects on which most existing product may be optimized. Potential comprises four different viewpoints: definitions agree.7 A DT is virtual (that benefits include, among others, opti- business, usage, functional, and imple- is, digital), it includes both static (that mizing production scheduling, identi- mentation. The functional viewpoint is, design documents, process specifi- fying potential bottlenecks, assessing (see the left side of Figure 1) is, in turn, cations, and so forth) and dynamic (that asset utilization, and minimizing pro- divided into five domains (control, op- is, data acquisition and simulation) duction lead times. erations, information, application, and parts, and it addresses every instance The DT offers a total life cycle - ap business), four cross-cutting functions of its twin product or process for its to- proach with reference to its physical (connectivity, distributed data man- tal life cycle. twin, either a thing or a process. Prod- agement, analytics, and intelligent and Could DT be a new marketing buzz- uct design, new product launch, manu- resilient control), and six system char- word for a concept that already exists? facturing process setup, and integrated acteristics (safety, security, resilience, There might be some truth in this: a part supply chain management are facili- reliability, privacy, and scalability). of the DTs’ expected capabilities (for ex- tated. It is essential, though, to point The implementation viewpoint uti- ample, precise simulation of the physical out the principal difference of the vir- lizes a three-tier implementation archi- thing’s behavior) could already be uti- tual representation of a DT compared tecture comprising the enterprise, plat- lized in current engineering practices. with well-known and widely used form, and edge tiers (see the lower part

46 ComputingEdge September 2019 96 COMPUTER WWW.COMPUTER.ORG/COMPUTER CYBER-PHYSICAL SYSTEMS

of Figure 1). A mapping between RAMI guide some optimization in the product frameworks are already in the market, 4.0 and IIRA is also possible.5 Finally, So- or process. Some action is triggered in justifying their necessity in various in- ciety 5.0 represents the related Japanese the real world by the DT to achieve this. dustrial settings.10 This is expected to initiative driving toward a new hyper- Furthermore, as the cyber and phys- lead to a wider adoption of this specific toward detection of attacks, resilience, Still, the DT concept attempts to mate- relevant modeling engineering in de- smart society and extending to differ- ical perspectives of the DT may already architectural paradigm, considering the and privacy concerns. rialize a bidirectional integration of the sign, simulation, and testing: the per- ent application domains. Its enabling blur the borders between the DT of a high achievable degree of containment The Industrial IoT (IIoT) is an en - digital and physical worlds, intercon- manent connection between the real technologies comprise the Internet of complex cyber-physical system and the for a relatively small “twin” of a tiny but abling technology of CPSs, providing necting physical things with their digi- and the virtual part for the total life Things, big data, ambient intelligence, system itself, embedded technology critical “thing,” which is able to create, the networking infrastructure for phys- tal counterparts while also bringing to cycle of a specific system instance. This and robotics.6 evolution has already started to chal- train, consult, and adapt its DT onboard ical objects to sense, communicate, and the physical world changes to its DT. connection means that information The interweaving of the DT with its lenge typical architectural patterns in and in real time without any interaction interact. The spread of the IIoT has led to an Gartner included DT as top strate- exchange between the system instance physical counterpart starts with the cap- the realization of DTs, which usually with the cloud. Such twins can be then explosion of data and information. With gic technology trend number four for and its DT counterpart (that is, sensing ture of data generated by sensing things call for relatively heavy centralized envisaged as capable of being combined 21 billion connected things by 2020,1 the 2018 (among the top 10 trends that will and often also actuating infrastruc- in the manufacturing environment. computing power and relevant data in a system-of-systems fashion to cre- IIoT market is estimated to add $14.2 tril- contribute to the intelligent digital ture) can be part of a cyber-physical sys- This provides the DT with real-time data center and cloud infrastructures. There ate larger distributed models for DTs of lion to the global economy by 2030.2 mesh, that is, the integration of things, tem on its own, depending on the type of the physical world it is twinned with. are applications that have inherent highly complex cyber-physical systems. Manufacturing, connected logistics and services, content, and people). of information exchange—whether a This step can be mapped to the asset characteristics and requirements that The DT concept in the IIoT context transportation, and energy and utilities One direct utilization of a DT is in real-time data flow or some systematic layer of the architecture axis of RAMI provide a natural fit to highly distrib- generates a number of challenges. represent the three largest markets for the field of asset management. Being data collection that is integrated offline 4.0. These data are then aggregated and uted intelligence at the edge. However, Its constituent elements are quite di- the IIoT. Digitization of these markets the cyber twin of a physical thing and in the behavior of the DT. combined with historical data pertain- the benefits of similar setups, mainly verse. Product and production process creates a plethora of data and new oppor- having access to real-time information ing to the manufacturing process as network bandwidth and cloud process- models must be seamlessly integrated tunities for companies to extract knowl- regarding the physical thing as well INTEGRATION CHALLENGES well as relevant data at the enterprise ing cost reductions but also responsive- with simulation and prediction models, edge out of these data. as to related historical data, the DT The need for a bidirectional life cycle - level. This step drives from the physical ness, dependability, scalability, and se- as well as with tools and systems dealing can help optimize physical asset per- extended integration between the to the cyber part and corresponds to the curity improvements, can be exploited with data analytics and optimization; THE DT CONCEPT formance through efficient predictive physical world and its DT mandates a integration layer of RAMI 4.0. in a wider set of domains, especially in nontypical, embedded, and mobile com- The increasing availability and ubiq- and preventive maintenance opera- relevant supporting reference architec- Then, at the communication layer, fault detection and diagnosis for pre- puting platforms must also be consid- uity of real-time operational data, as tions, thus reducing overall mainte- ture. Different initiatives deal with the data move to the fog, the edge, or the ventive and predictive maintenance. ered. This interoperability challenge of well as the boost of AI implementation nance costs and downtime. IIoT providing relevant reference archi- cloud depending on the architecture There is ongoing research on en- combining completely different models, capabilities in learning and reasoning, Furthermore, the DT can simulate tectures, the most important of which that is followed. Data analytics are ap- abling AI capabilities in embedded systems, and tools represents an area represent drivers toward realizing a vi- the behavior of the physical thing that are Industry 4.0, the Industrial Internet plied to these data, and useful informa- devices,8,9 while specialized hardware in need of significant research. Map- sion of physical products or processes it is twinned with—or of an associated Consortium (IIC), and Society 5.0. tion is derived. This information can and real-time embedded analytics ping and integration of the DT and its having accompanying virtual repre- process—and can thus contribute sig- The Reference Architecture Model sentations that evolve throughout their nificantly to performance optimiza- for Industry 4.0 (RAMI 4.0) is a three- entire life cycle. Such virtual represen- tion. It can act as a tool for predictive dimensional model along three axes tations, or DTs, represent real-time dig- analysis, predicting the performance of (hierarchy, architecture, and product ital counterparts of physical objects. the physical thing or its associated pro- life cycle), with IT security and data pri- There is not any unique, globally access. If this example is enlarged in the vacy as enablers (see the right side of cepted, and common closed definition scope of a digital enterprise, then the Figure 1).3 The Industrial Internet Ref- IIRA Functional Viewpoint RAMI 4.0 Hierarchy Levels 4 of the DT concept; however, there are overall process, production system, or erence Architecture (IIRA) of the IIC Business Connected World certain aspects on which most existing product may be optimized. Potential comprises four different viewpoints: 7 definitions agree. A DT is virtual (that benefits include, among others, opti- business, usage, functional, and imple- Enterprise is, digital), it includes both static (that mizing production scheduling, identi- mentation. The functional viewpoint DT is, design documents, process specifi- fying potential bottlenecks, assessing (see the left side of Figure 1) is, in turn, Components Work Centers Application Operations and Interfaces cations, and so forth) and dynamic (that asset utilization, and minimizing pro- divided into five domains (control, op- Information is, data acquisition and simulation) duction lead times. erations, information, application, and Connectivity Control Station Asset Industrial Analytics Business Functional Integration parts, and it addresses every instance The DT offers a total life cycle - ap business), four cross-cutting functions Information of its twin product or process for its to- proach with reference to its physical (connectivity, distributed data man- Sense and Actuation Communication Distributed Data Management

Intelligent and Resilient Control Control Device tal life cycle. twin, either a thing or a process. Prod- agement, analytics, and intelligent and IIRA Implementation Viewpoint Could DT be a new marketing buzz- uct design, new product launch, manu- resilient control), and six system char- Physical Systems word for a concept that already exists? facturing process setup, and integrated acteristics (safety, security, resilience, Field Device

There might be some truth in this: a part supply chain management are facili- reliability, privacy, and scalability). IIRA Crosscutting Edge RAMI 4.0 Layers

Platform Product of the DTs’ expected capabilities (for ex- tated. It is essential, though, to point The implementation viewpoint uti- Functions Enterprise RAMI 4.0 Life Cycle ample, precise simulation of the physical out the principal difference of the vir- lizes a three-tier implementation archi- Value Stream thing’s behavior) could already be uti- tual representation of a DT compared tecture comprising the enterprise, plat- lized in current engineering practices. with well-known and widely used form, and edge tiers (see the lower part FIGURE 1. The CPSs and digital twins (DTs) in the Industrial IoT.

www.computer.org/computingedge 47 96 COMPUTER WWW.COMPUTER.ORG/COMPUTER NOVEMBER 2018 97 CYBER-PHYSICAL SYSTEMS This article originally appeared in Computer, vol. 51, no. 11, 2018.

functionalities on the prevalent refer- 2. Accenture. (2015). Winning with the volume G1: Reference architec- ence architectures for the IIoT is also industrial Internet of things. Accen- ture. Industrial Internet Consortium. a necessity (see Figure 1). ture. [Online]. Available: [Online]. Available: https://www https://www.accenture.com .iiconsortium.org/IIC_PUB_G1_ /t20160909T042713Z—w— V1.80_2017-01-31.pdf he DT is still mostly at a concep- /us-en/_acnmedia/Accenture 5. S.-W. Lin, B. Murphy, E. Clauer, U. tual stage, in terms of demonstrat- /Conversion-Assets/DotCom Loewen, R. Neubert, G. Bachmann, ing wide industrial adoption and /Documents/Global/PDF/Dualpub_11 M. Pai, and M. Hankel. (2017). Archi- Tbecoming a well-defined engineering /Accenture-Industrial-Internet- tecture alignment and interoperabil- practice within the industry. There is of-Things-Positioning- ity Industrial Internet Consortium a need for research dealing with the Paper-Report-2015.pdfla=en and Platform Industrie 4.0 Joint previously listed challenges. Further, 3. P. Adolfs, S. Berlik, W. Dorst, Whitepaper. Industrial Internet there is a need for a unified framework J. Friedrich, C. Gericke, M. Hankel, R. Consortium, Needham, MA. [Online]. to build, out of the corresponding physi- Heidel, M. Hoffmeister, C. Mosch, R. Available: https://www.iiconsortium cal world model, its DT. This framework Pichler, U. Rauschecker, T. Schulz, K. .org/pdf/JTG2_Whitepaper_ should offer the full range of tools neces- Schweichhart, E. J. Steffens, M. Taube, final_20171205.pdf sary for DT operation. I. Weber, M. Wollschlaeger, and S. Mät- 6. Japanese Cabinet Office, “Soci- zler, “Reference Architecture Model ety 5.0,‘” Japanese Cabinet Office. REFERENCES Industrie 4.0 (RAMI4.0),” Deutsche Accessed on: Nov. 19, 2018. [Online]. 1. M. Hung. (2017). Leading the IoT. Institut für Normung e.V. (DIN), Berlin, Available: http://www8.cao.go.jp Gartner insights on how to lead in a Germany, DIN SPEC 91345, 2016. /cstp/english/society5_0/index.html connected world. Gartner. [Online]. 4. S.-W. Lin, B. Miller, J. Durand, G. 7. l. Negri, L. Fumagalli, and M. Mac- Available: https://www.gartner Bleakley, A. Chigani, R. Martin, B. chi, “A review of the roles of digital .com/imagesrv/books/iot/iotEbook_ Murphy, and M. Crawford. (2017). twin in CPS-based production sys- digital.pdf The industrial Internet of things, tems,” Procedia Manufacturing, vol. 11, pp. 939–948, Sept. 2017. 8. N. D. Lane and P. Warden, “The deep (learning) transformation of mobile and embedded computing,” IEEE Com- Call for Articles puter, vol. 51, no. 5, pp. 13–16, May 2018. 9. J. Tang, D. Sun, S. Liu, and J.-L. Gaud- iot, “Enabling deep learning on IoT IEEE Software seeks devices,” IEEE Computer, vol. 50, no. practical, readable articles 10, pp. 92–96, Oct. 2017. 10. C. Middleton. (2018) Digital twin that will appeal to experts AIs designed to learn at the edge. and nonexperts alike. The Control Engineering. [Online]. Avail- magazine aims to deliver able: https://www.controleng.com reliable information to software /single-article/digital-twin-ais- designed-to-learn-at-the-edge developers and managers to help them stay on top of rapid technology change. Submissions must be original and no more CHRISTOS KOULAMAS is a research director at the Industrial Systems than 4,700 words, including 250 Institute/ ATHENA Research Center. words for each table and gure. He is a Senior Member of IEEE. Con- tact him at [email protected].

ATHANASIOS KALOGERAS is a re- Author guidelines: search director at the Industrial Sys- www.computer.org/software/author tems Institute/ATHENA Research Further details: [email protected] Center. He is a Senior Member of IEEE. Contact him at [email protected]. www.computer.org/software

48 ComputingEdge September 2019 98 COMPUTER WWW.COMPUTER.ORG/COMPUTER Harry H. Goode Memorial Award Call for Award Nominations Deadline: 1 October 2019

2019 Harry H. Goode Named for a pioneer and leader in the field of systems Memorial Award engineering, the Harry H. Goode Memorial Award Recipient was established to encourage further developments in and honor outstanding contributions to the field of information processing sciences. A bronze medal and $2,000 are awarded by the IEEE Computer Society on the basis of achievements in the information processing field.

About Harry H. Goode Marilyn C. Wolf One of the first scientists to fully comprehend the Georgia Institute of Technology powers and abilities of computers, Harry H. Goode formulated many principles of systems engineering For contributions to and developed techniques for the design, analysis, and embedded, hardware- software codesign, and evaluation of large-scale systems. He was instrumental real-time computer in initiating early systems projects, including the vision systems. Typhoon computer and Whirlwind computer at MIT. He participated in the study that led to the creation of the Bomarc missile and conceived and developed the Air Defense Integrated System Project.

Nomination Requirements This award requires a minimum of three endorsements. Nominations are being accepted electronically by 1 October 2019 to bit.ly/harry-goode.

Questions? Visit bit.ly/harry-goode or contact [email protected] CAREER OPPORTUNITIES

Use promo code Get SWCM to save up to an additional $200 off Inspired your registration* at the premier event for Sept. 29–Oct. 4, 2019 software Testing Anaheim, CA Professionals STARWEST.TECHWELL.COM

*Valid on packages over $400.

The College of Engineering, Mathematics and Science (EMS) Chair of the Computer Science and Software Engineering (CSSE) Department University of Wisconsin-Platteville is seeking a visionary leader to serve as the Department Chair of Computer Science and Software Engineering. We are pursuing exceptional candidates with proven leadership experience, demonstrated commitment to excellence in undergraduate education, and a strong track record of industry, federal, or state-funded research. The CSSE department is student-focused and is known for its hands-on, experiential approach and highly sought-after graduates. This is a tenured faculty position with a start date of August 1, 2020. The CSSE Department at UW-Platteville is growing and transforming, with several new tenure-track faculty, increasing student enrollments and research activities, and involvement in several campus initiatives. CSSE currently has 13 faculty and staff, serving approximately 400 students in three on-campus majors WWW.COMPUTER.ORG and one collaborative online degree. The Software Engineering program is one /COMPUTINGEDGE of two ABET-accredited SE programs in the state of Wisconsin. For more information about our campus, the complete position description and how to apply, please visit our Career Center at careers.uwplatt.edu

www.computer.org/computingedge 50 Richard E. Merwin Award for Distinguished Service

Call for Award Nominations Deadline: 1 October 2019

The Richard E. Merwin Award is the highest-level volunteer service award of the IEEE Computer Society for outstanding service to the profession at large, including significant service to the IEEE Computer Society or its predecessor organizations.

About Richard Merwin 2018 Richard E. Richard Merwin was a pioneer in digital computer engineering Merwin Award who participated in the development of the ENIAC, MANIAC, Recipient for and STRETCH computers. Despite a busy and productive Distinguished technical career, Merwin found time to be active in professional Service societies, including the IEEE Computer Society, ACM, and AFIPS. His generosity of spirit and genuine helpfulness was an important element in the progress of the computer profession.

Award A bronze medal and $5,000 honorarium are awarded.

Sorel Reisman Presentation California State University, The Richard E. Merwin Award is presented at the Fullerton IEEE Computer Society’s Annual Awards Ceremony.

For sustained Nomination Requirements contributions, This award requires 3 endorsements. leadership, and service to the Nominations are being accepted electronically by IEEE Computer 1 October 2019 to bit.ly/richard-merwin Society, IEEE, and the computing Questions? profession at large. Email [email protected] Call for 2019 Major Awards Nominations

Deadline: 1 October 2019

Help Recognize Computing’s Most Prestigious Individuals IEEE Computer Society awards recognize outstanding achievements and highlight signiﬁ cant contributors in the teaching and R&D computing communities. All members of the profession are invited to nominate individuals they consider most eligible to receive international recognition through an appropriate society award.

Charles Babbage Award Computer Pioneer Award Certifi cate/$1,000 Silver Medal In recognition of signifi cant contributions in the fi eld Pioneering concepts and development of the of parallel computation. computing fi eld. Computer Entrepreneur Award W. Wallace McDowell Award Sterling Silver Goblet Certifi cate/$2,000 Vision and leadership resulting in the growth of some Recent theoretical, design, educational, practical, segment of the computer industry. or other tangible innovative contributions. Edward J. McCluskey Taylor L. Booth Award Technical Achievement Award Bronze Medal/$5,000 Certifi cate/$2,000 Contributions to computer science and Contributions to computer science or computer engineering education. technology. Computer Science & Engineering Harry H. Goode Memorial Award Undergraduate Teaching Award Bronze Medal/$2,000 Plaque/$2,000 Information sciences, including seminal ideas, Recognizes outstanding contributions to algorithms, computing directions, and concepts. undergraduate education. Hans Karlsson Award Harlan D. Mills Award Plaque/$2,000 Plaque/$3,000 Team leadership and achievement through Contributions to the practice of software engineering collaboration in computing standards. through the application of sound theory. Richard E. Merwin Award for Distinguished Service Bronze Medal/$5,000 Nomination Deadline Outstanding volunteer service to the profession at large, including service Submit your nomination by to the IEEE Computer Society. 1 October 2019 to www.computer.org/awards Contact us at [email protected] SHARE AND MANAGE YOUR RESEARCH DATA IEEE DataPort is an accessible online platform that enables researchers to easily share, access, and manage datasets in one trusted location. The platform accepts all types of datasets, up to 2TB, and dataset uploads are currently free of charge.

2TB Cloud Storage Supports Research Open Access Options Per Dataset Reproducibility

Link Dataset to Generates Citations ORCID Manuscript Integration

Host Data Competitions DOI Provided Broad Range of Data Topics

UPLOAD DATASETS AT IEEE-DATAPORT.ORG Conference Calendar Questions? Contact [email protected]

EEE Computer Society conferences are valuable forums for learning on broad and dynamically I shifting topics from within the computing profession. With over 200 conferences featuring leading experts and thought leaders, we have an event that is right for you.

Find a region: Africa ■ Australia ◆ North America ◗ Asia ▲ Europe ● South America ★

OCTOBER 28 October 1 October • EDOC (IEEE 23rd Int’l Enterprise Distributed • MCSoC (IEEE 13th Int’l Symposium on Embed- Object Computing Conf.) ● ded Multicore/Many-core Systems-on-Chip) ▲ • ISSRE (IEEE 30th Int’l Symposium on Soft- 2 October ware Reliability Eng.) ● • DFT (IEEE Int’l Symposium on Defect and Fault Tolerance in VLSI and Nanotechnology Systems) ● NOVEMBER 12 October 4 November • MICRO (52nd Annual IEEE/ACM Int’l Sympo- • ICTAI (IEEE 31st Int’l Conf. on Tools with Arti- sium on Microarchitecture) ◗ fi cial Intelligence) ◗ 14 October 7 November • ISMAR (IEEE Int’l Symposium on Mixed and • SEC (IEEE/ACM Symposium on Edge Com- Augmented Reality) ▲ puting) ◗ • LCN (IEEE 44th Conf. on Local Computer Net- 8 November works) ● • ICDM (IEEE Int’l Conf. on Data Mining) ▲ • VL/HCC (IEEE Symposium on Visual Lan- 9 November guages and Human-Centric Computing) ◗ • FOCS (IEEE 60th Annual Symposium on Foun- 15 October dations of Computer Science) ◗ • AIPR (IEEE Applied Imagery Pattern Recogni- 11 November tion Workshop) ◗ • ASE (34th IEEE/ACM Int’l Conf. on Automated 16 October Software Eng.) ◗ • FIE (IEEE Frontiers in Education Conf.) ◗ 17 November 20 October • ICCD (IEEE 37th Int’l Conf. on Computer • VIS (IEEE Visualization Conf.) ◗ Design) ▲ 27 October • SC19 (SC19: Int’l Conf. for High Performance • ICCV (IEEE/CVF Int’l Conf. on Computer Computing, Networking, Storage and Analy- Vision) ▲ sis) ◗

72 September 2019 Published by the IEEE Computer Society 2469-7087/19 © 2019 IEEE 18 November Computer Vision) ◗ • BIBM (IEEE Int’l Conf. on Bioinformatics and 9 March Biomedicine) ◗ • DATE (Design, Automation & Test in Europe Conf. & Exhibition) ● DECEMBER • IRC (4th IEEE Int’l Conf. on Robotic Comput- 3 December ing) ▲ • RTSS (IEEE Real-Time Systems Symposium) 16 March ▲ • ICSA (IEEE Int’l Conf. on Software Architec- 4 December ture) ★ • IREHI (IEEE Int’l Rural and Elderly Health Infor- 22 March matics Conf.) ■ • VR (IEEE Conf. on Virtual Reality and 3D User 9 December Interfaces) ◗ • AIVR (IEEE Int’l Conf. on Artificial Intelligence 23 March and Virtual Reality) ◗ • ICST (13th IEEE Conf. on Software Testing, • Big Data (IEEE Int’l Conf. on Big Data) ◗ Validation and Verification) ● • ISM (IEEE Int’l Symposium on Multimedia) ◗ • PerCom (IEEE Int’l Conf. on Pervasive Com- 10 December puting and Communications) ◗ • ISSPIT (IEEE Int’l Symposium on Signal Pro- cessing and Information Technology) ▲ April 16 December 5 April • CDKE (IEEE Int’l Conf. on Conversational Data • ISPASS (Int’l Symposium on Performance & Knowledge Eng.) ◗ Analysis of Systems and Software) ◗ 14 April 2020 • PacificVis (IEEE Pacific Visualization Sympo- sium) (Location TBD) January 20 April 13 January • ICDE (IEEE 36th Int’l Conf. on Data Eng.) ◗ • ICCPS (Int’l Conf. on Cyber-Physical Systems) ● May February 3 May 3 February • FCCM (IEEE 28th Annual Int’l Symposium • ICSC (IEEE 14th Int’l Conf. on Semantic Com- on Field-Programmable Custom Computing puting) ◗ Machines) ◗ 18 February • SANER (IEEE 27th Int’l Conf. on Software Anal- ysis, Evolution and Reengineering) ◗ 19 February • BigComp (IEEE Int’l Conf. on Big Data and Smart Computing) ▲ 22 February Learn more about • CGO (IEEE/ACM Int’l Symposium on Code Generation and Optimization) ◗ IEEE Computer Society Conferences March 2 March www.computer.org/conferences • WACV (IEEE Winter Conf. on Applications of

ce9con(all).indd 73 8/1/19 11:32 AM ipdps.org

IPDPS 2020 CALL FOR PAPERS

The five-day IPDPS program includes three days of contributed papers, GENERAL CO-CHAIRS invited speakers, industry participation, and student programs, framed by Anu Bourgeois (Georgia State University, USA) two days of workshops with peer reviewed papers that complement and Ramachandran Vaidyanathan (Louisiana State University, broaden the main program. For full details, see www.ipdps.org. USA)

Authors for the main conference are invited to submit manuscripts PROGRAM CHAIR that present original unpublished research in all areas of parallel and Yuanyuan Yang (NSF and Stony Brook University, USA) distributed processing, including the development of experimental or commercial systems. Work focusing on emerging technologies and PROGRAM AREA CHAIRS AND VICE CHAIRS interdisciplinary work covering multiple IPDPS areas are especially • Algorithms: welcome. Topics of interest include: Xiaotie Deng (Peking University, China) and • Parallel and distributed computing theory and algorithms Songtao Guo (Chongqing University, China) (Algorithms) • Architecture: Ahmed Louri (George Washington University, USA) and • Experiments and practice in parallel and distributed Avinash Karanth (Ohio University, USA) computing (Experiments) • Experiments: • Programming models, compilers and runtimes for parallel Xin Yuan (Florida State University, USA) and applications and systems (Programming Models & Scott Pakin (Los Alamos National Laboratory, USA) Compilers) • System Software: Alan Sussman • System software and middleware for parallel and (NSF and University of Maryland, College Park, USA) and distributed systems (System Software) Zhiling Lan (Illinois Institute of Technology, USA) • Architecture • Programming Models & Compilers: • Multidisciplinary Rudolf Eigenmann (University of Delaware, USA) Zhiyuan Li (Purdue University, USA) • Abstracts due October 7, 2019 • Multidisciplinary: Manish Parashar (NSF and Rutgers University, USA) • Submissions due October 14, 2019 Ivona Brandic´ (Vienna University of Technology, Austria) • Preliminary decisions December 9, 2019 New Orleans is one of the most eccentric and lively cities in the • Final submissions due January 6, 2020 world. Whatever your interests are, New Orleans has you covered. From its diverse culture, distinctive cuisine, rich history, colorful • Final notification January 20, 2020 celebrations, live music, vibrant nightlife, and world-class restaurants, there is something for everyone. It is home to a number of engaging museums, including the World War II Museum, the New Orleans Museum of Art, the Historic Voodoo Museum, Mardi Gras World, and the Pharmacy Museum. A visit wouldn’t be complete without a swamp tour, Mississippi river cruise and a stop at Café du Monde for beignets. Join IPDPS at the Hilton New Orleans Riverside in 2020 to find out what makes New Orleans so unique and special.

IPDPS2020.SeptAd.indd 1 7/29/19 10:41 PM