Blockchain > Internet of Things > Internet

> Blockchain > Internet of Things > Internet Trends and Applications > Virtual and Augmented Reality

JANUARY 2019 www.computer.org

IEEE COMPUTER SOCIETY computer.org • +1 714 821 8380

STAFF

Editor Publications Portfolio Managers Cathy Martin Carrie Clark, Kimberly Sperka

Publications Operations Project Specialist Publisher Christine Anthony Robin Baldwin Publications Marketing Project Specialist Meghan O’Dell Senior Advertising Coordinator Debbie Sims Production & Design Carmen Flores-Garvey

Circulation: ComputingEdge (ISSN 2469-7087) is published monthly by the IEEE Computer Society. IEEE Headquarters, Three Park Avenue, 17th Floor, New York, NY 10016-5997; IEEE Computer Society Publications Office, 10662 Los Vaqueros Circle, Los Alamitos, CA 90720; voice +1 714 821 8380; fax +1 714 821 4010; IEEE Computer Society Headquarters, 2001 L Street NW, Suite 700, Washington, DC 20036. Postmaster: Send address changes to ComputingEdge-IEEE Membership Processing Dept., 445 Hoes Lane, Piscataway, NJ 08855. Periodicals Postage Paid at New York, New York, and at additional mailing offices. Printed in USA. Editorial: Unless otherwise stated, bylined articles, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in ComputingEdge does not necessarily constitute endorsement by the IEEE or the Computer Society. All submissions are subject to editing for style, clarity, and space. Reuse Rights and Reprint Permissions: Educational or personal use of this material is permitted without fee, provided such use: 1) is not made for profit; 2) includes this notice and a full citation to the original work on the first page of the copy; and 3) does not imply IEEE endorsement of any third-party products or services. Authors and their companies are permitted to post the accepted version of IEEE-copyrighted material on their own Web servers without permission, provided that the IEEE copyright notice and a full citation to the original work appear on the first screen of the posted copy. An accepted manuscript is a version which has been revised by the author to incorporate review suggestions, but not the published version with copy-editing, proofreading, and formatting added by IEEE. For more information, please go to: http://www.ieee.org/publications_standards/publications/rights/paperversionpolicy.html. Permission to reprint/republish this material for commercial, advertising, or promotional purposes or for creating new collective works for resale or redistribution must be obtained from IEEE by writing to the IEEE Intellectual Property Rights Office, 445 Hoes Lane, Piscataway, NJ 08854-4141 or [email protected]. Copyright © 2019 IEEE. All rights reserved. Abstracting and Library Use: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy for private use of patrons, provided the per- copy fee indicated in the code at the bottom of the first page is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923. Unsubscribe: If you no longer wish to receive this ComputingEdge mailing, please email IEEE Computer Society Customer Service at [email protected] and type “unsubscribe ComputingEdge” in your subject line. IEEE prohibits discrimination, harassment, and bullying. For more information, visit www.ieee.org/web/aboutus/whatis/policies/p9-26.html.

IEEE Computer Society Magazine Editors in Chief

Computer IEEE Security & Privacy Computing in Science David Alan Grier (Interim), David Nicol, University of Illinois & Engineering George Washington University at Urbana-Champaign Jim X. Chen, George Mason University IEEE Micro IEEE Software IEEE Intelligent Systems Lizy Kurian John, University of Ipek Ozkaya, Software V.S. Subrahmanian, Dartmouth Texas, Austin Engineering Institute College IEEE MultiMedia IEEE Internet Computing IEEE Computer Graphics Shu-Ching Chen, Florida George Pallis, University of and Applications International University Cyprus Torsten Möller, University of Vienna IEEE Annals of the History IT Professional of Computing Irena Bojanova, NIST IEEE Pervasive Computing Marc Langheinrich, University of Gerardo Con Diaz, University of Lugano California, Davis

www.computer.org/computingedge 1 JANUARY 2019 • VOLUME 5, NUMBER 1

THEME HERE 17 24 29 Adaptable Accurate The Last Mile Blockchain-Based Indoor for IoT Privacy Systems Location for the IoT Blockchain 10 Blockchain: The Emperor’s New PKI? HILARIE ORMAN 17 Adaptable Blockchain-Based Systems QINGHUA LU AND XIWEI XU Internet of Things 24 Accurate Indoor Location for the IoT ROY WANT, WEI WANG, AND STAN CHESNUTT 29 The Last Mile for IoT Privacy RICHARD CHOW Internet Trends and Applications 33 5G in E-Commerce Activities NIR KSHETRI 38 A New Era for Web AR with Mobile Edge Computing XIUQUAN QIAO, PEI REN, SCHAHRAM DUSTDAR, AND JUNLIANG CHEN Virtual and Augmented Reality 48 360-Degree Virtual-Reality Cameras for the Masses JACK TAN, GENE CHEUNG, AND RUI MA 56 Virtual Travel CHARLES DAY

Departments 4 Magazine Roundup 8 Editor’s Note: Emerging Blockchain Applications 58 Career Opportunities 38 A New Era for Web AR with Mobile Edge Computing Subscribe to ComputingEdge for free at www.computer.org/computingedge. CS FOCUS

Magazine Roundup

Computer, the authors provide insight into artifi cial emotional intelligence (AEI) and present three major areas of emotion— recognition, generation, and augmentation—needed to reach a new emotionally intelligent epoch of AI.

Computing in Science & Engineering

he IEEE Computer Computer The Heat Equation: High- Society’s lineup of 12 Performance Scientifi c T peer-reviewed technical The Age of Artifi cial Computing Case Study magazines covers cutting-edge Emotional Intelligence In recent years, high-performance topics ranging from software Science fi ction often portrays computing and powerful super- design and computer graphics to future AI technology as having computers have become staples Internet computing and security, sophisticated emotional intelli- in many areas of academia and from scientifi c applications gence skills to the degree industry. The author of this arti- and machine intelligence to where technology can develop cle from the September/October visualization and microchip compassion. But where are 2018 issue of Computing in Sci- design. Here are highlights from we today? In this article from ence & Engineering introduces recent issues. the September 2018 issue of the concept of shared memory

4 January 2019 Published by the IEEE Computer Society 2469-7087/19/$33.00 © 2019 IEEE programming in the context of propose real-time algorithms that uses in the areas of psychology, solving the heat equation, which detect multiple body poses and health informatics, pharmacovigi- will allow the exploration of several recognize punching and touching lance, and aff ective computing. fi nite diff erence and parallelization gestures from top-view depth schemes. images. A pointing and touching IEEE Internet Computing test shows that the proposed IEEE Annals of the History algorithm outperforms other Real-Time Identity-Deception of Computing algorithms. In addition, the authors Detection Techniques for created a game to make the best Social Media: Optimizations How Atex Helped an Industry use of the proposed system. and Challenges Change the World Identity-deception detection Douglas Drane was one of three IEEE Intelligent Systems methods have been proposed for co-founders of Atex, an early social-media platforms with high dominant supplier of digital Detecting Personal Intake of eff ectiveness, but their effi ciency systems for newspaper and Medicine from Twitter can vary. Previous literature has magazine publication. In the July– Mining social media messages not examined the potential of September 2018 issue of IEEE such as tweets, blogs, and Face- these methods to work as real-time Annals of the History of Computing, book posts for health- and drug- monitoring systems. Such imple- Drane recounts his memories of related information has received mentations further highlight the Atex’s founding and operation signifi cant interest in pharmaco- challenges of applying computa- until it was bought by Kodak. vigilance research. Social media tionally intensive methods in online websites have been used for moni- environments that involve large IEEE Computer Graphics toring drug abuse, searching for datasets and high speeds of data. and Applications adverse reactions to drug usage, This article, which appears in the and analyzing expression of sen- September/October 2018 issue of ThunderPunch: A Bare- timents related to drugs. Most of IEEE Internet Computing, attempts Hand, Gesture-Based, Large these studies are based on aggre- to classify detection methods Interactive Display Interface gated results from a large popula- based on the approaches and iden- with Upper-Body-Part tion rather than specifi c sets of tifi es factors that, in real-time sys- Detection in a Top View individuals. To conduct studies at tems, will impact the eff ectiveness The authors of this article from an individual level or in specifi c and effi ciency of these methods. the September/October 2018 groups of people, identifying posts Optimizations are proposed that issue of IEEE Computer Graphics mentioning intake of medicine can limit the computational over- and Applications present a new by the user is necessary. Toward head. Further challenges involving bare-hand gesture interface for this objective, the authors of this real-time identity-deception detec- large-screen interaction in which article from the July/August 2018 tion are discussed. multiple users can participate issue of IEEE Intelligent Systems simultaneously and interact with developed a classifi er for identify- IEEE Micro virtual content directly. Unlike ing mentions of personal intake of the conventional method, which medicine in tweets. They trained a Newton: Gravitating Towards involves positioning the camera stacked ensemble of shallow con- the Physical Limits of in front, the cameras are mounted volutional neural network (CNN) Crossbar Acceleration on the ceiling so that they avoid models on an annotated dataset Recent works take advantage of covering the large screen. To and used random search for tuning highly parallel analog in-situ com- achieve bare-hand interaction in the hyper-parameters of the CNN putation in memristor crossbars to this hardware structure, the authors models. The classifi er has direct accelerate the many vector-matrix www.computer.org/computingedge 5 MAGAZINE ROUNDUP

in power, 51-percent improvement and potentially constitute a huge in energy-effi ciency, and 2.1× mobile sensor network. These higher throughput/area, relative to advances motivate researchers the state-of-the-art In-Situ Analog to propose mobile crowdsensing Arithmetic in Crossbars (ISAAC) (MCS) systems that can provide accelerator. large-scale and complex social or geographical sensing applications. IEEE MultiMedia Read more in the July–September 2018 issue of IEEE Pervasive Digital Twins: The Computing. Convergence of Multimedia Technologies IEEE Security & Privacy WWW.COMPUTER.ORG Originally developed to improve manufacturing processes, digital Privacy and Civilian Drone /COMPUTINGEDGE twins are being redefi ned as Use: The Need for Further digital replications of living and Regulation nonliving entities that enable Current US regulation is not data to be seamlessly transmitted equipped to provide explicit between the physical and virtual privacy protection for drone use multiplication operations in deep worlds. Digital twins facilitate the in an era of sophisticated audio/ neural networks (DNNs). However, means to monitor, understand, video and social media. In 2016, these in-situ accelerators have two and optimize the functions of all the National Telecommunications signifi cant shortcomings: The physical entities and for humans and Information Administration ADCs account for a large fraction to provide continuous feedback to recognized this defi cit by releasing of chip power and area, and these improve quality of life and well- a set of best practices, which we accelerators adopt a homogeneous being. Read more in the April–June examine in light of the current design in which every resource 2018 issue of IEEE MultiMedia. privacy concerns with drone is provisioned for the worst case. use in the US. Read more in the By addressing both problems, a IEEE Pervasive Computing September/October 2018 issue of new architecture, called Newton, IEEE Security & Privacy. moves closer to achieving opti- Privacy-Preserving Incentive mal energy per neuron for cross- Mechanisms for Mobile IEEE Software bar accelerators. In this article Crowdsensing from the September/October 2018 The smartphone market has Software Engineering issue of IEEE Micro, the authors proliferated rapidly in recent years. Research and Industry: introduce new techniques that Ubiquitous smartphones not only A Symbiotic Relationship to apply at diff erent levels of the tile possess powerful computational Foster Impact hierarchy, some leveraging het- capabilities but also contain Software engineering is not only erogeneity and others relying on various built-in sensors. Along an increasingly challenging divide-and-conquer numeric algo- with providing easy access to endeavor that goes beyond the rithms to reduce computations communication networks such intellectual capabilities of any and ADC pressure. For many con- as Wi-Fi and 3G/4G networks, single individual engineer but also volutional-neural-network (CNN) smartphones have become an an intensely human one. Tools datafl ows and structures, Newton important information interface and methods to develop software achieves a 77-percent decrease between users and environments are employed by engineers of

6 ComputingEdge January 2019 varied backgrounds within a in achieving its potential, a financial technology innovation in large variety of organizations and description of the root causes of the payments market. This article, application domains. As a result, such challenges, and a proposal which appears in the March/April the variation in challenges and for the field to move forward 2018 issue of IT Professional, practices in system requirements, and become more impactful provides a holistic overview of architecture, and quality assurance through collaborative research the technological innovations is staggering. Human, domain, and innovation between public and challenges in the evolving and organizational factors define research and industry. payments landscape. the context within which software engineering methodologies and IT Professional technologies are to be applied. This article, which is part of the The Evolving Payments September/October 2018 issue Landscape: Technological of IEEE Software on software Innovation in Payment engineering’s 50th anniversary, Systems provides an assessment of the Established financial institutions FOLLOW US current challenges faced by and their technology partners @securityprivacy software engineering research are contributing significantly to

ADVERTISER INFORMATION

Advertising Personnel Southwest, California: Mike Hughes Debbie Sims: Advertising Coordinator Email: [email protected] Email: [email protected] Phone: +1 805 529 6790 Phone: +1 714 816 2138 | Fax: +1 714 821 4010

Advertising Sales Representative (Classifieds & Jobs Board) Advertising Sales Representatives (display) Heather Buonadies Central, Northwest, Southeast, Far East: Email: [email protected] Eric Kincaid Phone: +1 201 887 1703 Email: [email protected] Phone: +1 214 673 3742 Fax: +1 888 886 8599 Advertising Sales Representative (Jobs Board)

Northeast, Midwest, Europe, Middle East: Marie Thompson David Schissler Email: [email protected] Email: [email protected] Phone: 714-813-5094 Phone: +1 508 394 4026 Fax: +1 508 394 1707

www.computer.org/computingedge 7 EDITOR’S NOTE

Emerging Blockchain Applications

n the public consciousness, blockchain is on IoT devices is opening doors to possibilities almost synonymous with cryptocurrencies like in-building navigation and automated smart- I like Bitcoin. But many other exciting uses home operation. “The Last Mile for IoT Privacy,” for this emerging technology are in the works. from IEEE Security & Privacy, presents a conceptual Blockchain utilizes cryptography to log records framework for notifying people when IoT devices securely and reliably, leading to potential applica- are collecting their data. tions in important areas such as banking, health- Mobile Internet trends are pushing boundaries care, and voting. thanks to emerging technologies such as 5G and The January 2019 issue of ComputingEdge dis- edge computing. The author of IT Professional’s cusses two new blockchain applications. IEEE “5G in E-Commerce Activities” details the impact Internet Computing’s “Blockchain: The Emperor’s of 5G networks, platforms, and devices on the New PKI?” proposes the idea that blockchain online shopping experience. The authors of IEEE could become the foundation of our future digi- Internet Computing’s “A New Era for Web AR with tal identities. Meanwhile, IEEE Software’s “Adapt- Mobile Edge Computing” discuss how mobile able Blockchain-Based Systems” presents a novel edge computing on 5G networks can reduce decentralized system for tamper-proof product latency, thereby improving web AR performance. tracking in complex supply chains. Finally, VR is seeing novel implementations, Other technologies and trends—such as the such as the 360-degree camera presented in IEEE Internet of Things (IoT), mobile Internet use, and MultiMedia’s “360-Degree Virtual-Reality Cameras virtual and augmented reality (VR and AR)—are for the Masses.” VR is also playing an increas- also expanding into new territories. In Comput- ingly important role in the tourism industry, as er’s “Accurate Indoor Location for the IoT,” the shown in Computing in Science & Engineering’s “Vir- authors explain that improved location accuracy tual Travel.”

8 January 2019 Published by the IEEE Computer Society 2469-7087/19/$33.00 © 2019 IEEE PURPOSE: The IEEE Computer Society is the world’s largest EXECUTIVE COMMITTEE association of computing professionals and is the leading President: �i��i �asa�a�a provider of technical information in the field. President-Elect: �e�ilia Me��a; Past President: Jea��u� �au�i��; First VP, MEMBERSHIP: Members receive the monthly magazine ��: ��e��y �� y��; Second VP, �� e��is J� ��ailey; VP, Computer, discounts, and opportunities to serve (all activities Member & Geographic Activities: �� ; VP, Professional & are led by volunteer members). Membership is open to all IEEE Educational Activities: �� ; VP, Standards Activities: J�� R�s�a�l; members, affiliate society members, and others interested in the VP, Technical & Conference Activities: �ausi Mulle�; 201�� IEEE computer field. Division V ��: �� ; 201��201� IEEE Division V�� : COMPUTER SOCIETY WEBSITE: www.computer.org �e�a� Mil��i�i�� 201� IEEE Division V�� -Elect: �li�a�e�� u�� OMBUDSMAN: Direct unresolved complaints to ombudsman@ computer.org. BOARD OF GOVERNORS CHAPTERS: Regular and student chapters worldwide provide the Term Expiring 201�: �� eMa�le� ��e� �ie��i�� e� ��u�lis� �la�i�i� �e�� opportunity to interact with colleagues, hear technical experts, ��u�e M� M�Milli�� u�i� ��iya�a� ��e�a�� a�e�� and serve the local professional community. Term Expiring 201�: �au�a�� a��i� �eila �e�l��ia�i� �a�i� �� e�� Jill �� AVAILABLE INFORMATION: To check membership status, report ��s�i�� illia� �� u�i �elal� ��i Me��els�� an address change, or obtain more information on any of the Term Expiring 20��: ��y ��e�� J�� J��s�� y��e� �u�� a�i� following, email Customer Service at [email protected] or call ��e�� i�i��i�s �e��a��s� ��es� ��ull� �aya�� a�a�a +1 714 821 8380 (international) or our toll-free number, +1 800 272 6657 (US): EXECUTIVE STAFF • Membership applications Executive Director: Melissa Russell • Publications catalog Director, Governance & Associate Executive Director: Anne Marie Kelly • Draft standards and order forms Director, Finance & Accounting: Sunny Hwang • Technical committee list Director, Information Technology & Services: Sumit Kacker • Technical committee application Director, Membership Development: Eric Berkowitz • Chapter start-up procedures • Student scholarship information • Volunteer leaders/staff directory COMPUTER SOCIETY OFFICES • IEEE senior member grade application (requires 10 years practice and significant performance in five of those 10) Washington, D.C.: 2001 L St., Ste. 700, Washington, D.C. 20036-4928 Phone: +1 202 371 0101 • Fax: +1 202 728 9614 PUBLICATIONS AND ACTIVITIES Email: [email protected] Computer: The flagship publication of the IEEE Computer Los Alamitos: 10662 Los Vaqueros Circle, Los Alamitos, CA 90720 Phone: Society, Computer, publishes peer-reviewed technical content that +1 714 821 8380 covers all aspects of computer science, computer engineering, Email: [email protected] technology, and applications. MEMBERSHIP & PUBLICATION ORDERS Periodicals: The society publishes 13 magazines, 19 transactions, Phone: +1 800 272 6657 • Fax: +1 714 821 4641 • Email: [email protected] and one letters. Refer to membership application or request Asia/Pacific: Watanabe Building, 1-4-2 Minami-Aoyama, Minato-ku, Tokyo information as noted above. 107-0062, Japan Conference Proceedings & Books: Conference Publishing Phone: +81 3 3408 3118 • Fax: +81 3 3408 3553 Services publishes more than 275 titles every year. Email: [email protected] Standards Working Groups: More than 150 groups produce IEEE standards used throughout the world. IEEE BOARD OF DIRECTORS Technical Committees: TCs provide professional interaction in President & CEO: Ja�es Je��e�ies more than 30 technical areas and directly influence computer President-Elect: J�se M�� M�u�a engineering conferences and publications. Past President: �a�e� �a��les�� Conferences/Education: The society holds about 200 conferences Secretary: �illia� �� als� each year and sponsors many educational activities, including Treasurer: J�se�� illie computing science accreditation. Director & President, IEEE-USA: �a��a ��a��y� R��i�s�� Certifications: The society offers two software developer Director & President, Standards Association: ��es� �� i�� credentials. For more information, visit www.computer.org/ Director & VP, Educational Activities: �i��l� M� �i�s�e� certification. Director & VP, Membership and Geographic Activities: Ma��i� �as�iaa�s Director & VP, Publication Services and Products: �a�i� M� �l��a�aly Director & VP, Technical Activities: �usa� ��a��y� �a�� Director & Delegate Division V: J�� al� Director & Delegate Division VIII: �e�a� .JMPKJʊJʉ

revised 03 July 201�

DEPARTMENT: Practical Security

Blockchain: the Emperor’s New PKI?

Hilarie Orman I would like to jump on the blockchain bandwagon. I Purple Streak would like to be able to say that blockchain is the solution to the longstanding problem of secure identity on the Internet. I would like to say that everyone in the world will soon have a digital identity. Put yourself on the blockchain and never again ask yourself, “Who am I?” – you are your blockchain address.

“All the world’s a blockchain, And all the men and women merely entries in it.”

Certainly, it is important to solve the online identity problem. Identity can be destiny. Today, the mere fact of a person’s physical existence is not enough to guarantee some rights. Citizenship, refugee status, right to travel, government benefits, etc., are tied to a provable identity of the right type. Some have gone so far as to say that identity is a human right.1 Is it feasible to solve this problem through digital technology? And if so, is blockchain part of the solution? And do we get privacy with that? The digital divide leaves first-world citizens with the problem of having too many online identities to manage, while those on the other side of the chasm, some two billion people, perhaps, not only have no digital identity, but further have no documented identity at all. Everyone in the world needs at least one persistent and secure identity that is easily manageable by the individual. Public key technology has always held the elusive promise of a universal digital identity system that makes paper obsolete, but attempts to do this have never measured up. Fraught with complex management, lookup, and trust issues, public key has been largely relegated to a way to protect Internet communication channels. Although some countries use digital identification via public key and electronic ID cards, there is no widespread movement towards this kind of solution. Blockchain technology seems like an odd savior, especially because its main use to date has been for digital cash with anonymity features. A verified identity is nearly the opposite of anonymity, so what does blockchain have to do with it? And how can we have verified identities that pre- serve privacy? The technology for doing this is in its infancy, and the management problems are largely unexplored, but when has that stopped technology ebullience? Let’s look at what identity means and how life might be if lived in the blockchain bubble.

IEEE Internet Computing Published by the IEEE Computer Society March/April10 2018 January 2019 Published23 by the IEEE Computer Society 1089-7801/18/$33.002469-7087/19/$33.00 USD ©2018© 2019 IEEE IEEE IEEE INTERNET COMPUTING

WHITHER COMEST (OR GOEST) IDENTITY? DEPARTMENT: Practical Security Identity is a psychological, philosophical, and legal concept that is carried over into the digital world in a variety of ways. For an interesting discussion of desirable principles in the digital realm, see the work of Kim Cameron.2 The simple story is that the various authorities that control the resources that we need to participate in life today need to know who we are, which means they want enough information to dis- ambiguate all people from one another. Somehow, there must be a short index from a person into a table of data about many people, potentially all people on earth. Some authorities want to at- tach attributes to identities, other authorities need to examine those attributes. At the center of it Blockchain: the Emperor’s all is a unique human being. In order to avoid some kinds of dystopian nightmares about authori- tarians dictatorships, it seems important to let the person, the identity subject, control how and when the attributes are disclosed. For example, people need to communication their immuniza- New PKI? tion status to a health organization, residence to local housing authorities, financial status to lending organizations, etc. A recent trend is to use cryptographic zero-knowledge proofs for this purpose. But this is not new. What is new is using blockchains for bootstrapping the trust process and for the persistence and availability of the information. We all know that our identity is more than our name and appearance, and that name recognition Hilarie Orman I would like to jump on the blockchain bandwagon. I and personal relationships are not enough to maintain one’s status in an impersonal digital world. Purple Streak would like to be able to say that blockchain is the People need to document their place of origin and residence, their health status, and their economic ties if they are going to participate in the global economy and make use of the rights to solution to the longstanding problem of secure which they are entitled. identity on the Internet. I would like to say that everyone in the world will soon have a Today, identity is established in a variety of ways for people and for governments and businesses digital identity. Put yourself on the blockchain and never again ask yourself, “Who am and almost all organizations. On social media, identity means a username, a communication history, and interactions with some kind of online community. For email, it is simply an email ad- I?” – you are your blockchain address. dress. For a website it is a URL and perhaps a certificate. For most website interactions, it is a username and a password, perhaps augmented by a cell phone number. For newer mobile devices, various kinds of biometrics identify the device owner. The essential government issued “All the world’s a blockchain, identity documents are still based on paper: driver’s licenses, passports, birth certificates, etc. (I And all the men and women merely entries in it.” told my doctor’s office to address me by my date of birth, as it seems to be essential for them to verify it multiple times in any conversation.) Certainly, it is important to solve the online identity problem. Identity can be destiny. Today, the We may be on the verge of a major change, one in which public keys are of central importance, mere fact of a person’s physical existence is not enough to guarantee some rights. Citizenship, users have total control over disclosure of their identity attributes, and blockchains are the root refugee status, right to travel, government benefits, etc., are tied to a provable identity of the points of trust for obtaining credentials. Or, we may be on the verge of discovering that block- right type. Some have gone so far as to say that identity is a human right.1 Is it feasible to solve chains recapitulate the well-known problems of public key infrastructure. this problem through digital technology? And if so, is blockchain part of the solution? And do we get privacy with that? The digital divide leaves first-world citizens with the problem of having too many online identi- WHAT IS A BLOCKCHAIN? ties to manage, while those on the other side of the chasm, some two billion people, perhaps, not only have no digital identity, but further have no documented identity at all. Everyone in the Blockchains offer the promise of a trustworthy way to record shared data. It is a publicly verifia- world needs at least one persistent and secure identity that is easily manageable by the individ- ble ledger that maintains the integrity of the individual entries. ual. The basic idea of a blockchain is quite simple: it is a shared, replicated log file (sometimes called Public key technology has always held the elusive promise of a universal digital identity system a ledger). The entries are sequential and time-stamped. A one-way function produces a short bit- that makes paper obsolete, but attempts to do this have never measured up. Fraught with com- string (for example, 512 bits) and depends on every item and its placement in the log. The func- plex management, lookup, and trust issues, public key has been largely relegated to a way to pro- tion has mathematical properties that assure that it would be astronomically difficult to produce a tect Internet communication channels. Although some countries use digital identification via different log with the same output. The output of the function is an abbreviation for the log itself. public key and electronic ID cards, there is no widespread movement towards this kind of solu- To add new entries, the function uses its current value and the contents of every new entry to tion. compute a new output. The log maintainer publishes the log and the output value so that independent parties can verify the correspondence. Blockchain technology seems like an odd savior, especially because its main use to date has been for digital cash with anonymity features. A verified identity is nearly the opposite of anonymity, In itself, a blockchain offers no particular security. To get to an interesting trust model, two addi- so what does blockchain have to do with it? And how can we have verified identities that pre- tional mechanisms are required. The first is to have a verifiable definition of correctness for each serve privacy? The technology for doing this is in its infancy, and the management problems are log file entry. This means that there is a correctness condition that is orthogonal to the correct- largely unexplored, but when has that stopped technology ebullience? Let’s look at what identity ness of the mere sequence of the entries; usually the log file entries are signed by public keys and means and how life might be if lived in the blockchain bubble. have a standardized format. Secondly, the Important Idea that makes blockchains useful is to dis-

IEEE Internet Computing Published by the IEEE Computer Society March/April 2018 23 1089-7801/18/$33.00 USD ©2018 IEEE March/Aprilwww.computer.org/computingedge 2018 24 www.computer.org/internet11 PRACTICAL SECURITY

tribute the maintenance of the blockchain to many (possibly distrusting) parties, thereby achieving “distributed consensus” in real time. As long as the majority of the parties have a mutual interest in publishing a consistent view of the blockchain, it is a reliable record of transactions. BitCoin has the interesting property that any party doing a sufficient amount of work can extend the blockchain. Anyone can become part of the peer-to-peer network and get the current state of the blockchain and list of transactions that are queued for inclusion. After checking that the transactions satisfy the correctness conditions, and then computing the validation function, a party can advertise the transactions and validation output to other nodes. All the other nodes will check the computation and begin publishing the blockchain with the new block of transactions. The salient property of the BitCoin validation function is that it is very difficult to compute but very easy to check. To incentivize validators (miners), the system automatically rewards them when their computation is accepted.

WHAT IS A BLOCKCHAIN IDENTITY? There are proposals to create blockchain systems and communities that use blockchains in which the entries are somehow related to identification for an individual person or unique entity, such as a DNS name. In this model, the public key for a person or entity is represented in a blockchain entry, where it could be accessed by a relying party. In the simplest case, an entry would be signed by an identity provider of some kind, using its own public key, to assert: “The public key represented in this data item belongs to a person known as John Q. Public.” This is similar to existing public key systems, such as X.509v3 and PGP. Blockchains offer a couple of improvements, though. One is that the blockchain replication by a diverse set of entities ensure accessibility and reliability. Another is the immutability of the blockchain and the timestamps; there can be little argument that an entry was made, and that can be shown for as long as the blockchain exists. In order to know if blockchains solve other fundamental identity problems, we need to look more deeply into why identity has become so important. Even though we still use passwords and special purpose apps, digital identities based on public keys are the only hope of getting out of the morass of relying on companies and governments to keep their customer or citizen data private. Their inability to do so is demonstrated on a regular basis. There are two kinds of identity being discussed today. One is for most of the people in countries with good governmental record keeping, reliable power for computing devices, and nearly universal Internet service; the other is for people who have been cut off from government recognition, isolated by virtue of abuse or persecution, who are poor, or who do not have physical safety for possession. This can happen either because the their government is too weak to provide identity services, because they are ignored by their government, because they are victims of human trafficking, because they are refugees, etc. The digitally dispossessed cannot rely on cellphones, cards with chips, or anything physical to provide their identity. What then identifies them? One way would be to record biometric data such as iris scans, facial features, fingerprints, or DNA. A cryptographic hash of the data could be an anchor point for attaching attributes such as country of residence, immunization status, etc. The blockchain would hold the cryptographic hash, and various authorities could certify possession of attributes. The feasibility of this is uncertain because biometric attributes are “fuzzy,” while a cryptographic hash is anything but. Still, one could imagine a solution based on varying degrees of accuracy and multiple measurements to create attribute collections that could map to a unique match. In an ideal world, none of the identity attributes could be released without permission from the person being identified. But if that person is digitally dispossessed, then she has no way to present a public key to unlock the record; instead, the biometric measurements alone are enough to unlock the information. This may well be the future identification situation not only for the digitally dispossessed, but perhaps for most of the world’s population, if their governments insist on it.

March/April12 2018 ComputingEdge 25 www.computer.org/internetJanuary 2019 PRACTICAL SECURITY IEEE INTERNET COMPUTING

tribute the maintenance of the blockchain to many (possibly distrusting) parties, thereby achiev- There are many in the tech world today who are banking on the assumption that blockchain tech- ing “distributed consensus” in real time. As long as the majority of the parties have a mutual in- nology will be the way register identity and to redesign the entire concept of digital identity. The terest in publishing a consistent view of the blockchain, it is a reliable record of transactions. identity owner could get the unfettered freedom to manage access to his personal information, to BitCoin has the interesting property that any party doing a sufficient amount of work can extend his credentials, and to his organizational records. A whole economy of identity might arise, a lib- the blockchain. Anyone can become part of the peer-to-peer network and get the current state of ertarian utopia where the person and his keys are self-determining entities. the blockchain and list of transactions that are queued for inclusion. After checking that the transactions satisfy the correctness conditions, and then computing the validation function, a party can advertise the transactions and validation output to other nodes. All the other nodes will WHO IS DOING IT? check the computation and begin publishing the blockchain with the new block of transactions. The salient property of the BitCoin validation function is that it is very difficult to compute but A working example, albeit minimal, of how a blockchain can enable a user to control his per- 3 very easy to check. To incentivize validators (miners), the system automatically rewards them sonal data can be seen in the MIT Media Lab’s credentials experiment. This is a blockchain- when their computation is accepted. based system for issuing certification of academic achievement (it is confusing to security- minded folks that they use the term “certificate” to mean “credential”). There is a central authority (The Media Lab) that manages a blockchain. This system encodes information about a stu- WHAT IS A BLOCKCHAIN IDENTITY? dent’s completion of a course and the student’s public key. The signed hash of the data structure is put on the blockchain, and the student receives a copy. The student can prove that he has the There are proposals to create blockchain systems and communities that use blockchains in which certificate by sending his copy of the certificate and the blockchain address to, for example, a the entries are somehow related to identification for an individual person or unique entity, such potential employer. The employer can see that the hash of the certificate matches the blockchain as a DNS name. In this model, the public key for a person or entity is represented in a blockchain entry and that it is signed by MIT. This system gives control over the information to the learner, entry, where it could be accessed by a relying party. In the simplest case, an entry would be and MIT does not need to communicate with the potential employer. Indeed, taken to the ex- signed by an identity provider of some kind, using its own public key, to assert: “The public key treme, MIT would not even need to retain any of the information; it belongs to the learner, and represented in this data item belongs to a person known as John Q. Public.” This is similar to ex- with great autonomy comes great responsibility. The learner must not lose his public key pair, isting public key systems, such as X.509v3 and PGP. and he must not lose track of his signed course completion document. Blockchains offer a couple of improvements, though. One is that the blockchain replication by a An additional advantage is that the Media Lab does not need to grant any special access provi- diverse set of entities ensure accessibility and reliability. Another is the immutability of the sions to the student. Not only does the granting institution not have to retain the record, they do blockchain and the timestamps; there can be little argument that an entry was made, and that can not have to protect any sensitive login information for the student. be shown for as long as the blockchain exists. Those familiar with the trials and travails of managed public key infrastructure (PKI) may note In order to know if blockchains solve other fundamental identity problems, we need to look more that there are similarities between this approach and traditional ones, and there are some striking deeply into why identity has become so important. Even though we still use passwords and spe- shortcuts. In this approach, the learner does not present a public key to the granting institution. cial purpose apps, digital identities based on public keys are the only hope of getting out of the Although the experimenters intended to add that in the future, the reality of the situation is that morass of relying on companies and governments to keep their customer or citizen data private. only a very sophisticated user can generate a public key pair, so it is up to the granting institution Their inability to do so is demonstrated on a regular basis. to generate a key pair for the learner and to communicate it to him securely. Of course, in the future, that will all be taken care of “in the app.” Neither does the learner, or anyone else, have a There are two kinds of identity being discussed today. One is for most of the people in countries certificate identifying the granting institution, nor any way to determine the legitimacy of the site with good governmental record keeping, reliable power for computing devices, and nearly uni- with the blockchain. When the learner presents the document and its associated blockchain ad- versal Internet service; the other is for people who have been cut off from government recogni- dress to a potential employer, the employer can verify that the hash of the document is in the tion, isolated by virtue of abuse or persecution, who are poor, or who do not have physical safety blockchain and signed by the granting institution’s key. for possession. This can happen either because the their government is too weak to provide identity services, because they are ignored by their government, because they are victims of human While this seems straightforward, one cannot but wonder what advantage it offers over using trafficking, because they are refugees, etc. PKI certificates for the granting institution and the learner. The granting institution can give the signed copy of the course completion document to the student, and then any third party can re- The digitally dispossessed cannot rely on cellphones, cards with chips, or anything physical to ceive the document from the student and validate the signature. provide their identity. What then identifies them? One way would be to record biometric data such as iris scans, facial features, fingerprints, or DNA. A cryptographic hash of the data could The Media Lab system does include a provision for that bugaboo of public key systems, the rev- be an anchor point for attaching attributes such as country of residence, immunization status, etc. ocation of keys. In this case, the revocation of the course completion document can be achieved The blockchain would hold the cryptographic hash, and various authorities could certify posses- by adding an entry to that effect onto the blockchain. sion of attributes. The feasibility of this is uncertain because biometric attributes are “fuzzy,” Finally, the Media Lab felt that Merkle trees offered advantages over blockchains.4 They envi- while a cryptographic hash is anything but. Still, one could imagine a solution based on varying sion a hybrid system in which users store their identity credentials on a blockchain, but the grant- degrees of accuracy and multiple measurements to create attribute collections that could map to a ing institutions use Merkle trees for storing their endorsed documents for users. Though unique match. minimal, this kind of system might be a solution for storing biometric data for the digitally dis- In an ideal world, none of the identity attributes could be released without permission from the possessed. There is still a great challenge to be met in providing user consent, but there is an im- person being identified. But if that person is digitally dispossessed, then she has no way to pre- minent need to help the world’s vulnerable people, and circumstances may overtake technology sent a public key to unlock the record; instead, the biometric measurements alone are enough to in this sector. unlock the information. This may well be the future identification situation not only for the digi- There are much more ambitious projects that promise users detailed control over their identifying tally dispossessed, but perhaps for most of the world’s population, if their governments insist on information and credentials. IBM and SecureKey are working with Canadian partners for an it. identity system based on the Linux Hyperledger software core (one of many open source blockchain projects). Microsoft got on their bandwagon recently with an announcement from their

March/April 2018 25 www.computer.org/internet March/Aprilwww.computer.org/computingedge 2018 26 www.computer.org/internet13 PRACTICAL SECURITY

March/April14 2018 ComputingEdge 27 www.computer.org/internetJanuary 2019 PRACTICAL SECURITY IEEE INTERNET COMPUTING

identity division.5 Another open source blockchain project, Sovrin, is the foundation for “self- sovreign” identity technology being developed by Evernym (https://www.evernym.com). THE FUTURE These systems appear to promote the concept of releasing all or part of the attributes in a per- An underlying current in all the blockchain identity fervor is the desire to start over with digital sonal credential. The canonical example is a driver’s license. People use their licenses for a vari- identity. A fresh beginning, unhampered by stodgy standards and the shackles of complex soft- ety of purposes that have nothing to do with driving: age (to get a drink), residence (to vote, to ware, a new identity ecosystem is beginning to be born. I think that vision, vague as it is, might use a public library), date of birth (for health care, sometimes), appearance (to prevent identity be the future of identity for the world. We could have the forced Orwellian government-issued fraud). Many people would prefer to release only the minimal necessary information to the re- privacy-depriving biometric credentials coexisting with subversive underground identities. free questor. In the last decade or so, cryptographic techniques for achieving this kind of controlled flowing reputation systems, and multiple personality disorder do-it-yourself identities for the disclosure have been developed in whole or in part: secret handshakes, hidden credentials, obliv- people on the fringes. ious attributes, and the general technique of homomorphic encryption. The developers of these It is clear that eventually, every human will have some representation as digital attributes. This is new identity systems would like to build in the privacy protections afforded by controlled release good and bad, like civilization itself. As humans, we seem to enjoy building bigger and bigger of information. societies, even though that has a huge effect on how we view ourselves as individuals. It is des- While this seems laudable, there are many questions to be answered, not the least of which is an tiny. overall analysis of security and usability. Some of the advanced cryptographic techniques are Based on the nonproliferation of PKI, my prediction is that when we do finally reach the point of impossibly slow, for example. Another risk is that there may be a hidden reliance on a trusted having digital identities instead of paper documents and login passwords, then they will be built third-party underlying the cryptography. If that third-party is shown to be untrustworthy or cor- on some kind of distributed ledger, something resembling a blockchain. I only hope that it im- rupted, then the privacy of the system might be undermined in ways that are difficult to detect. proves security, rather than making it ever more obscure and tenuous. The problem of generating secure public keys remains a crucial issue, one that seems to trip up hardware vendors again and again.6 More practical issues center on whether or not the user should be able to hide essential information like the expiration date of a driver’s license or special notations about wearing glasses. And, how can the user give consent to organ donation in the event of a fatality? Working out REFERENCES each contingency, even for ordinary credentials, may take us some years. 1. ID2020, identity for the world, ID 2020 Initiatve, 2017; https://id2020.org. 2. K. Cameron, “The Laws of Identity,” Microsoft Corporation, 2005; From the point of view of a credential issuing or attribute using organization, there are ad- https://msdn.microsoft.com/en-us/library/ms996456.aspx. vantages to being able to reduce the attack surface for privacy, but there are uses for collections 3. “What we learned from designing an academic certificates system on the blockchain,” of personal data that are at odds with privacy. Organizations need to do analyses of their user MIT Media Lab, 2016; https://medium.com/mit-media-lab/what-we-learned-from- base in order to tailor their services. How many users are under the age of 25? How many live in designing-an-academic-certificates-system-on-the-blockchain-34ba5874f196. cold climates? Will companies really be motivated to take privacy more seriously in a block- 4. R.C. Merkle, “A Digital Signature Based on a Conventional Encryption Function,” chain world? Advances in Cryptology — CRYPTO ’87, Springer, 1987. 5. A. Simon, “The Future as We See It,” blog, 2018; Another concern is the complications for the user of managing a large number of secure block- https://cloudblogs.microsoft.com/enterprisemobility/2018/02/12/decentralized-digital- chain “blobs.” Many new systems tout the ability of the user to create different “identities” identities-and-blockchain-the-future-as-we-see-it. (keys) for different uses, but that gets complicated. Moreover, it remains to be seen if credential 6. N. Heninger et al., “Mining Your Ps and Qs: Detection of Widespread Weak Keys in issuers will go along with such a model. For example, if someone uses a driver’s license to get a Network Devices,” Proc. 21st USENIX Security Symposium, 2012; special TSA boarding privilege, will TSA issue the credential to an arbitrary key provided by the https://factorable.net/weakkeys12.extended.pdf. user, or will they issue it to the same key that is bound to the driver’s license? Realistically, in the fullness of time, all of the PKI management problems will devolve to the blockchain. Companies acquiring other companies will want to re-sign all the credentials issued by subsidiaries, teenagers will want to have credentials that their parents controlled reissued to ABOUT THE AUTHOR their “grown up” key, etc. It seems that these issues, and similar ones, might lead to the same kind of inconvenient X.509 certificate chains that bog down PKI; will we have blockchain certif- Hilarie Orman is a security consultant and president of Purple Streak. Her research inter- icate chains that replicate the same problems? ests include applied cryptography, secure operating systems, malware identification, security through semantic computing, and personal data mining. Orman has a BS in Public keys are essential for safe identities, but everything hinges on generating “good” keys, mathematics from the Massachusetts Institute of Technology. She’s a former chair of the ones that do not have hidden pitfalls. That, in turn, depends on the quality of the software and the IEEE Computer Society’s Technical Committee on Security and Privacy. Contact her at systems that generate them. [email protected]. A further risk is that there will be no reason for institutions to stay in the game without substan- tive advantages. If members drop out of a blockchain, then the guarantees of availability and non-malleability become weakened. The remaining members may decide to abandon the previous blockchain, either by not maintaining it at all and moving to a new blockchain, or by forking the previous blockchain. At some point, the Wayback Machine may be littered with the charred remains of blockchains that fell out of favor. This article originally appeared in Nonetheless, almost anyone in the identity game today probably is considering a move to block- IEEE Internet Computing, vol. 22, no. 2, 2018. chain for the convenience that it could bring in widespread availability and risk reduction.

March/April 2018 27 www.computer.org/internet March/Aprilwww.computer.org/computingedge 2018 28 www.computer.org/internet15 CALL FOR PAPERS 2019 IEEE World Congress on Services BigData/CLOUD/EDGE/ICCC/ICIOT/ICWS/SCC July 8-13, 2019 Milan, Italy http://conferences.computer.org/services/2019/

2019 IEEE World Congress on Services (SERVICES) will be held on July 8-13, 2019 in Milan, Italy. The Congress is solely sponsored by the IEEE Computer Society under the auspice of the Technical Committee on Services Computing (TCSVC). The scope of the Congress will cover all aspects of services computing and applications, current or emerging. It covers various systems and networking research pertaining to cloud, edge and Internet-of-Things (IoT), as well as technologies for intelligent computing, learning, big data and blockchain applications, while addressing critical issues such as high performance, security, privacy, dependability, trustworthiness, and cost-effectiveness. Authors are invited to prepare early and submit original papers to any of these conferences at www.easychair.org. All submitted manuscripts will be peer-reviewed by at least 3 reviewers. Accepted and presented papers will appear in the conference proceedings published by the IEEE Computer Society Press. The Congress will be organized with the following seven affiliated conferences/congresses: IEEE International Congress on Big Data (BigData Congress) 2019 Congress General Chair Big data acquisitions, analyses, storage, and mining for various services and applications Peter Chen, Carnegie Mellon University, USA

IEEE International Conference on Cloud Computing (CLOUD) 2019 Congress Program Chair-in-Chief Innovative cloud computing for both high quality infrastructure and mobile services Elisa Bertino, Purdue University, USA

IEEE International Conference on Edge Computing (EDGE) 2019 Congress Vice Program Chair-in-Chief High quality services computing between cloud systems and Iot devices Ernesto Damiani, University of Milan, Italy IEEE International Conference on Web Services (ICWS) Workshop Chairs Innovative web services for various effective applications Shangguang Wang, BUPT IEEE International Conference on Cognitive Computing (ICCC) Stephan Reiff-Marganiec, University of Leicester Cognitive computing, learning algorithms for intelligent services and applications Steering Committee IEEE International Congress on Internet of Things (ICIOT) Elisa Bertino, Purdue University, USA Innovative IoT technology for digital world services Carl K. Chang, Chair, Iowa State University, USA, Rong N. Chang, TCSVC Chair, IBM, USA IEEE International Conference on Services Computing (SCC) Intelligent services computing, lifecycles, infrastructure and mobile environments Peter Chen, Carnegie Mellon University USA Ernesto Damiani, University of Milan, Italy Key Dates Ian Foster, University of Chicago Early Paper submission due: December 1, 2018 & Argonne National Lab, USA Review comments to authors of early-submission papers: January 15, 2019 Dennis Gannon, Indiana University, USA Normal Paper submission due: February 4, 2019 Michael Goul, Arizona State University, USA Final notification to authors: March 15, 2019. Frank Leymann, University of Stuttgart, Germany Camera-ready manuscripts due: April 1, 2019 Hong Mei, Beijing Institute of Technology, China Congress Date: July 8 – 13, 2019 Stephen S. Yau, Arizona State University, USA CALL FOR WORKSHOP PROPOSALS: See http://conferences.computer.org/services/2019/ for more information. Send inquiries to: [email protected] Editor: Cesare Pautasso University of Lugano INSIGHTS [email protected] Editor: Olaf Zimmermann University of Applied Sciences of Eastern Switzerland, Rapperswil [email protected] Adaptable Blockchain- Based Systems

A Case Study for Product Traceability

Qinghua Lu and Xiwei Xu

From the Editors Tracing the origin of products across complex supply chains requires a transparent, tamper-proof metadata infrastructure that’s not only trusted by all the involved parties but also adaptable to changing environments and regulations. Can such advanced infrastructure be implemented in a decentralized way? Qinghua Lu and Xiwei Xu share their story of developing the originChain system, which leverages emerging blockchain technology to do so. —Cesare Pautasso and Olaf Zimmermann

A TRACEABILITY SYSTEM en- In this context, security is impor- suppliers and retailers use its trace- ables tracking products by provid- tant for accountability and forensic ability services to manage their ing information about them (for information. Traceability systems products’ traceability information, example, originality, components, normally store information in con- and millions of product consumers or locations) during production and ventional databases controlled by use it to access the information. distribution. Product suppliers and the service providers. Such central- Here, we share our experience retailers usually require independent ized data storage becomes a poten- of building originChain. It restruc- traceability service providers who tial single point of failure and runs tures the service provider’s current are government-certifi ed to inspect the risk of tampering. traceability system by replacing the products throughout the supply One of our partners is an inde- the central database with a blockchain. If everything satisfi es the re- pendent third-party traceability ser- chain. (For more on blockchains, quirements, the traceability service vice provider certifi ed by the Chinese see the two sidebars.) OriginChain providers issue inspection certifi cates government. Its system provides provides transparent tamper-proof that verify the products’ quality and traceability information for products traceability data, enhances the originality. To expose information imported to China. This system has data’s availability, and automates and generate certifi cates, these ser- been integrated with several big Chi- regulatory-compliance checking. vice providers employ a traceability nese e-commerce websites (for exam- We implemented originChain and system. ple, JD.COM). Hundreds of product tested it under realistic conditions

BLOCKCHAINS AND SMART public access provides data transparency, and every CONTRACTS participant has potentially the same ability to access and manipulate the blockchain. However, those rights As a data structure, a blockchain is an ordered list of blocks can be weighted by the participants’ stake or computational that contain transactions such as monetary transfers and power. To facilitate transactions, the participants rely on smart-contract creation and invocation.1 Each block contains the blockchain network itself instead of trusted third-party a hash of the previous block’s representation, thus creating organizations. the chain. So, historical transactions in the blockchain can’t be deleted or altered without invalidating the chain of hashes. LIMITATIONS Combined with computational constraints and incentive Blockchains lack data privacy; there are no privileged users, schemes for block creation, this can prevent the tampering and, as we just mentioned, every participant can access with and revision of information in the blockchain. all the information on the blockchain. In addition, public Blockchains also provide a general-purpose program- blockchains have limits on the amount of data, transaction mable infrastructure. Programs can be deployed and run on processing rate, and data transmission latency. Consortium a blockchain; such programs are called smart contracts.2 blockchains, in which the consensus process is limited to The result of a smart-contract invocation is stored in public several participants, perform much better. However, develop- data storage. Smart contracts can express triggers, condi- ers still must consider these factors when designing systems. tions, and business logic to enable more complex program- Furthermore, some public blockchains use a proof-of-work mable transactions. Hence, smart contracts differ from consensus mechanism that “wastes” signifi cant electricity service contracts in service-oriented computing, which are because it doesn’t lead directly to a successful solution. interfaces between services and consumers so that they Researchers are developing alternative consensus mecha- can successfully interact. A common simple example of nisms for public blockchains. One example is the proof-of-stake a smart-contract-enabled service is escrow, which holds mechanism, which isn’t computationally expensive. Consortium funds until the obligations defi ned in the smart contract are and private blockchains also often use consensus mechanisms f u l fi l l e d. that don’t rely on proof of work.

PROPERTIES References Any data in a committed transaction eventually becomes 1. M. Swan, Blockchain: Blueprint for a New Economy, O’Reilly Media, immutable. The immutable chain of cryptographically signed 2015. historical transactions provides nonrepudiation of the stored 2. S. Omohundro, “Cryptocurrencies, Smart Contracts, and Artifi cial data. Cryptographic tools also support data integrity, the Intelligence,” AI Matters, vol. 1, no. 2, 2014, pp. 19–21.

employing the users’ traceability in- Each product supplier that uses and Notation).1 In the real world, formation. We’re planning how to our partner’s services has on average the sequence of some activities in replace the existing system with the 20 products to be traced. The trace- Figure 1 (such as “examine factory” restructured one. ability information’s granularity is and “test sample”) is dynamic ow- rather large because it corresponds ing to customization of the quality Dealing with Traceability to product packages rather than indi- control and inspection processes. Product suppliers and retailers apply vidual products. This information’s The labs that test samples must for traceability services for different size isn’t easy to estimate because adapt to the labs’ availability and purposes. Suppliers want to receive many documents currently aren’t the characteristics of the products, certifi cates to show their products’ digitized, such as certifi cates issued such as powdered milk or clothing origin and quality to consumers and by traceability service providers. materials. to comply with regulations. Retail- Traceability is fl exible. Figure 1 Furthermore, regulatory-compliance ers want verifi cation of the products’ shows a simplifi ed possible process checking can change because of new origin and quality. in BPMN (Business Process Model regulations. For example, China’s

1822 IEEE SOFTWAREComputingEdge | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE January 2019 INSIGHTS INSIGHTS

BLOCKCHAINS AND SMART public access provides data transparency, and every BLOCKCHAINS IN THE secure the real-time data generated during the samples’ CONTRACTS participant has potentially the same ability to access SUPPLY CHAIN delivery.4 and manipulate the blockchain. However, those rights As a data structure, a blockchain is an ordered list of blocks can be weighted by the participants’ stake or computational The supply chain is a promising area for applying block- References that contain transactions such as monetary transfers and power. To facilitate transactions, the participants rely on chains.1 There are blockchain startups in this field. For 1. M. Staples et al., Risks and Opportunities for Systems Using Block- smart-contract creation and invocation.1 Each block contains the blockchain network itself instead of trusted third-party example, Everledger (www.everledger.io) uses blockchains chain and Smart Contracts, Commonwealth Scientific and Industrial a hash of the previous block’s representation, thus creating organizations. to track diamonds’ features, such as cut and quality, and Research Org., 2017. the chain. So, historical transactions in the blockchain can’t to help reduce risk and fraud for banks, insurers, and open 2. M. del Castillo, “Microsoft Unveils Project Manifest, a Plan for be deleted or altered without invalidating the chain of hashes. LIMITATIONS marketplaces. Blockchain Product Tracking,” CoinDesk, 25 Jan. 2017; www Combined with computational constraints and incentive Blockchains lack data privacy; there are no privileged users, Big enterprises are also applying blockchains in supply .coindesk.com/microsoft-unveils-project-manifest-a-plan-for schemes for block creation, this can prevent the tampering and, as we just mentioned, every participant can access chains for different domains. For example, in January -product-tracking-via-blockchain. with and revision of information in the blockchain. all the information on the blockchain. In addition, public 2017, Microsoft started the Manifest project through a 3. M. del Castillo, “Microsoft’s Blockchain Supply Chain Project Grows Blockchains also provide a general-purpose program- blockchains have limits on the amount of data, transaction partnership with Mojix to leverage an Internet-of-Things to 13 Partners,” CoinDesk, 3 May 2017; www.coindesk.com mable infrastructure. Programs can be deployed and run on processing rate, and data transmission latency. Consortium platform with a blockchain to help factories, distribution /microsofts-blockchain-supply-chain-project-grows-to-13 a blockchain; such programs are called smart contracts.2 blockchains, in which the consensus process is limited to centers, and retailers track goods using RFID devices.2 In -partners. The result of a smart-contract invocation is stored in public several participants, perform much better. However, develop- May 2017, Manifest grew to 13 partners.3 BHP Billiton 4. P. Rizzo, “World’s Largest Mining Company to Use Blockchain for data storage. Smart contracts can express triggers, condi- ers still must consider these factors when designing systems. has been exploring blockchain technology to track Supply Chain,” CoinDesk, 23 Sept. 2016; www.coindesk.com tions, and business logic to enable more complex program- Furthermore, some public blockchains use a proof-of-work movements of wellbore rock and fluid samples and /bhp-billiton-blockchain-mining-company-supply-chain. mable transactions. Hence, smart contracts differ from consensus mechanism that “wastes” signifi cant electricity service contracts in service-oriented computing, which are because it doesn’t lead directly to a successful solution. interfaces between services and consumers so that they Researchers are developing alternative consensus mecha- can successfully interact. A common simple example of nisms for public blockchains. One example is the proof-of-stake a smart-contract-enabled service is escrow, which holds mechanism, which isn’t computationally expensive. Consortium Food Safety Law, which took effect consortium blockchain can perform about which traceability services funds until the obligations defi ned in the smart contract are and private blockchains also often use consensus mechanisms in October 2015, set out new require- better and cost less. are covered. OriginChain generates f u l fi l l e d. that don’t rely on proof of work. ments for formulating national food Blockchains grow continually a smart contract that represents the safety standards and traceability sys- because the data and code on them legal agreement. (For more infor- PROPERTIES References tems (specifying what information are immutable. So, a major design mation on smart contracts, see the Any data in a committed transaction eventually becomes 1. M. Swan, Blockchain: Blueprint for a New Economy, O’Reilly Media, should be provided). So, adaptability decision is to choose what data and sidebar “Blockchains and Smart Con- immutable. The immutable chain of cryptographically signed 2015. was one of our main concerns when computation to keep on-chain and tracts.”) The smart contract codifies historical transactions provides nonrepudiation of the stored 2. S. Omohundro, “Cryptocurrencies, Smart Contracts, and Artifi cial we designed originChain. off-chain. We discuss this in more the combination of services and other data. Cryptographic tools also support data integrity, the Intelligence,” AI Matters, vol. 1, no. 2, 2014, pp. 19–21. detail later. conditions defined in the agreement. OriginChain’s Architecture As Figure 2 shows, product sup- So, the smart contract can automati- Figure 2 illustrates originChain’s pliers or retailers manage product cally check and enforce these condi- architecture. OriginChain currently or enterprise information through tions. It also checks whether all the employing the users’ traceability in- Each product supplier that uses and Notation).1 In the real world, employs a geographically distrib- the product and enterprise manage- information required by regulation formation. We’re planning how to our partner’s services has on average the sequence of some activities in uted private blockchain at the trace- ment module. They access the infor- is provided, to enable automated replace the existing system with the 20 products to be traced. The trace- Figure 1 (such as “examine factory” ability service provider, which has mation on the blockchain through a regulatory-compliance checking. restructured one. ability information’s granularity is and “test sample”) is dynamic ow- branch offices in three countries. webserver hosted by originChain. In The traceability service provider rather large because it corresponds ing to customization of the quality The plan is to establish a trust- the future, suppliers and big retail- manages traceability information, Dealing with Traceability to product packages rather than indi- control and inspection processes. worthy traceability platform that ers that host a node by themselves certificates, and onsite photos using Product suppliers and retailers apply vidual products. This information’s The labs that test samples must covers other organizations, including will be able to access their own the traceability management mod- for traceability services for different size isn’t easy to estimate because adapt to the labs’ availability and government-certified labs, big sup- nodes to obtain information on the ule. Because of the blockchain’s lim- purposes. Suppliers want to receive many documents currently aren’t the characteristics of the products, pliers, and retailers that have long- blockchain. ited data storage, originChain stores certifi cates to show their products’ digitized, such as certifi cates issued such as powdered milk or clothing term relationships with the company After the traceability service pro- two types of data on-chain as vari- origin and quality to consumers and by traceability service providers. materials. (such as e-commerce companies vider validates an application from ables of smart contracts: to comply with regulations. Retail- Traceability is fl exible. Figure 1 Furthermore, regulatory-compliance that have already built their repu- a product supplier or retailer on the ers want verifi cation of the products’ shows a simplifi ed possible process checking can change because of new tation among customers). Com- basis of paperwork (see Figure 1), the • the hash of traceability certifi- origin and quality. in BPMN (Business Process Model regulations. For example, China’s pared to a public blockchain, such a two parties sign a legal agreement cates or photos and

22 IEEE SOFTWARE | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE www.computer.org/computingedge NOVEMBER/DECEMBER 2017 | IEEE SOFTWARE 2319 INSIGHTS

information stored in the block- Supplier Apply for Receive chain. It can’t directly access the or retailer traceability certiﬁcate states of external systems (for ex-

Check Issue ample, testing results and product Administration application certiﬁcate geolocations). Thus, the labs periodi-

Factory Examine cally inject the result of sample test- + examiner factory ing from the external world into the blockchain. Freightyard Check Supervise + Seal examiner products loading The information of blockchain- Traceability service provider Traceability layer permission control (for example, Lab Test sample permission for content management, for writing smart contracts, or for joining a consortium blockchain) FIGURE 1. The dynamism of the traceability process. The sequence of some activities can be on-chain or off-chain. How- (such as “examine factory” and “test sample”) is dynamic owing to customization of the ever, an off-chain centralized per- quality control and inspection processes. mission management module could become a single point of failure from both an operational and a manage- Supplier or Traceability service Lab UI retailer UI provider UI ment perspective. So, originChain

Product and stores the control information, such Sample test Traceability enterprise management management as permission to join the blockchain management network (to own a copy of all the Product and historical transactions). On-chain Suppliers or retailers Sample test Traceability enterprise results database database database permission management leverages the blockchain’s decentralized nature so that all the participants can access the blockchain. Node Node Node Node Figure 3 shows how smart contracts are designed using our blockchain; Figure 4 shows part of the Node Node Node Consortium blockchain related pseudocode. In originChain, Suppliers or big retailers Labs Traceability service provider a factory contract creates smart contracts. This reduces the complexity FIGURE 2. OriginChain’s architecture. OriginChain currently employs a geographically of creating customized smart con- distributed private blockchain at the traceability service provider company. The plan is to tracts. The factory contract contains establish a consortium blockchain, which will include other organizations. code fragments representing different traceability services. The generation of smart contracts requires • the small amount of traceabil- database hosted by originChain. authority from both the traceability ity information required by the Other partners can still maintain service provider and the supplier or traceability regulation, such as their own database of product infor- retailer. the batch number, traceability mation (for the suppliers or retailers) When the factory contract is results, place of origin, and in- or other numbers shown in the sam- called, it creates two kinds of smart spection date. ple testing (for the labs). contract: a registry contract and The labs manage sample-testing service contract. The registry con- The raw files of traceability certifi- results through the sample test man- tract represents the legal agreement cates and photos (.pdf or .jpg) and agement module. A blockchain’s exe- and contains the address of the ser- the addresses of the smart contracts cution environment is self-contained. vice contract, which codifies the le- are off-chain in a centralized MySQL So, a smart contract can access only gal agreement. The service contract

2024 IEEE SOFTWAREComputingEdge | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE January 2019 INSIGHTS INSIGHTS

information stored in the block- could be updated by replacing its Supplier or Traceability service Administrator Supplier Apply for Receive chain. It can’t directly access the address stored in the registry con- retailer provider or retailer traceability certiﬁcate states of external systems (for extract with the address of a new ver-

Check Issue ample, testing results and product sion. Possible updates include adding Administration Authorizes Authorizes application certiﬁcate geolocations). Thus, the labs periodi- or removing services from the le- Authorizes Authorizes Factory Examine cally inject the result of sample test- gal agreement after the initial legal + examiner factory ing from the external world into the agreement is signed, or selecting blockchain. labs for sample testing on the ba- Registry Freightyard Check Supervise + Seal Factory contract Blockchain examiner products loading The information of blockchain- sis of their availability. The registry Traceability service provider Traceability contract Creates (contains permission layer permission control (for example, contract specifies a list of addresses address of control contract Lab Test sample service permission for content management, allowed to update the registry con- contract) for writing smart contracts, or for tracts, and a threshold of the mini- Creates Refers to joining a consortium blockchain) mum number of addresses required FIGURE 1. The dynamism of the traceability process. The sequence of some activities can be on-chain or off-chain. How- to authorize an update. (such as “examine factory” and “test sample”) is dynamic owing to customization of the ever, an off-chain centralized per- If a testing sample involves multi- quality control and inspection processes. mission management module could ple labs for crosschecking, signatures Service become a single point of failure from from all the labs are required before contract both an operational and a manage- the traceability application can un- Supplier or Traceability service Lab UI retailer UI provider UI ment perspective. So, originChain dergo further processing. To enable

Product and stores the control information, such more dynamic lab selection, users Sample test Traceability enterprise Blockchain management management as permission to join the blockchain can employ an M-of-N multisigna- management network (to own a copy of all the ture to define that M out of N labs Lab assistant Product and historical transactions). On-chain are required to authorize the testing Suppliers or retailers Sample test Traceability enterprise results database database database permission management leverages results. FIGURE 3. Designing smart contracts in originChain. The factory contract reduces the blockchain’s decentralized na- the complexity of creating customized smart contracts. ture so that all the participants can Lessons Learned access the blockchain. Our experiences with originChain Node Node Node Node Figure 3 shows how smart con- led to the following insights. its size because the code is stored in Performance depends highly on the tracts are designed using our block- the blockchain, which entails data blockchain’s deployment. For exam- chain; Figure 4 shows part of the The Design of Blockchain-Based storage fees proportional to the con- ple, a consortium blockchain can be Node Node Node Consortium blockchain related pseudocode. In originChain, Systems tract’s size. So, a structural design configured to perform much better Suppliers or big retailers Labs Traceability service provider a factory contract creates smart con- Owing to blockchains’ unique prop- with more lines of code costs more than a public blockchain. tracts. This reduces the complexity erties, some design considerations money. With originChain, owing to the FIGURE 2. OriginChain’s architecture. OriginChain currently employs a geographically of creating customized smart con- are specific to blockchain-based A consortium blockchain doesn’t current traceability system’s char- distributed private blockchain at the traceability service provider company. The plan is to tracts. The factory contract contains applications—for example, the con- have to have a token or currency, so acteristics (for example, it has low establish a consortium blockchain, which will include other organizations. code fragments representing differ- sideration of on-chain and off-chain. monetary cost isn’t an issue there. writing throughput because of the ent traceability services. The gen- On the other hand, because smart However, the blockchain’s size is still large granularity of traceability in- eration of smart contracts requires contracts are programs running a design concern because it grows formation), a blockchain’s limited • the small amount of traceabil- database hosted by originChain. authority from both the traceability on a blockchain, some existing ar- with every transaction and because throughput isn’t the main concern. ity information required by the Other partners can still maintain service provider and the supplier or chitectural patterns might be ap- every participant has a replica of However, as we mentioned before, traceability regulation, such as their own database of product infor- retailer. plicable to them. From the business the whole blockchain. In addition, a all the participants can access the the batch number, traceability mation (for the suppliers or retailers) When the factory contract is process perspective, approaches such more structural design might affect data on the blockchain. So, private results, place of origin, and in- or other numbers shown in the sam- called, it creates two kinds of smart as model-driven development2 and performance because it might re- data (for example, customer infor- spection date. ple testing (for the labs). contract: a registry contract and behavior-driven development1 are quire more transactions. mation) shouldn’t be on-chain. Re- The labs manage sample-testing service contract. The registry con- also applicable. garding traceability, large sensitive The raw files of traceability certifi- results through the sample test man- tract represents the legal agreement The smart contract’s structural On-Chain vs. Off-Chain raw data (for example, traceability cates and photos (.pdf or .jpg) and agement module. A blockchain’s exe- and contains the address of the ser- design has a large impact on the cost Regarding what should be on-chain certificates and photos) must be im- the addresses of the smart contracts cution environment is self-contained. vice contract, which codifies the le- if the blockchain is public. The con- and off-chain, two factors are immutable. Thus, the raw data is off- are off-chain in a centralized MySQL So, a smart contract can access only gal agreement. The service contract tract’s deployment cost depends on portant: performance and privacy. chain, whereas its hash is on-chain.

24 IEEE SOFTWARE | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE www.computer.org/computingedge NOVEMBER/DECEMBER 2017 | IEEE SOFTWARE 2521 INSIGHTS

The Adaptability of contract FactoryContract { Blockchain-Based Systems address[] registryContracts; address[] serviceContracts; Adaptability is a quality attribute // deploy a new registry contract required by many industrial proj- function newRegistryContract() returns(address ects that are inherently dynamic. newRegistryContract){ For example, changes to the legal RegistryContract r 5 new RegistryContract(); agreement or new regulations could registryContracts.push(r); return r; necessitate adaptation. Adaptation } here means that the smart contract // deploy a new service contract could be updated by a number of au- function newServiceContract() returns(address thorities above the threshold defined newServiceContract){ in the factory contract. However, re- ServiceContract s 5 new ServiceContract(); serviceContracts.push(s); search on blockchain-based systems return s; rarely discusses adaptability. } We view the blockchain as a com- } ponent of a larger distributed system. In originChain, we implement contract RegistryContract{ some of the business logic on-chain address[] serviceContracts; address[] authorities; as smart contracts. Thus, smart con- uint threshold; tracts’ structural design also affects uint authorityNum; their updatability and the whole sys- function authorizeVotingRight(address authority) { tem’s adaptability (for example, sep- authorities.push(authority); aration between data and control). } function setThreshold(uint threshold) { However, if the blockchain is for threshold 5 threshold; data storage only, not much can be } done to affect the whole system’s function vote(address authority) { adaptability. Moving some logic authorityNum11; to the blockchain can leverage the } function update() returns(address newServiceContract){ trustworthiness (and the interoper- // If there are enough authorities then ability and the transparency of data … and operations) that the blockchain If(authorityNum .5 threshold){ provides as a computational plat- ServiceContract s 5 new ServiceContract(); form. In addition, the data in smart serviceContracts.push(s); return s; contracts is easier to query (directly } on the blockchain) than is the data } in transactions.3 } Access Control for Smart Contracts contract ServiceContract { Smart contracts running on a block- bool testsampleselected; bool examinefactoryselected; chain can be accessed and called by bool superviseloadingselected; all the participants. A smart con- function ServiceContract (bool testsample, bool examinefactory, tract, by default, has no owner; once bool superviseloading){…} it’s deployed, its author no longer function testSample(){…} has any special privileges on it. Un- function examineFactory(){…} function superviseLoading(){…} authorized users could accidentally trigger a permissionless function. So, FIGURE 4. Pseudocode for the design of a smart contract in originChain. The smart contracts should have an em- generation of smart contracts requires authority from both the traceability service bedded permission control mecha- provider and the supplier or retailer. nism to check permission for every

2226 IEEE SOFTWAREComputingEdge | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE January 2019 INSIGHTS INSIGHTS

This article originally appeared in IEEE Software, vol. 34, no. 6, 2017.

The Adaptability of caller that triggers the functions de- contract FactoryContract { Blockchain-Based Systems fi ned in the contracts. address[] registryContracts; address[] serviceContracts; Adaptability is a quality attribute QINGHUA LU is an associate professor in the College of Computer // deploy a new registry contract required by many industrial proj- and Communication Engineering at China University of Petroleum function newRegistryContract() returns(address ects that are inherently dynamic. raceability processes in sup- (East China). Contact her at [email protected]. newRegistryContract){ For example, changes to the legal ply chain management are RegistryContract r 5 new RegistryContract(); agreement or new regulations could complex and dynamic be- registryContracts.push(r); T return r; necessitate adaptation. Adaptation cause they involve multiple par- } here means that the smart contract ties. A blockchain provides neutral // deploy a new service contract could be updated by a number of au- ground that should help integrate function newServiceContract() returns(address thorities above the threshold defined the disparate participants into those XIWEI XU is a research scientist in Data61 at the Commonwealth newServiceContract){ in the factory contract. However, re- processes. Also, the integrity and Scientifi c and Industrial Research Organisation (CSIRO). Contact ServiceContract s 5 new ServiceContract(); serviceContracts.push(s); search on blockchain-based systems audit trail in a blockchain ledger her at [email protected]. return s; rarely discusses adaptability. should improve transparency and } We view the blockchain as a com- confi dence across the processes. ABOUT THE AUTHORS THE ABOUT } ponent of a larger distributed sys- Although joining a consortium tem. In originChain, we implement blockchain benefi ts all the relevant contract RegistryContract{ some of the business logic on-chain stakeholders, adopting a new tech- address[] serviceContracts; address[] authorities; as smart contracts. Thus, smart con- nique such as a blockchain is always uint threshold; tracts’ structural design also affects a challenge to traditional industries uint authorityNum; their updatability and the whole sys- because of the learning curve and the Nevertheless, industry needs to take for Behavior-Driven Development,” function authorizeVotingRight(address authority) { tem’s adaptability (for example, sep- cost of integrating the blockchain into the time to understand their risks IEEE Software, vol. 33, no. 5, 2016, authorities.push(authority); aration between data and control). the existing systems. Negotiating the and opportunities. pp. 15–21. } function setThreshold(uint threshold) { However, if the blockchain is for business details also takes time. In ad- 2. I. Weber et al., “Untrusted Business threshold 5 threshold; data storage only, not much can be dition, the development of smart con- Acknowledgments Process Monitoring and Execution } done to affect the whole system’s tracts must take into account quality The National Natural Science Foundation Using Blockchain,” Business Process function vote(address authority) { adaptability. Moving some logic attributes such as adaptability. of China (grant 61402533) and the Fun- Management, LNCS 9850, Springer, authorityNum11; to the blockchain can leverage the Data transparency and shar- damental Research Funds for the Central 2016, pp. 329–347. } function update() returns(address newServiceContract){ trustworthiness (and the interoper- ing data with others are main con- Universities (grant 16CX02047A) sup- 3. X. Xu et al., “A Taxonomy of // If there are enough authorities then ability and the transparency of data cerns for most companies that ported this project. Blockchain-Based Systems for Archi- … and operations) that the blockchain provide intermediary services in in- tecture Design,” Proc. 2017 IEEE If(authorityNum .5 threshold){ provides as a computational plat- dustries. Overall, blockchains are References Int’l Conf. Software Architecture ServiceContract s 5 new ServiceContract(); form. In addition, the data in smart a good option for providing trace- 1. D. Lübke and T. van Lessen, (ICSA 17), 2017, pp. 243–252. serviceContracts.push(s); return s; contracts is easier to query (directly ability in supply chain management. “Model ing Test Cases in BPMN } on the blockchain) than is the data } in transactions.3 }

Access Control for Smart Contracts • a jUlY COMPUTING INTERNET IEEE • a jUlY COMPUTING INTERNET IEEE M COMPUTING INTERNET IEEE M COMPUTING INTERNET IEEE contract ServiceContract { M COMPUTING INTERNET IEEE M COMPUTING INTERNET IEEE july • augustjuly 2016 • augustjuly 2016

Smart contracts running on a block- NO COMPUTING INTERNET IEEE NO COMPUTING INTERNET IEEE may • junemay 2016 • junemay 2016 march • april 2016 march • april 2016 bool testsampleselected; COMPUTING INTERNET IEEE COMPUTING INTERNET IEEE UGU UGU a a 2016 sT 2016 sT aY • j aY • j RC RC h • a h • a UNE UNE

PRI PRI november • december 2015 november • december 2015 2016 2016 january • february 2016 january • february 2016 v v l 2016 l 2016

chain can be accessed and called by EM EM ja ja NU NU b b

ER ER aRY • fEb aRY • fEb aRY

bool examinefactoryselected; • D • D ECEM ECEM RU RU b b 2016 aRY 2016 aRY ER ER 2015 2015

ME ME ExPl ExPl all the participants. A smart con- as as bool superviseloadingselected; ORING ORING URING URING Th Th Cl Cl I I NTERNET NTERNET T T E I E I OU OU ThE INTERNET ThE INTERNET ThE OMORRO OMORRO NTERNET NTERNET D s D s E E TOR TOR CONOMIC CONOMIC Of YOU YOU Of YOU Of w’s INTERNET w’s INTERNET w’s aGE aGE aGE function ServiceContract (bool testsample, bool examinefactory, tract, by default, has no owner; once s s vOl. 20, NO. 1 NO. 20, vOl. 1 NO. 20, vOl. vO vO l. 19, 6 NO. l. 19, 6 NO. l. vOl. 20, NO. 3 NO. 20, vOl. 3 NO. 20, vOl. vOl. 20, NO. 2 NO. 20, vOl. 2 NO. 20, vOl. bool superviseloading){…} 4 NO. 20, vOl. 4 NO. 20, vOl. it’s deployed, its author no longer www. www. www. www. COMPUTER COMPUTER COMPUTER COMPUTER . ORG . ORG www. www. www. www. / /

function testSample(){…} INTERNET INTERNET . ORG . ORG www. www. COMPUTER COMPUTER COMPUTER COMPUTER / / INTERNET INTERNET / / COMPUTER COMPUTER / / has any special privileges on it. Un- . ORG . ORG . ORG . ORG / / INTERNET INTERNET / /

INTERNET INTERNET IC-20-01-c1 Cover-1 IC-20-01-c1 Cover-1 December 7, 2015 1:45 PM December 7, 2015 1:45 PM function examineFactory(){…} . ORG . ORG / /

IC-19-06-c1 Cover-1 IC-19-06-c1 Cover-1 October 9, 2015 3:26 PM October 9, 2015 3:26 PM INTERNET INTERNET / / / / authorized users could accidentally / / function superviseLoading(){…} IC-20-03-c1 Cover-1 IC-20-03-c1 Cover-1 April 13, 2016 8:45 PM April 13, 2016 8:45 PM IC-20-02-c1 Cover-1 IC-20-02-c1 Cover-1 February 11, 2016 10:30 PM February 11, 2016 10:30 PM trigger a permissionless function. So, FIGURE 4. Pseudocode for the design of a smart contract in originChain. The smart contracts should have an em- WantWant to know to know more more about about the Internet?the Internet? This magazineThis magazine covers coversall aspects all aspects of Internet of Internet computing, computing, from programming from programming and standards and standards to security to security and networking. and networking. generation of smart contracts requires authority from both the traceability service bedded permission control mecha- provider and the supplier or retailer. nism to check permission for every www.computer.org/internetwww.computer.org/internet

26 IEEE SOFTWARE | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE www.computer.org/computingedge NOVEMBER/DECEMBER 2017 | IEEE SOFTWARE 2723 THE IOT CONNECTION

Accurate Indoor Location for the IoT

Roy Want, Wei Wang, and Stan Chesnutt, Google

The combination of IoT and accurate location- thermostats, barometers, moisture ﬁ nding technologies will enable a wide range of sensors, and when extended to in- novel context-aware services to automate tasks dustry, a wide variety of exotic sensors. Actuator outputs include and support everyday work practice. lamps, valves (water sprinklers), speakers, and a variety of displays— again, industry has many additional needs for bespoke actuators. The de ning characteristic of the Internet of interrelationship of the position and function of each of Things (IoT) is that devices are both networked these devices allows us to design systems that are often and embedded in the environment. Unlike tra- called “smart homes” or “smart buildings.” ditional networked server technology that has An additional consideration of spatially arranged in- veryA little connection with the environment, IoT devices frastructure is that mobile wireless components can move are designed and positioned so that they can respond to through it. In a smart home, people will be present and the and in uence their surroundings. For example, if a user infrastructure is intended to make their lives easier and accesses a web server from an o ce in California, it makes more comfortable. Devices such as lights, HVAC, and se- very little di erence whether the service is located in the curity systems can be turned on or o by the presence and next building or situated in Europe. However, if a user ac- detection of people entering or leaving an area. This is not cesses an IoT device that is responsible for measuring the only convenient, but also addresses human forgetfulness temperature of a room, the sensor and device need to be and has cost-saving implications because it reduces en- located in that speci c room. ergy consumption. As a result, IoT design is a unique departure from in- In the modern smart home, voice-driven assistants frastructural networking that tries to abstract away the have become popular (such as Amazon Echo and Goo- network topology; instead, the topology and position of gle Assistant) and allow devices in smart homes to be IoT nodes are assets in performing the networking task. activated by casual voice commands. Typically, human IoT devices often contain sensors and actuators. Con- speech is often tersely phrased with the assumption that sumer products for IoT sensors include light sensors, mi- the context is understood. This is challenging for a com- crophones, cameras, security switches, motion detectors, puter, which doesn’t have the shared context to respond

r8iot.indd 66 7/31/18 11:17 AM EDITOR ROY WANT THE IOT CONNECTION Google; [email protected]

to these commands. One of the biggest quali ers of context is a user’s posi- AP Range tion. For example, an ambiguous com- 2 AP 1 Range mand such as “turn on this lamp” can be immediately quali ed if the system knows the relative position of the user and all the nearby lamps. Range Although it might be hard to deter- 4 3 mine the location of a person, it is now Range Range 5 possible to accurately calculate the position of a mobile device using Wi-Fi radios (see www.ieee .org/) with positioning support. Since the early Accurate Indoor AP AP Wi-Fi ( .b) products appeared in AP , Received Signal Strength Indica- tion (RSSI) has been used to determine Figure . Multilateration, the process of converting ranges to a position. AP: access point. Location for the IoT the approximate position of a mobile device relative to surrounding access Roy Want, Wei Wang, and Stan Chesnutt, Google points installed at known positions. capability based on time of ight (ToF) to cover a work area are not More recently, Bluetooth low-energy of Wi-Fi packets. As these packets are great; beacons have been used in a similar transmitted as RF signals that travel › its power consumption is low The combination of IoT and accurate location- way, but at higher density due to their at the speed of light c, the distance enough that it can be embedded thermostats, barometers, moisture small size and low cost. However, due can be calculated by knowing the ToF into small mobile devices with ﬁ nding technologies will enable a wide range of sensors, and when extended to in- to multipath fading, the RSSI fall-o of a packet from a mobile to an access acceptable impact on battery life novel context-aware services to automate tasks dustry, a wide variety of exotic from the antenna does not follow a point and multiplying by the speed c. (for smartphones and tablets); sensors. Actuator outputs include uniform inverse square law and is Knowing the distance or range to three and and support everyday work practice. lamps, valves (water sprinklers), distorted by the building materials or more access points (and using a pro- › the data rate of Wi-Fi products speakers, and a variety of displays— that it travels through. As a result, the cess called multilateration), the mo- has remained competitive and again, industry has many additional position accuracy for Wi-Fi can often bile’s position can be calculated (see has typically had a x advan- needs for bespoke actuators. The have large errors of to meters. Figure ). tage over cellular data, which de ning characteristic of the Internet of interrelationship of the position and function of each of Improvements have been made by In recent times, there have been often has a data limit and results Things (IoT) is that devices are both networked these devices allows us to design systems that are often measuring the RSSI ngerprints for all other radio ranging standards (for ex- in additional cost if exceeded. and embedded in the environment. Unlike tra- called “smart homes” or “smart buildings.” detectable access points at each square ample, ultrawideband [UWB], which ditional networked server technology that has An additional consideration of spatially arranged in- meter of a building, but this requires was standardized in IEEE ..a), Wi-Fi has an additional characteris- veryA little connection with the environment, IoT devices frastructure is that mobile wireless components can move considerable investment in surveys, but none have had much impact on tic in that it has already been improved are designed and positioned so that they can respond to through it. In a smart home, people will be present and the which are labor intensive and have to the mobile industry. Wi-Fi is di erent several times to meet growing user and in uence their surroundings. For example, if a user infrastructure is intended to make their lives easier and be regularly updated. because of its wide-scale use in mobile demands. There is a natural upgrade accesses a web server from an o ce in California, it makes more comfortable. Devices such as lights, HVAC, and se- A more accurate approach was phones, laptops, and tablets through- cycle for Wi-Fi as radio technology very little di erence whether the service is located in the curity systems can be turned on or o by the presence and needed that could localize a device to out the world. And most importantly, improves and as users crave higher next building or situated in Europe. However, if a user ac- detection of people entering or leaving an area. This is not about square meter (the area occu- Wi-Fi access points’ infrastructure data rates to support video and faster cesses an IoT device that is responsible for measuring the only convenient, but also addresses human forgetfulness pied by a person), but only required has grown as a grassroots e ort and downloads. These upgrades also serve temperature of a room, the sensor and device need to be and has cost-saving implications because it reduces en- the knowledge of Wi-Fi access point is deployed ubiquitously, including in as a vector for new features (such as located in that speci c room. ergy consumption. positions. many public spaces. ranging) that can be introduced in As a result, IoT design is a unique departure from in- In the modern smart home, voice-driven assistants The spread of Wi-Fi has been each upgrade cycle. frastructural networking that tries to abstract away the have become popular (such as Amazon Echo and Goo- THE IMPORTANCE OF A enhanced because of three value In short, Wi-Fi .-REVmc is ide- network topology; instead, the topology and position of gle Assistant) and allow devices in smart homes to be STANDARD propositions: ally suited as a standard to disseminate IoT nodes are assets in performing the networking task. activated by casual voice commands. Typically, human In December , the IEEE . a new positioning technology with the IoT devices often contain sensors and actuators. Con- speech is often tersely phrased with the assumption that working group rati ed amendment › it has a practical range of about potential for a wide reach and the op- sumer products for IoT sensors include light sensors, mi- the context is understood. This is challenging for a com- .-REVmc for the Wi-Fi stan- m, and as a result the density portunity for mobile application devel- crophones, cameras, security switches, motion detectors, puter, which doesn’t have the shared context to respond dard, which included a new ranging and related cost of access points opers to build on its novel capabilities.

r8iot.indd 66 7/31/18 11:17 AM r8iot.indd 67 7/31/18 11:17 AM THE IOT CONNECTION

Phone FTM request AP of 10m and is improved to 9m or even 8m, does it make the system more useful? Perhaps not that much. However, Time FTM t 1 if you were to improve it to 1m, it would t 2 Ping cross a significant human threshold because a person typically occu- Ack t 3 distance = c × ((t 4-t1) - (t 3-t 2)) / 2 pies about 1 square meter of standing Pong t 4 space. It opens up the possibility to total round-tripturnaround time time place a person next to another person, FTM: t1, t4 product, or thing. For Wi-Fi RTT, 1 to 2m is just less than the width of most corridors or aisles found in a building. This resolu- Figure 2. The 802.11mc Fine Timing Measurement (FTM) Protocol. tion actually enables in-building navigation, for example, directing a person to a product in a store or an office IEEE 802.11-REVMC AND RTT = (t4 − t1) − (t3 − t2) worker to an unfamiliar conference WI-FI ROUND-TRIP TIME room. Errors two to three times this The core of the 802.11-REVmc proto- The desired range is determined by size would cause the indoor navigation col is built around a new packet type multiplying by the speed of light c and system to place you in the wrong aisle, called a Fine Timing Measurement dividing by 2, as we are only interested resulting in a confusing user experi- (FTM) frame. The recipient responds in the distance from mobile to access ence. In the same way that GPS is accu- to an FTM with an acknowledgement point and not the round-trip distance. rate enough (~5m) to place a car on the packet, and the transmission and re- correct road, in-building navigation ception times of each are recorded. range = c × RTT / 2 is enabled by a 1 to 2m Wi-Fi location It’s basically a ping-pong protocol that system. In combination with indoor calculates ToF by measuring the time Some general assumptions are maps and a routing system calculat- it takes for a packet to be sent from an made that the clocks are running at ing the shortest path, Wi-Fi RTT can access point to a mobile and then back about the same rate on the two de- show your progress along a route and again. This is called the round-trip vices, and with a modern crystal oscil- provide corrections by rerouting when time (RTT), so the protocol is some- lator with an accuracy of 40 parts per you deviate from the path. times referred to as Wi-Fi RTT. million, this is reasonable. Note that In the IoT world, ~1 meter accuracy For a mobile device to measure the the clocks of each device don’t need to can automatically determine the loca- ToF, the precise times of the packet be synchronized as the (t4 − t1) − (t3 − tion of devices such as lights and ther - leaving the access point (t1) and arriv- t2) terms provide time differences for mostats, which might not have their ing at the mobile receiver (t2) must be each separate clock at the beginning exact position recorded. Positioning to known. Further, the precise times the and ending of each exchange, and ~1m can automate the documentation mobile device transmits the response therefore do not require an absolute for an installation process, reducing (t3) and the time it is received back at time reference. Given the typical clock contractor time and removing tran- the access point (t4) must also be re- stability and an 80 MHz bandwidth in scription errors that inevitably occur corded. From the mobile’s perspective, the 5 GHz Wi-Fi band, the expected ac- in hectic construction environments. it measures only t2 and t3 as they occur curacy of this protocol is expected to Further, it can automatically keep re- locally at its radio. However, t1 and t4 be ~1m, which meets the design goals cords up to date as devices are replaced are measured at the access point, so for and considerably outperforms ap- and during remodeling. the mobile to calculate the RTT, the ac- proaches based on RSSI. In daily work practices, users car- cess point needs to send t1 and t4 as an rying mobile devices with Wi-Fi RTT additional message to the mobile (see INCREASED LOCATION can be located in a smart home (using Figure 2), which is piggybacked on the ACCURACY ENABLES NOVEL an application with permission from next FTM packet. APPLICATIONS the owner) and automate the opera- The phone can now calculate the Important questions for the design of tion of heating, lighting, door locks, RTT based on the total time t4 − t1, sub- any positioning system are “What is and security systems. The modern tracting t3 − t2 to account for the turn- the position system going to be used smartphone is a multipurpose device, around time of the acknowledgement for?” and “What target accuracy is and owners typically carry the device packet. needed?” If a system has an accuracy with them most of the time. As home

26 ComputingEdge January 2019 68 COMPUTER WWW.COMPUTER.ORG/COMPUTER

r8iot.indd 68 7/31/18 11:17 AM THE IOT CONNECTION

Phone FTM request AP of 10m and is improved to 9m or even monitoring and control capabilities data, not just position, as it can also be more spatial parallelism in data 8m, does it make the system more use- grow via smartphone apps, we expect converted to user location by a nefari- streams than before and resulting in ful? Perhaps not that much. However, that users will make use of these apps ous party. more efficient use of the communica- Time FTM t 1 if you were to improve it to 1m, it would in their everyday lives. Furthermore, an IoT device might tion channel. Location techniques can t 2 Ping cross a significant human thresh- There are further advantages for have capabilities that can threaten take advantage of this mechanism by old because a person typically occu- accurate location in peer-to-peer in- user privacy when coupled with loca- ranging between an access point and Ack t 3 distance = c × ((t 4-t1) - (t 3-t 2)) / 2 pies about 1 square meter of standing teractions. When two devices are next tion data. For example, a security cam- several stations simultaneously, thus Pong t 4 space. It opens up the possibility to to each other, there is the opportu- era that uploads video data could also supporting a larger number of poten- total round-tripturnaround time time place a person next to another person, nity to share information. For exam- add identity and location information. tial clients. FTM: t1, t4 product, or thing. ple, if a user wishes to share a photo, The European Union’s General An additional technique that sup- For Wi-Fi RTT, 1 to 2m is just less it might be done with a simple flick Protection Data Regulation (GDPR) ports even larger populations of cli- than the width of most corridors or gesture on a screen. A phone that can establishes clear standards for data re- ents is called passive ranging. This aisles found in a building. This resolu- detect the proximity of nearby devices tention, storage, and use. These guide- mechanism is characterized by mo- Figure 2. The 802.11mc Fine Timing Measurement (FTM) Protocol. tion actually enables in-building nav- can limit the scope of gesture-based lines should be treated as a starting bile devices that only need to listen igation, for example, directing a per- sharing to the closest device. You can point for data stewardship. to the transmissions made by other son to a product in a store or an office imagine many other forms of digital stations and never need to transmit IEEE 802.11-REVMC AND RTT = (t4 − t1) − (t3 − t2) worker to an unfamiliar conference peer-to-peer transfers that are made THE FUTURE OF LOCATION themselves. The result is that the WI-FI ROUND-TRIP TIME room. Errors two to three times this possible by automatically and accu- TECHNOLOGIES channel capacity is not affected by the The core of the 802.11-REVmc proto- The desired range is determined by size would cause the indoor navigation rately detecting close proximity, which Going forward, we expect 802.11-REVmc number of users trying to measure col is built around a new packet type multiplying by the speed of light c and system to place you in the wrong aisle, includes ticketing and payments. to be adopted by a wide variety of client their position—furthermore, privacy called a Fine Timing Measurement dividing by 2, as we are only interested resulting in a confusing user experi- devices and access points, which will is guaranteed as a mobile never needs (FTM) frame. The recipient responds in the distance from mobile to access ence. In the same way that GPS is accu- LOCATION DATA AND to an FTM with an acknowledgement point and not the round-trip distance. rate enough (~5m) to place a car on the PRIVACY packet, and the transmission and re- correct road, in-building navigation We have already seen that accurate ception times of each are recorded. range = c × RTT / 2 is enabled by a 1 to 2m Wi-Fi location location can bring many advantages It’s basically a ping-pong protocol that system. In combination with indoor to daily work practice and social in- An IoT device might have capabilities calculates ToF by measuring the time Some general assumptions are maps and a routing system calculat- teractions. However, there are import- that can threaten user privacy when coupled it takes for a packet to be sent from an made that the clocks are running at ing the shortest path, Wi-Fi RTT can ant privacy implications to consider, with location data. access point to a mobile and then back about the same rate on the two de- show your progress along a route and and great care needs to be taken when again. This is called the round-trip vices, and with a modern crystal oscil- provide corrections by rerouting when implementing and enabling these ser- time (RTT), so the protocol is some- lator with an accuracy of 40 parts per you deviate from the path. vices for users. result in many novel location-based to announce its presence. In that sense times referred to as Wi-Fi RTT. million, this is reasonable. Note that In the IoT world, ~1 meter accuracy Mobile devices that can repeatedly applications that can make use of ~1m it’s similar to the operation of GPS, For a mobile device to measure the the clocks of each device don’t need to can automatically determine the loca- link a user’s identity to the device’s lo- indoor accuracy. The designers of the although it differs in the geometric ToF, the precise times of the packet be synchronized as the (t4 − t1) − (t3 − tion of devices such as lights and ther - cation create a location “trail,” which first Wi-Fi RTT protocol had many ad- calculation needed to determine a leaving the access point (t1) and arriv- t2) terms provide time differences for mostats, which might not have their could result from location data being ditional features that they wanted to position. ing at the mobile receiver (t2) must be each separate clock at the beginning exact position recorded. Positioning to stored persistently on the device or up- include, but they had to be deferred In this technique, special types of known. Further, the precise times the and ending of each exchange, and ~1m can automate the documentation loaded to a server. With nefarious ac- due to the timing of the first release. access points called anchor stations mobile device transmits the response therefore do not require an absolute for an installation process, reducing cess to this information over time, it’s A more extensive set of location range to nearby access points. If a mo- (t3) and the time it is received back at time reference. Given the typical clock contractor time and removing tran- straightforward to determine the ha- protocol features are currently be- bile device listens to the protocol ex- the access point (t4) must also be re- stability and an 80 MHz bandwidth in scription errors that inevitably occur bitual whereabouts of the user, which ing undertaken by the follow-up task changes between anchor stations and corded. From the mobile’s perspective, the 5 GHz Wi-Fi band, the expected ac- in hectic construction environments. could be used to tip off thieves or oth- group 802.11az (Next Generation Po- access points, it can use the reported it measures only t2 and t3 as they occur curacy of this protocol is expected to Further, it can automatically keep re- ers with malicious intent. sitioning), whose primary goals are to timestamps to figure out the time dif- locally at its radio. However, t1 and t4 be ~1m, which meets the design goals cords up to date as devices are replaced Protecting the user’s privacy starts enable greater scalability and provide ference of arrival of two packets at its are measured at the access point, so for and considerably outperforms ap- and during remodeling. with an agreement with the user. First, for secure operation. receiver, as if they were sent simulta- the mobile to calculate the RTT, the ac- proaches based on RSSI. In daily work practices, users car- the software must request explicit per- The 802.11az task group is devel- neously from two known positions. cess point needs to send t1 and t4 as an rying mobile devices with Wi-Fi RTT mission to establish the user’s location oping a protocol that takes advantage This information allows a mobile additional message to the mobile (see INCREASED LOCATION can be located in a smart home (using and to store that information. Second, of the new high-efficiency (HE) data to place its position on a hyperbolic Figure 2), which is piggybacked on the ACCURACY ENABLES NOVEL an application with permission from persistent location information must standard being developed in paral- curve. When repeated for multiple next FTM packet. APPLICATIONS the owner) and automate the opera- be carefully protected after being up- lel in Task Group 802.11ax with four pairs of anchor stations and access The phone can now calculate the Important questions for the design of tion of heating, lighting, door locks, loaded to a central server (for exam- times the data throughput of existing points and at known positions, the mo- RTT based on the total time t4 − t1, sub- any positioning system are “What is and security systems. The modern ple, protection against reuse in unex- 802.11ac-based systems. In addition to bile can calculate its position by deter- tracting t3 − t2 to account for the turn- the position system going to be used smartphone is a multipurpose device, pected ways and against observation higher-density coding, the radio sup- mining the intersection point of these around time of the acknowledgement for?” and “What target accuracy is and owners typically carry the device by a rogue employee). This need for ports multi-user uplink MIMO (multi- hyperbolic curves. packet. needed?” If a system has an accuracy with them most of the time. As home protection also applies to raw ranging ple input, multiple output), allowing Authentication and security are

www.computer.org/computingedge 27 68 COMPUTER WWW.COMPUTER.ORG/COMPUTER AUGUST 2018 69

r8iot.indd 68 7/31/18 11:17 AM r8iot.indd 69 7/31/18 11:17 AM THE IOT CONNECTION

always important—even more so when should not be able to calculate location- nding technologies will wireless devices are involved, because the position of A. enable a wide range of novel con- it’s often hard to determine the entity › Infrastructure that determines text-aware services to automate tasks that a mobile device is communicat- the location of a mobile device and support everyday work practice. ing with. For example, when you are in (assuming a user has given it Over the next few years, we expect an unfamiliar location such as a large permission to use that informa- ne-grained location-based services to airport, are you aware of the access tion) should be able to validate become more prevalent and rapidly be- point your smartphone is associated that the mobile is not falsifying come a standard feature used by smart with? .az sets out to address this its position. infrastructure. problem and provides the following mechanisms. Lastly, .az provides for opera- REFERENCES tion in the GHz band, which prom- . P. Bahl and V.N. Padmanabhan, › A mobile device can ensure ises greater range accuracy to a few cen- “RADAR: An In-Building RF-based it’s ranging to a trusted infra- timeters with the increased bandwidth. User Location and Tracking System,” structure. In other words, a In practice, this means position accu- Proc. th Ann. Joint Conf. IEEE Com- mobile device should be able to racy could have times the resolu- puter and Communications Societies reject potentially false range or tion of .-REVmc, which will open (INFOCOM ), , doi:. positioning information from a up new categories of location-based /INFCOM... nearby rogue access point. applications. . P.-REVmc/D., Aug —IEEE › If a mobile device A is ranging to Approved Draft Standard for Informa- nearby access points, it should tion Technology— Telecommunications not be possible for another ocation technology is coming and Information Exchange Between nearby nefarious mobile device of age at a time when IoT tech- Systems—Local and Metropolitan Area B to overhear and decode the Lnologies are proliferating. The Networks— Speci c Requirements Part ranging packets. Therefore, B combination of IoT and accurate : Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Speci- cations, revision of IEEE Std. .- , IEEE, ; https://ieeexplore

.ieee.org/document/¢. EEMliei July–September 2016 IEEE MultiMedia . E. Karapistoli et al., “An Overview of the IEEE ..¤a Standard,” IEEE http://www.computer.org Communications Magazine, vol. ¤,

no. , , pp. ¤¢–. 2016 july–september ❚

Quality Modeling ROY WANT is a research scientist

at Google. Contact him at roywant@ google.com.

WEI WANG is a software engineer

Volume 23 Number 3 at Google. Contact him at weiwa@ google.com.

mult-22-03-c1 Cover-1 July 12, 2016 4:40 PM IEEE MultiMedia serves the community of scholars, developers, STAN CHESNUTT is a software practitioners, and students who are interested in multiple media engineer at Google. Contact him at types and work in fields such as image and video processing, audio email [email protected]. analysis, text retrieval, and data fusion.

Read It Today! Read your subscriptions This article originallythrough the appeared myCS in www.computer.org/multimedia publications portal at Computer, vol. 51, no. 8, 2018. http://mycs.computer.org

28 ComputingEdge January 2019 70 COMPUTER WWW.COMPUTER.ORG/COMPUTER

r8iot.indd 70 7/31/18 11:17 AM SOCIOTECHNICAL SECURITY AND PRIVACY Editors: Heather Richter Lipford, [email protected] | Jessica Staddon, [email protected]

The Last Mile for IoT Privacy

Richard Chow | Intel Corporation

ccording to Mark Weiser:1 dating back to the Fair Information A Practice Principles, states that per- The problem, while often sonal data collection should happen couched in terms of privacy, is only with appropriate notice. There- really one of control. If the com- fore, one of the goals in this article putational system is invisible as is to provide a framework—called well as extensive, it becomes hard the privacy stack—for user commu- to know what is controlling what, nication issues and needs regarding what is connected to what, where IoT privacy (see Figure 1). information is flowing, how it is being used ... and what are the con- The Privacy Stack sequences of any given action. Many design questions, such as discovery, usability, and privacy, Weiser was discussing ubiqui- involve end users. How do users tous computing more than a decade learn about the services in public ago, but he might as well have been spaces, their workplaces, or even talking about the Internet of Things their own homes? How do they (IoT) today. On our computers, we learn about these services’ privacy have at least a semblance of control properties and yet not be over- because we can, in principle, deter- whelmed by the sheer volume of mine what applications are running information? I give a couple moti- and what data they’re collecting. vating examples of IoT privacy For the IoT, traditional methods of notifications. control are largely absent. In fact, Suppose a mall installs a suite there are common cases where of surveillance cameras in its park- people are no longer users of an IoT ing structures for security purposes. service but rather subjects of the Rather than describing the cam- service, such as a smart city sound eras’ purpose on physical signage, monitor. Another example is pub- the mall uses beacons to interact lic Wi-Fi connectivity. A fraction with phones carried by passersby. of people in the vicinity might dis- The beacons emit an ID that can cover and use the service, but not be looked up in the cloud using a being the ones who installed the standard mobile app. The users can actual access points, most people thus be alerted to the surveillance would be unaware of the service’s cameras’ presence, which not only privacy properties. serves as a privacy notification but One of the IoT’s major privacy also provides them with a sense of problems is that users aren’t always security knowing that the area is aware when a device is collecting safe due to the service. personal data. IoT devices’ ubiq- Privacy notifications can also be uitous nature means that a person incorporated into smart home sys- can easily not know when sensors tems. Consider open space voice are present. A basic privacy tenet, controllers such as the Amazon

2469-7087/19/$33.00 © 2019 IEEE Published by the IEEE Computer Society January 2019 29 1540-7993/17/$33.00 © 2017 IEEE Copublished by the IEEE Computer and Reliability Societies November/December 2017 73 SOCIOTECHNICAL SECURITY AND PRIVACY

in a world of ubiquitous services, privacy decisions can be made with minimal cognitive burden. But even for the web, we don’t have privacy metadata standardization for various rea- IoT systems Awareness Inference Preferences Notiﬁcation sons. One interesting approach that sidesteps standardization is to build natural language processing tools for privacy policies (for example, see www.signifiers.io). Nevertheless, the risks of data collection with com- Figure 1. The privacy stack framework bridges from today’s Internet of Things pletely unaware subjects are greater (IoT) systems to users. for the physical world and IoT, which argues for giving a standard language for IoT privacy another try. Echo, which allow voice commands how public or private a space is, to to play music, adjust lighting, and so know how to act and how candidly Inferences on. Guests to the home might not to speak.2 Along these lines, Jason Awareness protocols can enable be comfortable with a whole-home Hong has issued a challenge to the IoT services to declare what they’re voice-recording system. Homeown- pervasive research community: can doing, but what do they declare? It ers might ask permission to use we make it so that when people seems simplest to declare what sen- such a service if their guests indicate enter a room, they can reliably iden- sor data is collected and what it will that they’d like to be notified about tify all of the sensors and dataflows be used for, but this might be insuf- audio recording in their vicinity. within 30 seconds?3 ficient. Users have limited under- Depending on the guests’ prefer- Awareness primarily concerns standing of what might be inferred ences, they could be notified as they how IoT services might open com- or learned from sensor data, and enter the house, or alternatively, munication channels to users and systems will only get better at learn- the system could disable the ser- subjects. These channels might be ing from sensor data. For instance, vice temporarily or partially. All this new visual signifiers of data collec- device or browser fingerprints are might be done silently or explicitly. tion,4 or more traditional network now a fact of life on the web, and In these motivating examples, protocols between the IoT service it’s reasonable to assume that fin- the users don’t understand what and a user’s device. For protocols, gerprints are even more prevalent services are active in the environ- multiple industry efforts, including in data measuring in the physical ment. In the sections that follow, I the Open Connectivity Foundation world. A couple of examples are describe how the examples might (openconnectivity.org), are work- location patterns and, more recently, be realized through the privacy ing on standardizing IoT interop- ambient audio.5 The inference prob- stack shown in Figure 1, which erability, allowing discovery and lem is central to privacy, and yet it’s takes us from today’s IoT protocols communication among devices. unclear how to declare inferences; to human notification. I emphasize Apple’s iBeacon is another exam- for instance, how does the system that this stack isn’t a protocol stack, ple of device discovery (developer handle probabilistic inferences and but simply a conceptual framework. .apple.com/ibeacon). A beacon inferences with auxiliary data? device broadcasts an ID through I propose a balanced approach Awareness Bluetooth, and a compatible device here: users can’t be relied on to The awareness part of the stack pro- can use this ID to retrieve asso- understand the inferences possi- vides for discovery of services’ pri- ciated information (such as for ble from the data collected, so ser- vacy properties by users or users’ location-based services). Beacons vices must explicitly provide basic agents. The difficulty with the IoT is can also place a person in proximity inferences. At the same time, users the potentially unobtrusive nature to devices in the environment. must understand that this set of of data collection. One can, in fact, Thus far, the IoT protocol work inferences is continually growing not realize that data is being col- has not gone into privacy metadata and refined, not only through the lected. According to Judith Don- standardization, a way for services to never-ending growth of user data ath, a fellow at Harvard’s Berkman declare their privacy policies so that but also through the advancement Center for Internet and Society, we they are universally understood. One of machine learning techniques. must design so that we can know advantage of doing so would be that, One way for IoT services to provide

30 ComputingEdge January 2019 74 IEEE Security & Privacy November/December 2017 SOCIOTECHNICAL SECURITY AND PRIVACY

in a world of ubiquitous services, pri- inferences is through the privacy vacy decisions can be made with min- policy; for instance, an IoT service imal cognitive burden. But even for collecting GPS data would declare Internet of Things Privacy Preferences: the web, we don’t have privacy meta- that identity is a possible inference Current Research data standardization for various rea- from the data, and a device collect- IoT systems Awareness Inference Preferences Notiﬁcation sons. One interesting approach that ing energy usage data would declare hat privacy factors are users most concerned about in an Internet of Things (IoT) sidesteps standardization is to build behavioral patterns as a possible W context? Here’s what some recent research has discovered. natural language processing tools inference. Hosub Lee and Alfred Kobsa approached the problem by studying user reactions to for privacy policies (for example, see An understanding of inferences scenarios with varying context elements.1 For each scenario, they chose from a small set of www.signifiers.io). Nevertheless, the wouldn’t just help users under- parameter values in the following categories: who was collecting the data, where the data risks of data collection with com- stand what the system is learn- was being collected, what kind of data was being collected, the reason for collection, and Figure 1. The privacy stack framework bridges from today’s Internet of Things pletely unaware subjects are greater ing about them but would also the persistence of collection. An example scenario had “government” for who was collect- (IoT) systems to users. for the physical world and IoT, which help systems protect privacy by ing and “safety” for the reason for collecting. They found clusters in this context space that argues for giving a standard language translating user preferences. For predicted user reaction. for IoT privacy another try. instance, a system might collect Reuben Binns and his colleagues pointed out that privacy decision making is influ- Echo, which allow voice commands how public or private a space is, to video data, timestamps, and Wi-Fi enced by particular attributes, such as the reputation and size, of the company doing the to play music, adjust lighting, and so know how to act and how candidly Inferences data. A user might be comfortable collecting.2 on. Guests to the home might not to speak.2 Along these lines, Jason Awareness protocols can enable with sharing video data collected Linda Lee and her colleagues investigated the “what” and “who” factors for wearable be comfortable with a whole-home Hong has issued a challenge to the IoT services to declare what they’re at work but not video data col- devices; the “what” had more effect than the “who” on user reactions, and users were most voice-recording system. Homeown- pervasive research community: can doing, but what do they declare? It lected at home. Security policies concerned about the collection of video or photos.3 ers might ask permission to use we make it so that when people seems simplest to declare what sen- for the raw data itself can be imple- Pardis Emami Naeini and her colleagues provided more statistical evidence for the such a service if their guests indicate enter a room, they can reliably iden- sor data is collected and what it will mented, but only after the system impact of various factors, such as what and where, and also advanced explanations for the that they’d like to be notified about tify all of the sensors and dataflows be used for, but this might be insuf- uses the data to infer “work” ver- impact, such as perceived benefit.4 audio recording in their vicinity. within 30 seconds?3 ficient. Users have limited under- sus “home.” This highlights the gap Depending on the guests’ prefer- Awareness primarily concerns standing of what might be inferred between low-level raw data and the References ences, they could be notified as they how IoT services might open com- or learned from sensor data, and language of human preferences, 1. H. Lee and A. Kobsa, “Privacy Preference Modeling and Prediction in a Simulated Cam- enter the house, or alternatively, munication channels to users and systems will only get better at learn- which consists of higher-level puswide IoT Environment,” Proc. IEEE Int’l Conf. Pervasive Computing and Communications the system could disable the ser- subjects. These channels might be ing from sensor data. For instance, concepts such as “home.” Sensors (PerCom 17), 2017, pp. 276–285. vice temporarily or partially. All this new visual signifiers of data collec- device or browser fingerprints are operate at a different level from 2. R. Binns et al., “My Bank Already Gets This Data: Exposure Minimisation and Com- might be done silently or explicitly. tion,4 or more traditional network now a fact of life on the web, and what users find meaningful, and pany Relationships in Privacy Decision Making,” Proc. CHI Conf. Extended Abstracts In these motivating examples, protocols between the IoT service it’s reasonable to assume that fin- inferences help bridge the gap to on Human Factors in Computing Systems (CHI EA 17), 2017, pp. 2403–2409; doi.acm the users don’t understand what and a user’s device. For protocols, gerprints are even more prevalent usable systems. .org/10.1145/3027063.3053255. services are active in the environ- multiple industry efforts, including in data measuring in the physical 3. L. Lee et al., “Information Disclosure Concerns in the Age of Wearable Comput- ment. In the sections that follow, I the Open Connectivity Foundation world. A couple of examples are Privacy Preferences ing,” Proc. NDSS Workshop Usable Security (USEC 16), 2016; dx.doi.org/10.14722 describe how the examples might (openconnectivity.org), are work- location patterns and, more recently, Moving on to the stack’s next layer, /usec.2016.23006. be realized through the privacy ing on standardizing IoT interop- ambient audio.5 The inference prob- suppose the system has a good 4. P. Emami Naeini et al., “Privacy Expectations and Preferences in an IoT World,” Proc. stack shown in Figure 1, which erability, allowing discovery and lem is central to privacy, and yet it’s understanding of what data an IoT 13th Symp. Usable Privacy and Security (SOUPS 17), 2017; www.usenix.org/conference takes us from today’s IoT protocols communication among devices. unclear how to declare inferences; service is collecting and what infer- /soups2017/technical-sessions/presentation/naeini. to human notification. I emphasize Apple’s iBeacon is another exam- for instance, how does the system ences might be possible. What does that this stack isn’t a protocol stack, ple of device discovery (developer handle probabilistic inferences and the user or subject actually care but simply a conceptual framework. .apple.com/ibeacon). A beacon inferences with auxiliary data? about? Whether a particular IoT device broadcasts an ID through I propose a balanced approach data collection scenario is consid- Because so many factors might particular factors might be impor- Awareness Bluetooth, and a compatible device here: users can’t be relied on to ered privacy sensitive depends on influence a privacy decision, the tant to some users can also influence The awareness part of the stack pro- can use this ID to retrieve asso- understand the inferences possi- the individual, but context is also central question is: what factors system design. For instance, an IoT vides for discovery of services’ pri- ciated information (such as for ble from the data collected, so ser- critical. For instance, audio record- do people really care about in an service at a bar might set patrons at vacy properties by users or users’ location-based services). Beacons vices must explicitly provide basic ing might be fine in a restaurant IoT context? The sidebar provides ease by informing them that video agents. The difficulty with the IoT is can also place a person in proximity inferences. At the same time, users depending on who’s doing the a sampling of what the current is being captured by the bar for the potentially unobtrusive nature to devices in the environment. must understand that this set of recording (restaurant or friend) and research has found. age-verification purposes only. of data collection. One can, in fact, Thus far, the IoT protocol work inferences is continually growing who you’re with (work or social). This research is just beginning, not realize that data is being col- has not gone into privacy metadata and refined, not only through the Context also gives clues as to what’s but the vision is that an under- Notification lected. According to Judith Don- standardization, a way for services to never-ending growth of user data surprising and what’s expected. standing of the relevant factors will The stack’s last layer involves noti- ath, a fellow at Harvard’s Berkman declare their privacy policies so that but also through the advancement Video surveillance in a football sta- encourage system creators to pre- fication of the end user. This noti- Center for Internet and Society, we they are universally understood. One of machine learning techniques. dium might be unremarkable, but sent this information to those mak- fication depends on awareness of must design so that we can know advantage of doing so would be that, One way for IoT services to provide perhaps not at a restaurant. ing privacy decisions. Knowing that the services in the environment

www.computer.org/computingedge 31 74 IEEE Security & Privacy November/December 2017 www.computer.org/security 75 SOCIOTECHNICAL SECURITY AND PRIVACY

and considers inferences and user from the data. The proxy man- 2. “Five Minutes with Judith Don- preferences. ages personalized privacy prefer- ath,” The MIT Press, 30 July Notification is envisioned to be a ences, for instance, allowing users 2014; mitpress.mit.edu/blog/five relatively rare event, as users won’t to declare preferences, or inferring -minutes-judith-donath. want to interact with most IoT them from past privacy decisions 3. J. Hong, “The Privacy Landscape services when they become ubiq- or demographics. A privacy proxy for Pervasive and Ubiquitous Com- uitous; even services that merit a builds and maintains a user’s pri- puting,” IEEE Pervasive Computing, notification the first time might vacy preferences and decisions, vol. 16, no. 3, 2017, pp. 40–48. not the second or third time. For and thus the proxy can use intel- 4. S. Zimmeck et al., “Automated instance, drivers might want to ligence to alert the user only when Analysis of Privacy Requirements be alerted when their car enters a appropriate. for Mobile Apps,” Proc. 24th Net- region where it might be tracked but work and Distributed System Secu- won’t want to be alerted every day rity Symp. (NDSS 17), 2017; dx.doi for the same street camera. basic privacy principle is .org/10.14722/ndss.2017.23034. Central questions for notifica- A that personal data collection 5. N. Karapanos et al., “Sound-Proof: tions include: How does a system should happen only with appropri- Usable Two-Factor Authentication present alerts, how does a user ate notice and choice. However, this Based on Ambient Sound,” Proc. express alert preferences, and how principle’s implementation—already 24th USENIX Conf. Security Symp. are these preferences learned? One difficult for traditional clients— (SEC 15), 2015, pp. 483–498. notable point is that an alert that’s is even more difficult for the IoT 6. E. Wang and R. Chow, “What Can I not privately communicated could because there’s no natural com- Do Here? IoT Service Discovery in allow others to draw undesirable munication channel with users. In Smart Cities,” Proc. IEEE Int’l Conf. inferences about a user’s history or many cases, users might find them- Pervasive Computing and Commu- preferences. Also, technology and selves in environments where ser- nication Workshops (PerCom 16), inference algorithms might improve vices are running and capturing 2016, pp. 1–6; dx.doi.org/10.1109 enough such that people who were their data, even though the users /PERCOMW.2016.7457097. previously notified about a service haven’t installed the services and 7. “Personalized Privacy Assistant might need to be notified again. aren’t aware that they’re running. Project,” Carnegie Mellon Univer- Notification represents the last A conceptual privacy stack sity, 2017; www.privacyassistant.org. layer of the privacy stack: actual describes how IoT systems being interaction with users. But other developed today interact with Richard Chow is a university re- forms of interaction are possible. users. This stack is being actively search manager and scientist at Notification implies interrupting researched in multiple areas. One Intel Corporation. Contact him a user involved in another activ- area is the user interface, both for at [email protected]. ity, but a more active user might be setting and learning privacy pref- interested in visualizing the privacy erences and for notification and properties for nearby IoT services. visualization. A second area is the How does a user visualize nearby inference problem for IoT data— IoT services and their privacy and how to translate raw sensor data trust properties? into human-understandable inferences. A third area of research is the Implementing the Stack privacy schema and how to repre- To illustrate, I describe how the sent the plethora of IoT services stack might be implemented in the and their privacy policies, which is commonly proposed privacy proxy critical for automatic filtering. This or privacy assistant on a user’s is an exciting and active area, com- device.6,7 For awareness, the pri- bining machine learning, human– vacy proxy manages protocols that computer interaction, and privacy. pull (or are pushed) privacy meta- This article originally data from nearby IoT services. Basic References appeared in inferences might be included in the 1. M. Wieser et al., “The Origins of IEEE Security & Privacy, privacy metadata, and the privacy Ubiquitous Computing Research at vol. 15, no. 6, 2017. proxy might also have the intelli- PARC in the Late 1980s,” IBM Syst. gence to make its own inferences J., vol. 38, no. 4, 1999, pp. 693–696.

32 ComputingEdge January 2019 76 IEEE Security & Privacy November/December 2017

DEPARTMENT: IT ECONOMICS

5G in E-Commerce Activities

5G cellular services are expected to be commercially available in developed countries later this year. These Nir Kshetri services, on their own or in combination with other University of North Carolina technologies, are likely to have a transformative impact on at Greensboro e-commerce activities. The development of 5G networks, Editor: platforms, and devices certainly require high investments. Nir Kshetri, University of There are, however, important economic benefits as well as North Carolina at psychological or intangible benefits associated with 5G. Greensboro; The faster speed of 5G networks and high-resolution [email protected] screens of 5G-enabled devices might lead to a higher degree of customer willingness to engage in e-commerce activities, more time spent on e-commerce websites, and more purchases online. The features of 5G can also lead to a higher degree of effectiveness of e- commerce vendors’ activities such as online advertising. Finally, faster speeds and higher- resolution screens are also associated with a higher degree of enjoyment (psychological or intangible benefits) when consumers engage in e-commerce activities. Consider online video ads, which are a key component of an e-commerce ecosystem. In 2017, social video ad spending in the US was estimated at over $4 billion dollars or 20 percent of the total social media ad sales.1 Most video ad tags contain sophisticated tracking codes, making it possible for advertisers to track users’ interaction with the ads.2 Due to their larger file sizes, loading video ads is more difficult and time consuming than loading text or photos. This is a major concern because many viewers strongly dislike ads and other content that do not download quickly. In a survey conducted by Adobe in December 2017, about 80 percent of respondents said that if content takes too long to download, they will stop viewing it or switch to a different device. Likewise, another study conducted in 2017 by video analytics firm Mux found that 85 percent of respondents would stop watching a video if it takes too long to load.2 5G offers great promise and potential to address these challenges. With 5G, data transmission and processing speeds will rocket to new levels. 5G will thus help create and deliver effective online video advertising that can attract customers’ attention and produce the best results. Likewise, 5G networks ensure that devices do not lose Internet connections when traveling from one location to another.

5G-RELATED PROGRESS SO FAR AND FUTURE PROSPECTS The 5G technology standard was finalized in December 2017, and saw its first deployment in the 2018 Winter Olympics in South Korea. An estimate by the cellphone trade group GSMA suggests that there will be 1.2 billion 5G connections worldwide by 2025.3

Asia and the US are expected to take the lead in 5G, whereas Europe is reported to be lagging.4 The first commercial 5G projects are expected to be launched in the US in 2018.5 Japan and South Korea are expected to launch 5G in 2019 and China in 2020. South Korean cellular company KT’s peak 5G network speeds were up to 3.5 gigabits per second (GBps) on Samsung tablets.6 It is worth noting that, in January 2018, South Korea’s average mobile broadband download speed (which was the fourth highest in the world) was 133.05 megabits per second (Mbps). This means that peak speeds in 5G networks during the 2018 Winter Olympics were more than 26 times the country’s mobile broadband download speed. During the 2018 Olympics, KT provided 5G networks, Samsung provided 5G tablets, and Intel provided 5G platforms.7 Samsung’s 5G-equipped tablets allowed viewers to switch among multiple cameras placed along the cross-country skiing route so they could track an athlete’s real-time location on a 3D map of the entire skiing course. Many other companies are developing products and services that would help enrich the global 5G ecosystem. China-based tech companies ZTE and Huawei announced that they would release 5G smartphones by early 2019.

COMBINING 5G WITH OTHER TECHNOLOGIES TO STIMULATE E-COMMERCE 5G—in combination with other technologies such as artificial intelligence (AI), Internet of Things (IoT), blockchain, augmented reality (AR), and virtual reality (VR)—is likely to be a transformative force in the e-commerce industry and market. Table 1 highlights some examples and benefits of combining 5G with other technologies to enhance e-commerce.

Table 1. Combining 5G with other technologies to enhance e-commerce.

Technology Examples of uses Benefits of combining with 5G Internet of Improve consumer experiences, Will make it easier to transfer data Things track inventory in real time, created by IoT devices (IoT) manage orders more effectively Artificial Order products online, track Will make it possible to access intelligence orders, and perform other e- additional information quicker and (AI) commerce activities understand the environment and context better Blockchain Smart contracts can be used by Will address security issues and online vendors to automate order feed the information (for example, fulfillment; supply-chain from IoT devices) required for a management; B2B e-commerce smart contract more efficiently Augmented AR-enabled apps allow a 5G networks have higher reality (AR) potential customer to virtually bandwidth, reduced latency, and a and virtual place real products in a real higher degree of uniformity to deal reality (VR) setting to provide a clear with the complex worlds and visualization of the products’ use. sophisticated inputs that require processing huge amounts of data.

Internet of Things A study found that 70 percent of retailers worldwide were ready to adopt IoT to improve consumer experiences.8 There are a number of ways in which e-commerce activities can benefit from IoT. For instance, IoT makes it easy to track inventory in real time and manage it more effectively. By doing so, human errors can be reduced. IoT can also help minimize waste, control

+VMZ"VHVTU34 ComputingEdge XXXDPNQVUFSPSHJUQSPJanuary 2019

IT PROFESSIONAL IT ECONOMICS

4 Asia and the US are expected to take the lead in 5G, whereas Europe is reported to be lagging. costs, and reduce shortage. For instance, temperature-monitoring sensors can be used to maintain 5 The first commercial 5G projects are expected to be launched in the US in 2018. Japan and optimal temperatures for perishable products and send alerts when certain conditions are met. South Korea are expected to launch 5G in 2019 and China in 2020. South Korean cellular company KT’s peak 5G network speeds were up to 3.5 gigabits per second (GBps) on Samsung Unsurprisingly, e-commerce companies have made heavy investments in IoT. China-based e- tablets.6 It is worth noting that, in January 2018, South Korea’s average mobile broadband retailer JD.com’s 3-System Fridge has sensors on every shelf and an internal camera. It registers download speed (which was the fourth highest in the world) was 133.05 megabits per second the time and date when items are stored inside. The data is fed to a smart screen on the fridge’s (Mbps). This means that peak speeds in 5G networks during the 2018 Winter Olympics were front side, which sends an alert when an expiration date is near. It can also order the next grocery 9 more than 26 times the country’s mobile broadband download speed. list from JD.com based on the fridge’s contents. During the 2018 Olympics, KT provided 5G networks, Samsung provided 5G tablets, and Intel Vast amounts of unstructured data are created by IoT devices, and the amount of data created is 10 provided 5G platforms.7 Samsung’s 5G-equipped tablets allowed viewers to switch among growing twice as fast as the available bandwidth. It is estimated that by 2020, a network 11 multiple cameras placed along the cross-country skiing route so they could track an athlete’s capacity that is at least 1,000 times the level of 2016 will be needed. The amount of real-time location on a 3D map of the entire skiing course. Many other companies are developing communication that needs to be handled will also increase costs exponentially. The current 4G products and services that would help enrich the global 5G ecosystem. China-based tech networks are not capable of handling this growth. 5G networks can play a role in addressing the companies ZTE and Huawei announced that they would release 5G smartphones by early 2019. potential bandwidth deficit.

COMBINING 5G WITH OTHER TECHNOLOGIES TO Virtual and Augmented Reality STIMULATE E-COMMERCE VR and AR are likely to emerge as driving forces in the e-commerce industry and market. By wearing a VR headset, a shopper can instantly find herself in a company’s virtual shop, where 5G—in combination with other technologies such as artificial intelligence (AI), Internet of she can “walk” around to explore items exactly as she would in the real shop. For instance, if she Things (IoT), blockchain, augmented reality (AR), and virtual reality (VR)—is likely to be a wants to know more about a new piece of jewelry in the shop, she can focus her sight on that transformative force in the e-commerce industry and market. Table 1 highlights some examples item and see the relevant information needed to make a purchasing decision. If she wants to buy and benefits of combining 5G with other technologies to enhance e-commerce. it, she can make the payment or add it to her cart and look for additional items.12 AR applies VR in the real world with live video imagery.13 For example, a furniture vendor can Table 1. Combining 5G with other technologies to enhance e-commerce. develop an AR-enabled app that allows a customer to point the mobile camera to the place where she wants to keep the furniture. The app places a 3D model of the furniture as an overlay on the Technology Examples of uses Benefits of combining with 5G living room’s live image. The shopper now has a clear visualization of how the furniture would fit in her living room. In addition, by rotating the camera, she can see where the new furniture Internet of Improve consumer experiences, Will make it easier to transfer data 12 14 Things track inventory in real time, created by IoT devices fits best. This has already been done by some companies such as Lego and IKEA. Likewise, the home furnishings and decor company Wayfair launched its AR app WayfairView in 2016, (IoT) manage orders more effectively which is available in Google Play. Artificial Order products online, track Will make it possible to access The complexity and richness of the AR and VR worlds require processing a large quantity of intelligence orders, and perform other e- additional information quicker and data. Current 4G networks standards suffer from some limitations such as those related to (AI) commerce activities understand the environment and bandwidth, latency, and uniformity, especially when the data needs to be fed remotely. In this context better regard, 5G is likely to unlock the full potential of VR and AR technologies. 5G’s significantly 15 Blockchain Smart contracts can be used by Will address security issues and faster speeds and lower latency would help overcome these weaknesses. 5G streams’ transmission delay is about 1 millisecond, which is much shorter than human beings can notice. online vendors to automate order feed the information (for example, 5G’s almost zero delay in transmission is thus likely to enrich customers’ experiences with AR fulfillment; supply-chain from IoT devices) required for a and VR technologies. management; B2B e-commerce smart contract more efficiently According to TimeTrade research, 85 percent of consumers prefer shopping in physical stores. Augmented AR-enabled apps allow a 5G networks have higher Many features of brick and mortar stores cannot be replaced by e-commerce.16 However, AR and reality (AR) potential customer to virtually bandwidth, reduced latency, and a VR can create consumer experiences that are close enough to brick and mortar stores. and virtual place real products in a real higher degree of uniformity to deal reality (VR) setting to provide a clear with the complex worlds and Retailers can also use AR and VR in concept testing to measure consumers’ responses and visualization of the products’ use. sophisticated inputs that require feedback before implementing their ideas. Such testing can provide retailers with insights into processing huge amounts of data. consumers’ emotional connection with the products, which can result in consumer adoption and market impact.17

Internet of Things Blockchain A study found that 70 percent of retailers worldwide were ready to adopt IoT to improve Sophisticated applications of blockchain to facilitate e-commerce activities have been or are 8 consumer experiences. There are a number of ways in which e-commerce activities can benefit being developed. JD.com has implemented blockchain in its supply-chain management system from IoT. For instance, IoT makes it easy to track inventory in real time and manage it more and B2B e commerce. In 2017, the system went live with beef manufacturer Kerchin as its first effectively. By doing so, human errors can be reduced. IoT can also help minimize waste, control supply-chain partner. The company announced that JD.com would have more than 10 brands of alcohol, food,Ǧ tea, and pharmaceutical products on its blockchain by the end of October 2017.18

+VMZ"VHVTU XXXDPNQVUFSPSHJUQSP +VMZ"VHVTUwww.computer.org/computingedge XXXDPNQVUFSPSHJUQSP35

IT PROFESSIONAL

One of the most high-profile future uses of blockchain is likely to be smart contracts. Online vendors can use smart contracts to automate fulfillment of orders for the delivery of digital products. In smart contracts that are executed “above” the blockchain, 5G can play a key role in feeding the information (for example, from IoT devices) more efficiently. Cyberattacks have been a main concern for the growth of e-commerce. Such concerns are further heightened by the rapid growth of IoT networks, as IoT sensors carry sensitive information about their users. Securing these systems is thus important. Another key use of blockchain would be addressing security issues in 5G. Telecom companies can provide an eSIM (embedded SIM) or an app to a subscriber that creates a unique virtual identity that is encrypted and stored on a blockchain. Subscribers can use the unique IDs for automatic authentication on e-commerce websites. Blockchain also allows for secure P2P network solutions.

AI AI-enabled devices are already playing important roles in helping consumers in e-commerce activities such as making buying decisions and tracking products. For instance, virtual assistants are transforming the way consumers purchase products online. Amazon’s personal assistant Alexa, for example, has been integrated into Amazon products as well as those from other manufacturers. Customers can use Alexa to find information about local concerts through eBay’s online ticket exchange company StubHub. In addition, they can arrange transportation to the event via Uber and order pre-event dinner from Domino’s. The order status can be tracked in real time.19 Likewise, consumers can order flower arrangements with 1-800-Flowers.com through Alexa or Google assistant-enabled devices.20 AI-enabled devices have already started handling unstructured information. For instance, social media platforms use AI to enable facial recognition and photo tagging.21 5G will dramatically improve consumer experiences with AI-based devices. With 5G, AI-based devices can access additional structured and unstructured information quicker and understand the environment and context better. Overall, AI-powered services will be more reliable in a wide range of contexts and situations in which they operate.

CONCLUSION Some of the current challenges in the development of e-commerce can be overcome with the deployment of 5G networks, such as with IoT devices. Addressing the exponential growth in IoT devices will be no small feat. Current 4G networks, however, cannot handle all the data coming from IoT devices. 5G’s higher data transmission and processing speeds will address this concern. Specifically, 5G (in combination with AI, VR, AR, and other technologies) will play a powerful role in transforming the e-commerce industry and market. Such a combination can result in a rich e-commerce ecosystem and a better customer experience.

REFERENCES 1. “Social Video Ad Spend Will Double in 2017,” BI Intelligence, 3 April 2017; http://uk.businessinsider.com/social-video-ad-spend-will-double-in-2017-2017- 4?r=US&IR=T. 2. R. Benes, “Many Users Will Stop Viewing Content That’s Slow to Load,” eMarketer, 9 February 2018; www.emarketer.com/content/half-of-u-s-users-stop-viewing-content- slow-to-load. 3. “10 Things We Learned at Mobile World Congress,” Rakuten.Today, 2 March 2018; https://rakuten.today/blog/10-things-learned-mobile-world-congress-2018.html. 4. A. Kharpal, “China 'Has the Edge' in the War for 5G and the US and Europe Could Fall Behind,” CNBC, 7 March 2018; www.cnbc.com/2018/03/07/china-has-the-edge- in-the-war-for-5g-us-and-eu-could-fall-behind.html.

+VMZ"VHVTU36 ComputingEdge XXXDPNQVUFSPSHJUQSPJanuary 2019

IT PROFESSIONAL IT ECONOMICS

One of the most high-profile future uses of blockchain is likely to be smart contracts. Online 5. E. Auchard, “Faster 5G Connections Still Far Off for Most Mobile Users,” CRN, 4 vendors can use smart contracts to automate fulfillment of orders for the delivery of digital March 2018; www.crn.com.au/news/faster-5g-connections-still-far-off-for-most- products. In smart contracts that are executed “above” the blockchain, 5G can play a key role in mobile-users-486257. feeding the information (for example, from IoT devices) more efficiently. 6. M. Dano, “KT’s Millimeter Wave 5G Network Transmitted 3,800 TB of Data during Winter Olympics,” FierceWireless, 25 February 2018; www.fiercewireless.com/5g/kt- Cyberattacks have been a main concern for the growth of e-commerce. Such concerns are further s-millimeter-wave-5g-network-transmitted-3-800-tb-data-during-winter-olympics. heightened by the rapid growth of IoT networks, as IoT sensors carry sensitive information about 7. E. Ramirez, “In the Race for 5G, South Korea Shows Off its Tech Prowess at the their users. Securing these systems is thus important. Another key use of blockchain would be Winter Olympics,” Forbes, 23 February 2018; addressing security issues in 5G. Telecom companies can provide an eSIM (embedded SIM) or www.forbes.com/sites/elaineramirez/2018/02/23/in-the-race-for-5g-south-korea- an app to a subscriber that creates a unique virtual identity that is encrypted and stored on a shows-off-its-tech-prowess-at-the-winter-olympics/#4c1384762853. blockchain. Subscribers can use the unique IDs for automatic authentication on e-commerce 8. N. Dave, “4 Major Ways in Which e-Commerce is Benefitting from IoT,” Business 2 websites. Blockchain also allows for secure P2P network solutions. Community, 15 November 2017; www.business2community.com/ecommerce/4-major- ways-e-commerce-benefitting-iot-01957768. 9. E. Lye, “Five Chinese Brands That Will Be Household Names Next Year,” The Drum, AI 14 June 2016; www.thedrum.com/opinion/2016/06/14/five-chinese-brands-will-be- household-names-next-year. AI-enabled devices are already playing important roles in helping consumers in e-commerce 10. N. Desai, “IBM Scientists Team with the Weather Company to Bring Edge Computing activities such as making buying decisions and tracking products. For instance, virtual assistants to Life,” IBM Research, 15 February 2017; are transforming the way consumers purchase products online. Amazon’s personal assistant www.ibm.com/blogs/research/2017/02/bringing-edge-computing-to-life. Alexa, for example, has been integrated into Amazon products as well as those from other 11. S. Waterman, “Industry to Government: Hands Off IoT Security,” FedScoop, 17 manufacturers. Customers can use Alexa to find information about local concerts through eBay’s November 2016; www.fedscoop.com/industry-to-government-hands-off-iot-security. online ticket exchange company StubHub. In addition, they can arrange transportation to the 12. “Retail e-commerce: How AR and VR Can Transform Shopping,” CIO, 2018; event via Uber and order pre-event dinner from Domino’s. The order status can be tracked in real www.cio.in/feature/retail-e-commerce-how-ar-and-vr-can-transform-shopping-0. time.19 Likewise, consumers can order flower arrangements with 1-800-Flowers.com through 13. T. Perdue, “Applications of Augmented Reality,” Lifewire, 7 July 2017; Alexa or Google assistant-enabled devices.20 AI-enabled devices have already started handling www.lifewire.com/applications-of-augmented-reality-2495561. unstructured information. For instance, social media platforms use AI to enable facial 14. “11 Creative Uses of Augmented Reality in Marketing and Advertising,” Forbes, 13 June 2017; www.forbes.com/sites/forbesagencycouncil/2017/06/13/11-creative-uses- recognition and photo tagging.21 of-augmented-reality-in-marketing-and-advertising/#268a5a3b6b12. 5G will dramatically improve consumer experiences with AI-based devices. With 5G, AI-based 15. J. Mundy, “How Will 5G Boost VR and AR?,” 5G.co.uk, 2018; devices can access additional structured and unstructured information quicker and understand the https://5g.co.uk/guides/5g-virtual-reality-and-augmented-reality. environment and context better. Overall, AI-powered services will be more reliable in a wide 16. T. Bragg, “Is IoT Technology Transforming Brick and Mortar Retail?,” TechWire range of contexts and situations in which they operate. Asia, 13 November 2017; http://techwireasia.com/2017/11/iot-technology- transforming-brick-mortar-retail. 17. S. Tamang, “Will 5G Truly Transform Our Lives?,” Enterprise Innovation, 30 May 2018; www.enterpriseinnovation.net/article/will-5g-truly-transform-our-lives- CONCLUSION 1244626393. Some of the current challenges in the development of e-commerce can be overcome with the 18. E. Xiao, “Alibaba, JD Tackle China’s Fake Goods Problem with Blockchain,” Tech in deployment of 5G networks, such as with IoT devices. Addressing the exponential growth in IoT Asia, 16 October 2017; www.techinasia.com/alibaba-jd-ecommerce-giants-fight-fake- devices will be no small feat. Current 4G networks, however, cannot handle all the data coming goods-blockchain. from IoT devices. 5G’s higher data transmission and processing speeds will address this concern. 19. D. Asling, “19 Powerful Ways to Use Artificial Intelligence in Ecommerce,” Linnworks Blog, 2018; http://blog.linnworks.com/artificial-intelligence-in-ecommerce. Specifically, 5G (in combination with AI, VR, AR, and other technologies) will play a powerful 20. J. Renfrow, “1-800-Flowers.com Offers AI-Powered Shopping for Valentine’s Day,” role in transforming the e-commerce industry and market. Such a combination can result in a FierceRetail, 1 February 2018; www.fierceretail.com/digital/1-800-flowers-adds-ai- rich e-commerce ecosystem and a better customer experience. powered-shopping-for-valentine-s-day. 21. T. Harbet, “How 5G Will Make AI-Powered Devices Smarter,” IQ Intel, 21 February 2018; https://iq.intel.com/how-5g-will-make-ai-powered-devices-smarter.

REFERENCES 1. “Social Video Ad Spend Will Double in 2017,” BI Intelligence, 3 April 2017; http://uk.businessinsider.com/social-video-ad-spend-will-double-in-2017-2017- 4?r=US&IR=T. ABOUT THE AUTHOR 2. R. Benes, “Many Users Will Stop Viewing Content That’s Slow to Load,” eMarketer, Nir Kshetri is a professor of management at the Bryan School of Business and Economics 9 February 2018; www.emarketer.com/content/half-of-u-s-users-stop-viewing-content- at the University of North Carolina at Greensboro. Contact him at [email protected]. slow-to-load. 3. “10 Things We Learned at Mobile World Congress,” Rakuten.Today, 2 March 2018; https://rakuten.today/blog/10-things-learned-mobile-world-congress-2018.html. 4. A. Kharpal, “China 'Has the Edge' in the War for 5G and the US and Europe Could Fall Behind,” CNBC, 7 March 2018; www.cnbc.com/2018/03/07/china-has-the-edge- This article originally appeared in in-the-war-for-5g-us-and-eu-could-fall-behind.html. IT Professional, vol. 20, no. 4, 2018.

+VMZ"VHVTU XXXDPNQVUFSPSHJUQSP +VMZ"VHVTUwww.computer.org/computingedge XXXDPNQVUFSPSHJUQSP37

DEPARTMENT: Internet of Things, People, and Processes

A New Era for Web AR with Mobile Edge Computing

Xiuquan Qiao Dedicated device-based and app-based augmented State Key Laboratory of Networking and Switching reality (AR) solutions have inherent limitations Technology, Beijing regarding cross-platform, pervasive AR application University of Posts and Telecommunications provisioning. Web-based AR (web AR), a promising lightweight and cross-platform approach to AR, is Pei Ren Beijing University of Posts gaining increasing attention owing to its extensive and Telecommunications application areas. However, for computationally Schahram Dustdar intensive AR applications, the weak computational TU Wien efficiency of current web browsers seriously hampers Junliang Chen Beijing University of Posts applications of web AR on a large scale. The and Telecommunications “browser + cloud” approach suffers from the high- latency dilemma. Now, with the emerging 5G networks, mobile edge computing (MEC) promises to greatly reduce network latency (even to 1 ms) by deploying applications at the network edge closer to users, which provides an opportunity for performance improvement of web AR. In this article, the authors envision that the application of MEC has the potential to bring web AR into a new era. Specifically, an MEC-oriented web AR solution is first proposed, followed by the design and deployment details. The authors also discuss future directions aimed at using MEC to tackle the performance issues of web AR in 3G, 4G, and 5G networks.

Mobile augmented reality (MAR)1 nowadays greatly attracts both academic and industrial attention, owing to its extensive application areas and the popularity of smartphones and wearable devices. Recently, many MAR use cases have already emerged in the fields of art, education, entertainment, and so on. These use cases typically are divided into two types: wearable-device- based and app-based MAR solutions. However, both approaches suffer from some inherent limitations regarding cross-platform, portability, and cost aspects:

IEEE Internet Computing Published by the IEEE Computer Society July/August 2018 1089-7801/18/$33.00 USD ©2018 IEEE 38 January 2019 Published46 by the IEEE Computer Society 2469-7087/19/$33.00 © 2019 IEEE IEEE INTERNET COMPUTING

DEPARTMENT: Internet of Things, People, and Processes • Wearable-device-based MAR solutions are costly and inconvenient to carry. • App-based MAR solutions require downloading and installing the specific app in advance and lack cross-platform support (i.e., an AR activity of one app cannot be used in other apps directly).

Therefore, it’s necessary to explore a lightweight, low-cost, and cross-platform pervasive MAR solution for mobile users. Owing to web-based AR’s cross-platform and no-installation advantages, it is attracting more and more attention and is opening up a new research field of A New Era for Web AR MAR. Currently, the proposed web AR implementations can be classified primarily as pure front-end solutions using JavaScript at a mobile Web browser, browser-kernel-based extension solutions, with Mobile Edge or “browser + cloud” solutions. However, all these web AR solutions are also struggling with some problems:

• Pure front-end solutions. MAR applications often rely on computationally intensive Computing computer vision algorithms. However, JavaScript cannot provide efficient computational capability for complex matrix computations. Therefore, pure front-end solutions can only use several typical algorithms to recognize some simple markers, and have poor recognition and tracking ability for nature images and real material objects. • Browser-kernel-based extension solutions. To more fully utilize the computational resources of smartphones than the pure front-end solutions, some web browser providers Xiuquan Qiao Dedicated device-based and app-based augmented State Key Laboratory of try to extend the browser kernel to enable web AR functions, even though this approach Networking and Switching reality (AR) solutions have inherent limitations still faces the challenges of cross-platform requirements owing to the lack of web AR standardization between different web browsers. In addition, if the application scenario Technology, Beijing regarding cross-platform, pervasive AR application University of Posts and needs more computing capacity (for example, to recognize more pictures), the compu- Telecommunications provisioning. Web-based AR (web AR), a promising ting capacity of end devices is still limited. • Browser + cloud solutions. Considering that MAR applications often involve complex lightweight and cross-platform approach to AR, is Pei Ren matrix computations with extreme latency requirements, the local computing operations Beijing University of Posts gaining increasing attention owing to its extensive are often limited by hardware capabilities (e.g., CPU, GPU, memory, and battery). To and Telecommunications address this issue, cloud computing is currently used to extend the computing ability of application areas. However, for computationally end devices. However, this approach also brings about significant communication la- Schahram Dustdar intensive AR applications, the weak computational tency, which often incurs poor quality of experience (QoE), such as in real-time object- TU Wien tracking scenarios. efficiency of current web browsers seriously hampers Junliang Chen Beijing University of Posts applications of web AR on a large scale. The On the basis of the above analysis, we need to explore a practical web AR implementation solu- and Telecommunications “browser + cloud” approach suffers from the high- tion to resolve both the computational inefficiency of the pure front-end and browser-kernel- based solutions and the high-latency of browser + cloud solutions. Fortunately, with the emer- latency dilemma. Now, with the emerging 5G gence of 5G networks, the new mobile edge computing (MEC) paradigm2 aims to provide cloud- networks, mobile edge computing (MEC) promises to greatly reduce network latency computing capabilities at the edge of networks close to mobile users. Thus, MEC can efficiently (even to 1 ms) by deploying applications at the network edge closer to users, which reduce the communication latency and strengthen the end device’s computing capability. We argue that MEC will provide an opportunity for the performance improvement of web AR and provides an opportunity for performance improvement of web AR. In this article, the open up a new era for web AR applications. authors envision that the application of MEC has the potential to bring web AR into a In this article, we first give an overview of the state of the art of web-based MAR solutions and new era. Specifically, an MEC-oriented web AR solution is first proposed, followed by discuss the problems to be addressed. Then, a practical web AR service-provisioning framework leveraging the MEC paradigm is presented, which achieves encouraging performance improve- the design and deployment details. The authors also discuss future directions aimed at ment. Finally, we discuss future research directions of web AR with MEC in current 3G and 4G using MEC to tackle the performance issues of web AR in 3G, 4G, and 5G networks. networks and future 5G networks.

THE STATE OF THE ART Mobile augmented reality (MAR)1 nowadays greatly attracts both academic and industrial attention, owing to its extensive application areas and the popularity of smartphones and wearable devices. Recently, many MAR use cases have already emerged in the fields of art, education, Pure Front-End Solutions entertainment, and so on. These use cases typically are divided into two types: wearable-device- In general, pure front-end web AR solutions utilize different JavaScript libraries to support web based and app-based MAR solutions. However, both approaches suffer from some inherent limi- AR applications, which can be adopted simply in users’ end devices. tations regarding cross-platform, portability, and cost aspects:

IEEE Internet Computing Published by the IEEE Computer Society July/August 2018 46 1089-7801/18/$33.00 USD ©2018 IEEE July/Augustwww.computer.org/computingedge 2018 47 www.computer.org/internet39 INTERNET OF THINGS, PEOPLE, AND PROCESSES

AR.js3 is one of the popular marker-based MAR solutions for efficiently performing AR applications on the Web, including recognition, tracking, and 3D object rendering on any mobile browser with WebRTC and WebGL. JSARToolKit (https://github.com/kig/JSARToolKit) is an AR library written in JavaScript, which allows rendering a 3D model of a detected marker inside a camera feed by computing the distance from the camera to the physical marker in real time. CaffeJS (https://github.com/chaosmail/caffejs) aims to run neural-network functions (e.g., image recognition) by porting Caffe models to a web browser using a modified version of ConvNetJS, which saves a lot of network traffic and server resources.

MOTIVATION AND PROBLEM STATEMENT The aforementioned web AR implementations all face inherent limitations. The pure front-end solutions lack computational efficiency, and the browser-kernel-based extension solutions are still in their infancy and have not yet been adopted on a large scale. The cloud-computing paradigm greatly extends the computing ability of the terminal devices and provides an opportunity for the promotion of web AR applications on users’ end devices with limited hardware resources. Benefiting from the cloud-computing paradigm, we have conducted several online real advertising promotion activities using web AR within WeChat, which have gained a great deal of attention. According to our practical project experience, nowadays most users’ end devices such as smartphones and tablet PCs have satisfied the deployment requirements of web AR. Technolo- gies that have enabled this situation include

• WebAssembly, a new portable, size- and load-time-efficient format suitable for compi- lation to the web; • Web Workers, an HTML5 technique that provides a means to run a script operation in a background thread separate from the main execution thread of a web application; and • WebRTC, a collection of communications protocols and APIs that enable real-time communication over peer-to-peer connections.

Moreover, all devices equipped with iOS 11 or later versions support WebRTC. All these enabling techniques provide a platform for web AR applications’ deployment, enabling the lightweight, cross-platform and pervasive application of web AR. Here, we detail two influential use cases as shown in Figure 1, describing the challenges we faced during design and development.

July/August40 2018 ComputingEdge 48 www.computer.org/internetJanuary 2019 INTERNET OF THINGS, PEOPLE, AND PROCESSES IEEE INTERNET COMPUTING

Browser-Kernel-Based Extension Solutions Another branch of web AR solutions is browser-kernel-based extension solutions, where the functions of AR applications have an extreme dependency on browsers. Currently, most browser-kernel-based web AR projects are still in closed beta stages. Google is developing WebARonARKit and WebARonARCore, which are experimental apps for iOS and Android that let developers create AR experiences using web technologies. Mozilla has announced its mixed-reality program: Mozilla WebXR (https://github.com/mozilla/webxr-api), which aims to make it easy for web developers to create web applications that adapt to the capa- Figure 1. Two real use cases of web augmented reality (web AR) for market promotions of China bilities of each platform. The Baidu mobile-browser team has also shared its advancements in Mobile and FenJiu. web AR and web VR in a recent open class. The Baidu mobile-browser T7 kernel supports the WebVR standard and has completed web AR prototype development (https://github.com/baidu/AR). Argon,4 an AR-supported web browser, provides a pervasive web • China Mobile use case. A web AR sales promotion for the CMCC A3s smartphone has AR application deployment platform. Moreover, Argon implements the JavaScript library ar- been proposed for advertising effects. The activity URL link is embedded in China Mo- gon.js, which aims to support web AR applications in any web browser. bile’s official account in WeChat. Users only need to access the activity item in the corresponding official account, by scanning the logo of China Mobile; an augmented 3D model then is rendered on a webpage. In this application scenario, the most important MOTIVATION AND PROBLEM STATEMENT and complex part is image matching. However, the available image-matching algorithms such as SURF (Speeded-Up Robust Features), SIFT (Scale-Invariant Feature The aforementioned web AR implementations all face inherent limitations. The pure front-end Transform), and ASIFT (Affine-SIFT) are too heavy to run on web browsers. The ORB solutions lack computational efficiency, and the browser-kernel-based extension solutions are (Oriented FAST and Rotated BRIEF) matching algorithm provides a faster feature ex- still in their infancy and have not yet been adopted on a large scale. The cloud-computing para- traction method and can be used directly by JavaScript; however, it results in insuffi- digm greatly extends the computing ability of the terminal devices and provides an opportunity cient matching accuracy. for the promotion of web AR applications on users’ end devices with limited hardware re- • FenJiu use case. To promote the company culture, FenJiu has adopted a web AR tech- sources. nique for its museum. As in the use case mentioned above, users can experience the web AR application by scanning different material objects such as wine bottles and other ex- Benefiting from the cloud-computing paradigm, we have conducted several online real advertis- hibits in the museum. However, once the image is captured by a user’s end device, it ing promotion activities using web AR within WeChat, which have gained a great deal of atten- needs to be compared with 10 other samples (the number of samples will increase in the tion. According to our practical project experience, nowadays most users’ end devices such as future), the results of which are used to identify different exhibits and then present dif- smartphones and tablet PCs have satisfied the deployment requirements of web AR. Technolo- ferent cultural-promotion pages. Such intensive computing apparently cannot be per- gies that have enabled this situation include formed by users’ end devices locally. Benefiting from the cloud-computing paradigm, computation tasks, therefore, can be migrated to cloud servers. However, the cloud- • WebAssembly, a new portable, size- and load-time-efficient format suitable for compi- computing approach introduces high latency due to long-distance data transmission lation to the web; through the wireless network, which significantly degrades the performance of the web • Web Workers, an HTML5 technique that provides a means to run a script operation in a AR effects. background thread separate from the main execution thread of a web application; and • WebRTC, a collection of communications protocols and APIs that enable real-time These two use cases placed us in a dilemma: communication over peer-to-peer connections. • MAR applications rely on complex computer vision algorithms, while web browsers Moreover, all devices equipped with iOS 11 or later versions support WebRTC. All these ena- always lack sufficient computational capability. bling techniques provide a platform for web AR applications’ deployment, enabling the light- • Although the cloud-computing paradigm extends the computational and storage capabil- weight, cross-platform and pervasive application of web AR. ity for users’ end devices, it causes significant latency in the mobile network. However, MAR applications require low latency. Here, we detail two influential use cases as shown in Figure 1, describing the challenges we faced during design and development. WEB AR WITH MOBILE EDGE COMPUTING By offloading computationally intensive workloads to the proximity servers at the edge of the network, the emerging edge-computing paradigm provides a promising solution to meet the ever-increasing computational demands of numerous mobile applications.5 To compensate for

July/August 2018 48 www.computer.org/internet July/Augustwww.computer.org/computingedge 2018 49 www.computer.org/internet41 INTERNET OF THINGS, PEOPLE, AND PROCESSES

Overview of the MEC Framework for Web AR The overall framework is first proposed in three perspectives as shown in Figure 2, followed by the details of three processing components on the terminal, the edge cloud server, and the remote cloud server. We conducted the experiments on the basis of our practical projects, and the results fully demonstrate the advantages of MEC for web AR applications.

Figure 2. Overview of the framework for web AR with mobile edge computing (MEC).

The Terminal Side The terminal-side computation platform is responsible mainly for web AR service scheduling and basic processing, which will not pose a severe computational burden, considering the weak computational capability of the web browsers in users’ end devices. The web AR service platform consists mainly of two modules: the scheduling layer and processing module. The processing module provides the underlying support for the scheduling layer. The scheduling layer processes all the web AR application logic and collaborates with the related service parts.

July/August42 2018 ComputingEdge 50 www.computer.org/internetJanuary 2019 INTERNET OF THINGS, PEOPLE, AND PROCESSES IEEE INTERNET COMPUTING

the lack of computing capability and to ease the new constraints caused by cloud computing, especially concerning latency and bandwidth, we propose a web AR service-provisioning framework with MEC, on the basis of our project experiences. The processing module in the web AR service layer is composed of three submodules: the image-capture, image-matching, and 3D-model-rendering submodules. The image-capture submodule uses a WebRTC technique to capture an image from a camera and then execute a resizing Overview of the MEC Framework for Web AR operation (set by the application service provider in advance) on it, taking into account the com- The overall framework is first proposed in three perspectives as shown in Figure 2, followed by munication cost of image transmission. The image-matching submodule performs an image-to- the details of three processing components on the terminal, the edge cloud server, and the remote image matching operation, leveraging the lightweight image-matching algorithms such as ORB cloud server. We conducted the experiments on the basis of our practical projects, and the results mentioned above, which is already supported in JSFeat (the JavaScript Computer Vision Li- fully demonstrate the advantages of MEC for web AR applications. brary). The 3D-model-rendering submodule leverages a WebGL technique to perform 3D model rendering. Users can interact with the 3D models to acquire more augmented information, which provides a more attractive and friendlier experience compared to traditional ways. However, for some complex images, the proposed ORB matching algorithm cannot work well. Then, the specific edge cloud services will be invoked in the case where the image-matching result acquired on the terminal side is insufficient for the web AR application.

The Edge Cloud Side The edge-cloud-side computation platform consists mainly of an abstract MAR service layer, which is used to process incoming edge service requests and manage the web AR application objects, including object deployment, object destruction, and other underlying service supports. The underlying abstract MAR service layer consists of some common web AR functional modules, which aim to provide simpler and faster service to the web AR application instances in the upper layer and reduce the access cost to hardware at the same time, so as to improve the edge server’s overall performance. Currently, it includes a performance-monitoring module and image-matching module. Once the edge cloud server receives a web AR request (i.e., an image-matching request) from the terminal side, it will directly forward that request to the specific web AR application instance. The AR cache in the edge cloud server can be accessed by all web AR applications. However, in the case where the requested web AR application has not been deployed yet, the abstract MAR service layer will send the application deployment request to the remote cloud server. The current performance of the edge server, including the CPU, memory, and storage usage, is also sent to the remote cloud server for the decision making of the web AR application deployment. The remote cloud server—i.e., the application service provider (ASP)—finally determines where the specific application is to be deployed, taking into account the overall cost of deployment and transmission. After the abstract MAR service layer in the edge server receives the requested application, it will perform deployment. Thus, it provides more efficient servicing of the follow-up web AR requests. Moreover, the processing module in the terminal performs the rendering operation on the specific 3D model according to the received matching result.

Figure 2. Overview of the framework for web AR with mobile edge computing (MEC). The Remote Cloud Side The remote-cloud-side computation platform aims at providing a more generalized service provision mechanism. One of the most important components in the framework is the resource pack- The Terminal Side aging service layer, which acts as a web AR resource manager. When the ASP faces different The terminal-side computation platform is responsible mainly for web AR service scheduling web AR application requirements, the resource-packaging-service layer combines specific web and basic processing, which will not pose a severe computational burden, considering the weak AR resource components into different web AR applications, which will then be deployed to ap- computational capability of the web browsers in users’ end devices. propriate edge cloud servers according to the deployment decisions. Meanwhile, different ASPs also have different web AR resources, including various 3D models, image-matching algorithms, The web AR service platform consists mainly of two modules: the scheduling layer and pro- etc. The resource-packaging-service layer is also responsible for the management of these con- cessing module. The processing module provides the underlying support for the scheduling layer. tents. The scheduling layer processes all the web AR application logic and collaborates with the related service parts. The database is used mainly for the user information store. Moreover, images that match successfully on the edge cloud server will also be transmitted to the remote cloud server, since these types of images are currently not handled by the terminal-side image-matching algorithm. These

July/August 2018 50 www.computer.org/internet July/Augustwww.computer.org/computingedge 2018 51 www.computer.org/internet43 INTERNET OF THINGS, PEOPLE, AND PROCESSES

images can be used to improve the performance of the terminal-side image-matching algorithm in the future.

Performance Evaluation Theoretically, the introduction of the edge server will bring the following advantages:

• In scenarios that need to match a single image, the matching accuracy can be improved by providing a more optimized algorithm on the edge server side. The edge server has greater computational capability, and the image-matching time can be reduced effectively compared to the same matching algorithm on the terminal side. Although upload- ing images to the edge server adds extra time, the total processing efficiency still improves. Moreover, the edge server performs better since it is closer to the users. Hence, the network delay is no longer the bottleneck for web AR applications when the MEC paradigm is adopted. • In the case where multiple images need to be compared, users’ end devices apparently cannot meet the performance requirement of the web AR application owing to their limited computational capability. Migrating the service module (i.e., the web AR application) from the remote cloud server to the edge server effectively eases the issue of network delay caused by the frequent image transmissions.

July/August44 2018 ComputingEdge 52 www.computer.org/internetJanuary 2019 INTERNET OF THINGS, PEOPLE, AND PROCESSES IEEE INTERNET COMPUTING

images can be used to improve the performance of the terminal-side image-matching algorithm in the future.

Performance Evaluation Theoretically, the introduction of the edge server will bring the following advantages:

To get performance comparisons, we conducted experiments on a Samsung Note 4 mobile phone, using Chrome under the use cases mentioned above. The mobile phone connected to a Wi-Fi access point on our campus (i.e., BUPT-mobile). By modifying the DNS setting, all the Figure 3. Experiment results in terms of frames per second, latency, the terminal’s power requests were redirected to the server in our lab, which played the role of the edge server. The consumption, and the average number of matched keypoints. MEC framework and two web AR applications were deployed in our lab and on Alibaba Cloud, respectively. A modified version of the ASIFT algorithm was employed to provide more accurate and effective image matching. Moreover, the feature points of image samples on the server The experiment results show that MEC indeed provides a new choice for the future deployment were computed in advance to achieve faster matching. Four performance metrics are considered: of MAR. frames per second (fps), latency, the terminal’s power consumption, and the average number of matched feature points. FUTURE DIRECTIONS In experiments, the image was first resized to 250 * 250 pixels, about 8.92 Kbytes on average. The round-trip time for terminal-edge and terminal-cloud was 13.787 ms and 39.781 ms, respec- The practical projects and performance results have given us great inspiration for the future de- tively, for 64 bytes. Each ORB image-matching operation cost 394 ms on a user’s end device, velopment and deployment of web AR applications. Next, we discuss some important research while only about 30 ms on the server. Hence, offloading computationally intensive tasks to the challenges and directions for web AR and MEC. edge cloud was still a good choice. Meanwhile, edge computing performed better than the cloud paradigm regarding latency and power consumption, as shown in Figure 3, since it benefited Future Directions for Web AR from a shorter image transmission distance. Web AR is becoming more and more popular owing to its lightweight and cross-platform features. We argue that it will provide a promising future for MAR. However, there is still a lot of work to be done to further promote the application of web AR.

Computational Efficiency for CPU-Hungry Tasks on Mobile Web Browsers The widely used JavaScript performs poorly for complex computation tasks, such as matrix computation and floating-point computation. Therefore, it is necessary to introduce a more efficient computation paradigm to web browsers to meet the computational-efficiency requirement. Alt- hough some advanced techniques such as WebAssembly and Web Workers are already supported in some mainstream web browsers such as Chrome, Firefox, and Safari, there is still room for improvement to achieve efficient computation for end devices.

Lack of Standardization Browser-kernel-based extension solutions for web AR implementations can make full use of end devices’ hardware resources to achieve better performance. We envision that this has more potential as a web AR solution. However, a variety of current browser-kernel-based solutions have

July/August 2018 52 www.computer.org/internet July/Augustwww.computer.org/computingedge 2018 53 www.computer.org/internet45 INTERNET OF THINGS, PEOPLE, AND PROCESSES

Network Constraints MAR is heavily dependent on network latency and bandwidth. However, wireless networks have an adverse effect on the performance of web AR applications. Although the current 4G networks already have good performance capability, they still cannot meet the low-latency requirements of new applications such as AR and VR. Technologies such as software-defined networking (SDN), device-to-device (D2D) communication, and mobile crowdsourcing mechanisms6 offer new ways for wireless-network resource optimization. However, there is still plenty of optimization room to further improve the performance of web AR applications.

Future Directions for MEC Because MEC is a newly proposed computing paradigm, it offers many opportunities for research, development, and deployment. In this section, we discuss potential research directions for MEC under 3G, 4G, and 5G networks.

MEC in 3G and 4G Networks MEC has been explored by many carriers, including China Mobile and Huawei. However, it is still at an initial stage. Under the current 3G and 4G networks, many problems and challenges regarding the application, development, and deployment of MEC still exist. For MAR applications, the severe delay issues under wireless networks greatly degrade their performance. Cur- rently, the content delivery network (CDN) closest to users is used mainly to cache the static content resources and has no computing and application deployment functions. Therefore, it’s necessary to upgrade the existing CDN as much as possible to support the edge-computing paradigm.

MEC in 5G Networks 5G mobile networks have recently gained momentum in both academia and industry. Benefiting from the inherent features of 5G cellular networks, such as enhanced mobile broadband (eMBB), massive machine type communications (mMTC), and ultra-reliable low-latency communications (URLLC), 5G is expected to enable a host of new applications, such as the Internet of Things (IoT), self-driving cars, AR and VR, and so on. Moreover, MEC is also available in 5G cellular networks to provide computing and storage abilities at the edge of a mobile network.7 Compared with current 3G and 4G networks, the issues of delay and bandwidth occupancy get alleviated in 5G networks. The introduction of MEC further optimizes the utilization of network resources and improves the performance of applications such as MAR, since the requests can be responded to faster in edge computing than with remote cloud-computing centers. However, when, where, and how to deploy MEC servers to fully balance the benefits, cost, and efficiency will be an essential issue in 5G networks.

CONCLUSION MAR provides an attractive visual experience to users. Web AR further offers a lightweight, cross-platform, and pervasive solution on the web. However, it faces more technical challenges in terms of QoE owing to the weak computational efficiency of JavaScript. The emerging MEC paradigm provides an opportunity for the performance improvement of computationally inten-

July/August46 2018 ComputingEdge 54 www.computer.org/internetJanuary 2019 INTERNET OF THINGS, PEOPLE, AND PROCESSES IEEE INTERNET COMPUTING

serious compatibility problems. Different web AR applications can be used only by their dedicated browser, which significantly limits the popularization of web AR. In addition, during the development of web AR applications, we found that there also exist serious compatibility issues sive applications such as web AR. In this article, we proposed an MEC-oriented web AR solu- between Web3D rendering technologies (such as three.js) and Web3D models made by different tion. Then, we discussed challenges and future directions for MEC-based web AR under 3G, 4G, modeling tools such as 3ds Max, Maya, and Blender. To promote web AR applications on a and 5G networks. We hope this article can provide insight on future research efforts in the large scale, the issue of compatibility mentioned above will be one of the most critical problems emerging research field of web AR with MEC. to be solved in the future.

Network Constraints MAR is heavily dependent on network latency and bandwidth. However, wireless networks have ACKNOWLEDGMENTS an adverse effect on the performance of web AR applications. Although the current 4G networks This work was supported by the National Natural Science Foundation of China under grants already have good performance capability, they still cannot meet the low-latency requirements of 61671081 and 61720106007, the Beijing Natural Science Foundation under grant 4172042, new applications such as AR and VR. Technologies such as software-defined networking (SDN), and the 111 project (B18008). device-to-device (D2D) communication, and mobile crowdsourcing mechanisms6 offer new ways for wireless-network resource optimization. However, there is still plenty of optimization room to further improve the performance of web AR applications. REFERENCES Future Directions for MEC 1. M. Billinghurst, A. Clark, G. Lee et al., “A Survey of Augmented Reality,” Because MEC is a newly proposed computing paradigm, it offers many opportunities for re- Foundations and Trends R in Human–Computer Interaction, vol. 8, no. 2-3, 2015, pp. search, development, and deployment. In this section, we discuss potential research directions 73–272. 2. A. Ahmed and E. Ahmed, “A Survey on Mobile Edge Computing,” Proc. 10th IEEE for MEC under 3G, 4G, and 5G networks. International Conference on Intelligent Systems and Control (ISCO), 2016, pp. 1–8. 3. P. Mendigochea, “WebAR: Creating Augmented Reality Experiences on Smart MEC in 3G and 4G Networks Glasses and Mobile Device Browsers,” Proc. ACM SIGGRAPH Studio, 2017, pp. 10. 4. B. MacIntyre, A. Hill, H. Rouzati, M. Gandy, and B. Davidson, “The Argon AR Web MEC has been explored by many carriers, including China Mobile and Huawei. However, it is Browser and Standards-based AR Application Environment,” Proc. 10th IEEE still at an initial stage. Under the current 3G and 4G networks, many problems and challenges International Symposium on Mixed and Augmented Reality (ISMAR), 2011, pp. 65– regarding the application, development, and deployment of MEC still exist. For MAR applica- 74. tions, the severe delay issues under wireless networks greatly degrade their performance. Cur- 5. W. S. Shi, S. Dustdar, ”The Promise of Edge Computing, “IEEE Computer Magazine, rently, the content delivery network (CDN) closest to users is used mainly to cache the static Vol. 29, No. 5, 2016, pp. 78-81. content resources and has no computing and application deployment functions. Therefore, it’s 6. D. Zhao, X. Y. Li, H. D. Ma, ”How to Crowdsource Tasks Truthfully Without necessary to upgrade the existing CDN as much as possible to support the edge-computing para- Sacrificing Utility: Online Incentive Mechanisms with Budget Constraint, “Proc. digm. INFOCOM 2014, pp. 1213-1221. 7. T. X. Tran, A. Hajisami, P. Pandey, and D. Pompili, “Collaborative Mobile Edge Computing in 5G Networks: New Paradigms, Scenarios, and Challenges,” IEEE MEC in 5G Networks Communications Magazine, vol. 55, no. 4, 2017, pp. 54–61. 5G mobile networks have recently gained momentum in both academia and industry. Benefiting Xiuquan Qiao is a full professor in the State Key Laboratory of Networking and Switching from the inherent features of 5G cellular networks, such as enhanced mobile broadband (eMBB), Technology at the Beijing University of Posts and Telecommunications. His main research massive machine type communications (mMTC), and ultra-reliable low-latency communications interests focus on 5G networks, augmented reality, and services computing. Contact him at (URLLC), 5G is expected to enable a host of new applications, such as the Internet of Things [email protected]. (IoT), self-driving cars, AR and VR, and so on. Moreover, MEC is also available in 5G cellular Pei Ren is a PhD candidate from the Beijing University of Posts and Telecommunications. networks to provide computing and storage abilities at the edge of a mobile network.7 His research interests lie in machine learning and augmented reality. Contact him at [email protected]. Compared with current 3G and 4G networks, the issues of delay and bandwidth occupancy get alleviated in 5G networks. The introduction of MEC further optimizes the utilization of network Schahram Dustdar is a full professor of computer science (informatics) with a focus on resources and improves the performance of applications such as MAR, since the requests can be Internet technologies and heads the Distributed Systems Group at TU Wien. He’s a member responded to faster in edge computing than with remote cloud-computing centers. However, of Academia Europaea. Contact him at [email protected]. when, where, and how to deploy MEC servers to fully balance the benefits, cost, and efficiency Junliang Chen is a professor at the Beijing University of Posts and Telecommunications. will be an essential issue in 5G networks. His research interests include services computing. He’s a member of the Chinese Academy of Sciences and Chinese Academy of Engineering. Contact him at [email protected]. CONCLUSION MAR provides an attractive visual experience to users. Web AR further offers a lightweight, cross-platform, and pervasive solution on the web. However, it faces more technical challenges in terms of QoE owing to the weak computational efficiency of JavaScript. The emerging MEC This article originally appeared in paradigm provides an opportunity for the performance improvement of computationally inten- IEEE Internet Computing, vol. 22, no. 4, 2018.

July/August 2018 54 www.computer.org/internet July/Augustwww.computer.org/computingedge 2018 55 www.computer.org/internet47

DEPARTMENT: STARTUPS

360-Degree Virtual-Reality Cameras for the Masses

It is human nature to tell stories and share life experiences Jack Tan with others. Methods of storytelling have evolved over Kandao time due in large part to technological advances in both Gene Cheung communication and experience recording. Camera technol- National Institute of ogies have evolved to the point where it is now possible to Informatics, Tokyo record a 360-degree 3D visual experience using virtual-re- Rui Ma ality (VR) video cameras. For the first time, people are free Kandao of the constraint of fixed camera viewpoint capture, so they can record and share what they see in 360 degrees as they Department Editor: experience it in real life. Due to the unprecedented immer- Tao Mei, Microsoft Research sive experience that VR can potentially bring to the Asia masses, investment in VR is growing fast. Facebook offi- cially acquired Oculus VR at a price of $2 billion. In 2015, more than $700 million was injected into VR and AR startups, according to Digi-Capital’s Augmented/Virtual Reality Report. In 2016, the report says, that number more than tripled to $2.3 billion (digi-capital.com/news/2017/02/record-2-3-billion- vrar-investment-in-2016). But if the technology is so promising, why aren’t VR videos already ubiquitous? There are several major factors preventing the technology from being widely deployed. A good-quality 3D, 360-degree camera is still too expensive for mass adoption, costing as much as $20,000. Usabil- ity is also an issue. For the masses to adopt VR cameras, it is important that users be able to shoot anywhere and anytime. Many of the systems currently on the market are too bulky to be moved easily. They often have to rely on external power sources due to the massive amount of energy consumed by the camera arrays, which hinders portability. For PC-based solutions, the rendering process is often very slow. For example, a solution from a major VR camera vendor takes 12 seconds to render one frame of a 4k x 4k high-quality stereoscopic video. Another challenge is that the increased data dimensionality of VR poses technical challenges for video compression and delivery. All of the aforementioned factors make the technology challenging to deploy on a large scale. To make the technology more accessible to the public, a device must be affordable, portable, reliable, high quality, and user friendly. In this article, we describe the challenges in meeting these goals and the techniques that Kandao—a VR startup company based in China—used to conquer them when designing its Obsidian cameras.

IEEE MultiMedia Published by the IEEE Computer Society January–March48 2018 January 2019 Published87 by the IEEE Computer Society 1070-986X/18/$33.002469-7087/19/$33.00 ©2018 © IEEE2019 IEEE IEEE MULTIMEDIA

CAPTURING STEREOSCOPIC PANORAMAS Capturing a good-quality stereoscopic, 360-degree panorama is a computationally challenging task. The omni-directional stereo (ODS) projection model was initially proposed in 1992 (shown in Figure 1).1,2 In the original ODS, a slit camera rotates around an inter-pupil-distance (IPD) circle and scans the real world. The image slices are stacked to form a rectangular panoramic image. An inherent drawback of this and other early methods is that they do not work on dynamic scenes with moving objects.

Figure 1. The original ODS uses rotating slit cameras for the left and right eyes.

Recent updates to ODS take a computational photography approach.3 Instead of actual scanning, the rotating slit camera is synthesized using computational imaging and vision techniques. In this process, an accurate dense pixel-to-pixel correspondence is calculated, and then the depth of each pixel is estimated from correspondence (for example, is the correspondence of in Fig- ure 1, both of which are images of point A). Each 3D point in the space is then projected to the two slit cameras tangent to the IPD circle (shown in green ina’ Figure 2). a

Figure 2. A modern ODS illustration. Point A is captured by two cameras centered at O and O’ as and . Conversely, given and , the corresponding point A can be reconstructed. Thus, the images in the left eye and right eye can be reconstructed, as well. a a’ a a’

January–Marchwww.computer.org/computingedge 2018 88 www.computer.org/multimedia49 STARTUPS

The quality of a stereoscopic panorama relies heavily on the accuracy of underlying pixel correspondence, which mainly depends on two factors: (1) whether correspondent pixels ( and in Figure 2) on different sensors are captured synchronously by multiple cameras and (2) the accuracy of finding actual pixel correspondence post-capture. a a’ The first factor could easily be addressed with a global-shutter imaging sensor, which captures every pixel at exactly the same time. However, one main design goal is affordability. The price of a global-shutter sensor with good imaging quality can be up to several thousand dollars, 10 times that of a good-quality rolling shutter. Unfortunately, building a stereo panorama with a rolling shutter entails formidable challenges for stereo-matching algorithms. The second factor depends on the accuracy of the dense pixel matching method itself. The underlying technology, optical flow—though extensively studied—is still plagued by the inherent un- certainty stemming from image noise, light changes, lack of texture regions, multiple motions, repeated textures, and other complexities.

KANDAO’S CAMERA SYSTEM DESIGN Kandao chose to use rolling shutters to bring the price down for consumers. But, to design a device that would be deployable to the general public, Kandao also had to focus on portability, reliability, high image quality, and user friendliness. Because a VR camera is essentially a multi-camera system composed of multiple individual traditional cameras, Kandao designers set the form factors and weights to be similar to that of off- the-shelf commercial digital single-lens reflex (DSLR) cameras. This small form factor contrib- utes to the device’s portability. Reliability is also important; no camera in the multi-camera system should break down during shooting. Such a system’s reliability decreases as the number of imaging modules increases. Clearly, portability and reliability both demand a less complex system—one with fewer imaging modules. Image quality refers to both raw image quality and stitching quality. Raw image quality is tradi- tionally constrained by the lens and the image sensor. With the recent advances in semiconduc- tors and computational photography, image quality is increasing steadily for small-sized imaging modules, as evident in the photo quality of modern smart phones. This enabled Kandao to fit multiple high-quality, compact lenses and sensors onto the same rig. The more formidable challenge is to achieve good stitching quality. Computing an accurate ODS is much harder with fewer cameras. Kandao solved this problem by incorporating a deep neural network (DNN) specialized in highly accurate optical flow calculation. Solving the ODS synthe- sis problem with fewer cameras improves system reliability: the fewer components in a design, the more reliable the final product is. Another important feature added to the design is a rich and convenient interface to further improve the camera’s user friendliness, as shown in Figure 3.

Figure 3. Kandao Obsidian cameras are easy to hold, carry, and interact with.

January–March50 2018 ComputingEdge 89 www.computer.org/multimediaJanuary 2019 IEEE MULTIMEDIA

SOLUTIONS TO ENABLE DESIGN CHOICES

Building an ODS Dataset A fundamental challenge for a machine-learning project is building a dataset that’s right for the problem. A dataset containing ground truth suitable for computational ODS is very hard to con- struct, because one would need to accurately map every point densely in 3D and generate two slit cameras scanning the scene by rotating around an IPD circle. Even with a dedicated 3D mapping device such as LiDAR, fusing the accurate dense 3D point clouds and visual data is still time- consuming and expensive for static scenes. It is nearly impossible for dynamic scenes. Collecting real-world ground truth is indeed too impractical, so Kandao took a computational approach. Kandao constructed a synthetic dataset with rich information, including 3D models (mesh and texture), simulated virtual cameras (with real intrinsic and extrinsic parameters), pixel correspondences (optical flow data), depth data, objects labels, and segmentation boundaries. The 3D scenes are built with realistic details such as lighting and reflection. Figure 4 shows an example of the dataset. The top image shows an indoor scene of a sitting room. The middle images show an image captured by a virtual fisheye camera and the associated depth map and label map. The bottom images were captured by other virtual cameras on the camera rig.

Figure 4. Images of the synthetic dataset built for simulation and training.

January–Marchwww.computer.org/computingedge 2018 90 www.computer.org/multimedia51 STARTUPS

Solving ODS with fewer Cameras The difficulty of building an ODS with fewer cameras is illustrated in Figure 5.

Figure 5. (left) Illustration of ODS projection error caused by error in optical flow. The error ∆θ in optical flow causes Q to drift to Q'. (right) The error (arc length) of QQ' due to a unit error in optical flow at different ray positions (angles in degree) with different numbers of imaging modules.

C0 and C1 are the optical centers of two neighboring cameras. C0P and C1P are the rays coming from a physical point P. Q is the projected point (right eye) of P on the IPD circle. An error Δθ due to inexact match causes Q to shift to Q' and generates an error in the final stitched ODS panorama. The length of the arc QQ' indicates the amount of error on ODS.

A further geometrical analysis reveals that the average error QQ' is mainly related to C0OC1, given that the average error (Δθ) of optical flow is known and form factor of the camera system is fixed. The average error decreases if C0OC1 decreases; in other words, the number∠ of imaging modules increases. An intuitive example is that when C0OC1 decreases to zero, the system literally becomes an ideal ODS system ∠with an infinite number of cameras. The right graph in Figure 5 shows the error (arc length) of QQ' at different ray∠ positions (angles in degree) with different numbers of imaging modules. A numerical simulation found that to build a six-lens ODS system with a comparable stitching quality (similar amount of ODS ray drift) to a 12-lens or 18- lens design, the average end point error (EPE) of optical flow algorithm needs to be much smaller (34 to 53 percent) than those rigs. To build a high-quality ODS with six lenses, a very accurate optical flow algorithm is necessary.

A Deep-Learning Approach to Solving ODS Optical flow is the method to compute a dense pixel correspondence, and its performance is crucial to image stitching and depth recovery. Although optical flow algorithms have advanced greatly, they are still error-prone in challenging situations such as lighting change, texture-less regions, multiple motions, and repeated textures. Encouraged by recent successes of deep convolutional neural networks (CNN) in a variety of computer vision tasks, Kandao constructed a robust and compact CNN, which solves the optical flow estimation problem as a supervised deep- learning task. The DNN can fuse multiple cues that many researchers believe are used in human vision, such as the binocular matching metric, monocular texture, and smoothness conformity. In this manner, the CNN results in more stable performance in the presence of different scenes, objects, and lighting conditions.

January–March52 2018 ComputingEdge 91 www.computer.org/multimediaJanuary 2019 IEEE MULTIMEDIA

To handle occlusion, the deep-learning process proposes a novel multi-view depth fusion. To achieve the goal of compactness, recent CNN techniques are adopted (such as dense block4, correlation volume5, and differential softmax function6), and a number of novel techniques (such as deep cost aggregation and deep flow refinement) are further developed. These techniques contribute to a CNN that is compact enough to run on embedded devices. The proposed neural network is first trained on popular open-source datasets and further fine-tuned on the constructed dataset described previously. Compared with traditional optimization-based optical flow, the deep-learning-based method is 100 times faster. Compared with other deep-learning methods, the new method is more accurate, efficient, and compact. Figure 6 shows a comparison between a state-of-the-art method named FlowNet 2.07 and Kandao’s new CNN method. The results of the new method show more accurate details than those of FlowNet 2.0.

Figure 6. Flow method comparison: (a) input image, (b) FlowNet 2.0 results, (c) results using Kandao’s CNN, and (d) a quantitative comparison on various criteria.

Minimizing Rolling-Shutter Effects Unlike its global counterpart, which records the entire scene in one instant, a rolling-shutter sensor scans across the scene either vertically or horizontally. In other words, each row or column is recorded at a slightly different time. This produces undesirable distortions (such as wobbling, smear, and skew), especially when capturing fast-moving objects or rapid flashes of light— breaking the geometrical constraints that stereo matching algorithms follow. To use rolling-shutter cameras in 3D geometric computer-vision tasks, the rolling-shutter effect must be carefully accounted for. Simply ignoring this effect and relying on a global-shutter method may lead to erroneous, undesirable, and distorted results. Instead, Kandao used both hardware and software methods to minimize the rolling-shutter effects. Instead of genlocked synchronization, Kandao chose a phase-delayed synchronization method to minimize the time dif- ference over the overlapping area between neighboring cameras. Algorithmically, thanks to the

January–Marchwww.computer.org/computingedge 2018 92 www.computer.org/multimedia53 STARTUPS

accurate inter-frame pixel correspondences obtained from the aforementioned neural network, it becomes possible to estimate the object trajectory, rewarp each scanline, and reassemble them utilizing classic methods.8,9

VR Video Compression Compressing VR video is challenging due to the large amount of data in the 360 degrees of viewing angles. The state-of-the-art high efficiency video coding (HEVC) standard for video compression can be Kandao applies the applied directly for compressing high-resolution VR videos. HEVC equal-angular cube- provides many advanced features that can effectively handle high-resolution videos. These features include enhanced intra-prediction, hier- map (EAC) archical partitions of coding blocks, and new mechanisms for parallel processing. However, despite its advanced compression performance, projection in its using HEVC to compress a full 360-degree VR video would still result in an unacceptably high bitrate at reasonable video quality. system because it Before compressing VR videos, one must first project the original provides the least spherical representation onto a planar surface. A simple and straightforward way is to use an equirectangular projection, which is typically variation in pixel how one would present the surface of the earth on a geographical map. In this projection, the pole regions are stretched, and hence they con- density distribution, tain more pixels per unit area compared to the equatorial regions. This unbalance in pixel density results in redundancy at the pole regions. saving about 25 The wasted pixels do not improve visual quality, but instead increase the total amount of data. percent of pixels An ideal projection would provide uniform pixel density distribution when compared to across different viewing angles similar to the original spherical surface. However, it is difficult to design such a projection due to geo- the classic metrical differences. Many alternative projections can be adopted to reduce the variations in pixel density and get closer to the ideal case. equirectangular These projections include the cube-map projection, the equal-angular cube-map (EAC) projection,10 and the barrel projection. Kandao ap- projection. plies the EAC projection in its system because it provides the least variation in pixel density distribution, saving about 25 percent of pixels when compared to the classic equirectangular projection.

CONCLUSION While VR has great potential to deliver a level of visual immersion unseen in previous video capture and display systems, bringing VR to the masses requires a camera that is affordable, lightweight, robust, and easy to use. As described in this article, companies like Kandao have tackled this challenge and are starting to deliver VR cameras to the market that satisfy these multiple, seemingly conflicting goals. Yet numerous technical challenges still remain. One challenge is how to exploit temporal correlation to further improve optimal flow for more temporal-consistent dense pixel correspondence. Another challenge is how to efficiently encode and package VR video into sub-streams for bandwidth-limited network transmission without adversely affect- ing user experience with round-trip time delay. Many research groups are currently addressing these and other challenges, and the day will soon arrive when VR video becomes truly ubiquitous—easily produced, consumed, and enjoyed by the masses.

January–March54 2018 ComputingEdge 93 www.computer.org/multimediaJanuary 2019 IEEE MULTIMEDIA

REFERENCES 1. H. Ishiguro, M. Yamamoto, and S. Tsuji, “Omni-directional stereo,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 2, 1992, pp. 257–262; ieeexplore.ieee.org/document/121792/. 2. C. Richardt et al., “Megastereo: Constructing High-Resolution Stereo Panoramas,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013; ieeexplore.ieee.org/document/6619010/. 3. R. Anderson et al., “Jump: virtual reality video,” ACM Transactions on Graphics, vol. 35, no. 6, 2016; dl.acm.org/citation.cfm?id=2980179.2980257. 4. G. Huang et al., “Densely Connected Convolutional Networks,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017; ieeexplore.ieee.org/document/8099726/. 5. A. Dosovitskiy et al., “FlowNet: Learning Optical Flow with Convolutional Networks,” IEEE International Conference on Computer Vision (ICCV), 2015; ieeexplore.ieee.org/document/7410673/. 6. A. Kendall et al., “End-to-End Learning of Geometry and Context for Deep Stereo Regression,” IEEE International Conference on Computer Vision (ICCV), 2017; ieeexplore.ieee.org/document/8237279/. 7. E. Ilg et al., “FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks,” arxiv.org, 2016; arxiv.org/abs/1612.01925. 8. S. Baker et al., “Removing rolling shutter wobble,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010; ieeexplore.ieee.org/document/5539932/. 9. O. Saurer et al., “Rolling Shutter Stereo,” IEEE International Conference on Computer Vision, 2013, pp. 465–472; computer.org/csdl/proceedings/iccv/2013/2840/00/2840a465-abs.html. 10. A. Wheeler, “Improving VR videos,” Engineering and Developers Blog, blog, YouTube, 2017; youtube-eng.googleblog.com/2017/03/improving-vr- videos.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+You TubeEngDevBlog+(YouTube+Engineering+and+Developers+Blog).

ABOUT THE AUTHORS Jack Tan is the cofounder of Kandao and has served as the CTO of the company since 2016. He has a PhD in electronic and electrical engineering from the University of Hong Kong. Before he started Kandao, he worked in roles ranging from engineering to administration in several different companies, including Hong Kong Applied Science and Tech- nology Research Institute, Meizu.com, and Shenzhen Jieshun Science And Technology Industry. Contact him at [email protected]. Gene Cheung is an associate professor at the National Institute of Informatics in Tokyo, Japan. He has been an adjunct associate professor at the Hong Kong University of Science and Technology (HKUST) since 2015. His research interests include 3D imaging and graph signal processing. He has a PhD in electrical engineering and computer science from the University of California, Berkeley. Contact him at [email protected]. Rui Ma is a research fellow at Kandao in Shenzhen, China. His research interests include VR video compression and transmission, panoramic representation, and multi-view navigation. He has a PhD in electronic and computer engineering from HKUST. Contact him at [email protected].

This article originally appeared in IEEE MultiMedia, vol. 25, no. 1, 2018.

January–Marchwww.computer.org/computingedge 2018 94 www.computer.org/multimedia55

COLUMN: The Last Word

This article originally appeared in Virtual Travel Computing in Science & Engineering, vol. 20, no. 4, 2018.

I spent the years 1988–90 as a postdoc at Japan’s Institute Charles Day of Space and Astronautical Science, whose campus is in The American Institute of Physics Sagamihara, an industrial suburb of Tokyo. During that time, I lived in an apartment above a ramen restaurant called Koguma (“bear cub”). Koguma, it turned out, was one of the best ramen restaurants in Sagamihara. I ate there at least twice a week. After sampling everything on the menu, I favored two dishes: spicy leek ramen in miso broth, and beef ramen with a raw egg in soy broth. My next job was at NASA’s Goddard Space Flight Center outside Washington, DC. I didn’t eat ramen in DC until 2014, when the ramen restaurant Toki Underground opened on trendy H Street. Now the city has more than a dozen places to savor the toothsome noodle dish. I applaud the proliferation of one of my favorite foods. On the other hand, it’s yet another mani- festation of the international homogenization of cuisine and culture. If you walk into a hipster coffee shop in Prague, Sydney, Buenos Aires, or Cape Town, you’ll see chalkboard menus, arti- sanal biscotti, responsibly harvested coffee beans, and bearded baristas. Only the language spo- ken in the shop varies. Also converging toward a uniform look and feel are luxury beach resorts—at least judging by the interchangeable photos I see in the travel sections of the New York Times, Wall Street Journal, and Financial Times. The Frenchman David Guetta, the Dutch- man Nicky Romero, the American Steve Aoki, the Finn Darude, and their ilk all produce blandly perky electronic dance music whose national provenance is impossible to discern. If things taste, look, and sound the same, why travel? I don’t know whether the leaders of Ja- pan’s largest airline, ANA, asked themselves that question, but they might have done. In April, ANA announced that it would be investing $10 million to develop a virtual travel system. The initiative is not as fanciful as it seems. The system’s main ingredient is a robot avatar through whose senses any number of armchair tourists can experience remote locales. Besides acting as a tourist surrogate, the robot could conceivably perform tasks in hazardous environments. Indeed, one of the project’s inspirations was the realization that a remotely controlled robot could have entered the reactor rooms of the Fukushima Daiichi power plant and averted nuclear disaster. But if homogenization is making real tourism pointless, won’t virtual tourism suffer the same fate? After all, virtual reality’s principal figure of merit is how closely it matches reality. If I were ANA’s CEO, Osamu Shinobe, I’d add fictional or past worlds to the robot avatar’s menu of virtual destinations. Wouldn’t you like to visit Middle Earth, Tatooine, ancient Rome, and Tang Dynasty China? It would be like Westworld but without rampaging robots and the need to fly.

ABOUT THE AUTHOR Charles Day is Physics Today’s editor in chief. The views in this column are his own and not necessarily those of either Physics Today or its publisher, the American Institute of Physics.

Computing in Science & Engineering Copublished by the IEEE CS and the AIP July/August56 2018 January 2019 Published125 by the IEEE Computer Society 2469-7087/19/$33.001521-9615/18/$33 ©2018© 2019 IEEE IEEE IEEE TRANSACTIONS ON SUBMIT TODAY SUSTAINABLE COMPUTING

SCOPE

The IEEE Transactions on Sustainable Computing (T-SUSC ) is a peer-reviewed journal devoted to publishing high-quality papers that explore the different aspects of sustainable computing. The notion of sustainability is one of the core areas in computing today and can cover a wide range of problem domains and technologies ranging from software to hardware designs to application domains. Sustainability (e.g., energy efficiency, natural resources preservation, using multiple energy sources) is needed in computing devices and infrastructure and has grown to be a major limitation to usability and performance.

Contributions to T-SUSC must address sustainability problems in different computing and information processing environments and technologies, and at different levels of the computational process. These problems can be related to information processing, integration, utilization, aggregation, and generation. Solutions for these problems can call upon a wide range of algorithmic and computational frameworks, such as optimization, machine learning, dynamical systems, prediction and control, decision support systems, meta-heuristics, and game-theory to name a few.

T-SUSC covers pure research and applications within novel scope related to sustainable computing, such as computational devices, storage organization, data transfer, software and information processing, and efficient algorithmic information distribution/processing. Articles dealing with hardware/software implementations, new architectures, modeling and simulation, mathematical models and designs that target sustainable computing problems are encouraged.

SUBSCRIBE AND SUBMIT For more information on paper submission, featured articles, calls for papers, and subscription links visit: www.computer.org/tsusc CAREER OPPORTUNITIES

TEST &VALIDATION ENGINEER, Gen- Science, or related. 6 mos exp as Engi- &documenting tooling details in Learning eral Motors, Detroit, MI. Perform test on neer, Test Lead, or related, using apps/ Management System (LMS) &Microsoft vehicles &backend communication for hardware tools incldg SFDC, IG, DBeaver, OneNote. Dvlp help pages for MDK appli- request completion (VNM-SPN communi- Cassandra &ADB commands, VehicleSpy, cation using HTML/CSS &JavaScript. Test cation) &validate back offi ce communica- &NeoVI, &CGM, TCP, &ECUs in CSM, to system engrg tools after monthly &weekly tions. Analyze VNM &SPN logs to validate test &validate infotainment &telemat- patching upgrades to servers. Support &measure accurate test results. Resolve ics modules to improve user experience code generators to produce embedded critical &complex hardware &software &V2V &Vehicle to Back Offi ce commu- code that integrates with embedded vehi- data setups. Create test data using SQL nications. Mail resume to Ref#2556, GM cle application &controller infrastructure query, manual data setup &verify data Global Mobility, 300 Renaissance Center, software. Bachelor, Electrical, Electronics, refl ection in various applications. Extract MC:482-C32-C66, Detroit, MI 48265 or Computer Engrg, or related. 12 mos exp system logs using JBM tool. Use apps as Engineer or Analyst, engrg using Java &hardware tools incldg Global Access SOFTWARE TOOLS DEVELOPMENT EN- &J2EE tools, testing, validating CMSDK App, Mobile No. Management/Vehicle GINEER, Milford, MI, General Motors. Engr, tool to manage software content in em- Communication Service, Vehicle Data Up- create, dvlp using Java &J2EE tools, test, bedded software in psgr vehicle ECUs. load, SalesForce.com (SFDC), GMOC, In- &validate Rhapsody Software Modelling Mail resume to Ref#35333-203, GM tegrated Gateway (IG), Vehicle Notifi cation Tool (RSMT) &Aspen Content Management Global Mobility, 300 Renaissance Center, Mgr, Server Push Notifi cation, DBeaver, Software Development Kit (CMSDK) tool to MC:482-C32-C66, Detroit, MI 48265. Cassandra &Android Debug Bridge (ADB) manage software content in embedded commands, VehicleSpy, &NeoVI, &Center software in psgr vehicle electronic control Gateway Module (CGM), Telematics Com- units (ECUs). Support global user commu- SOFTWARE DESIGN ENGINEER IN munication Platform (TCP), &ECUs in Cen- nity in resolving issues related to RSMT TEST. Uses C# and Powershell to: deter Stack Module (CSM), to test, improve &CMSDK until stable confi guration of mod- sign, develop & implement automated &validate infotainment &telematics models &Production Algorithm Content (PAC). test scripts, tools, processes and test re- ules to improve UX to V2V, Back Offi ce to Dvlp &enhance ProductLine reports (Cross porting. Req BS+2 yrs exp. To apply send Vehicle, &Vehicle to Back Offi ce commu- Network, U Code, Port tracing, Library ter- resume to pactera_resumeinc@pactera. nications. Mentor new technical profes- minator, System signal) required for engrg com and reference Job#11039.55. Em- sionals to dvlp training plans &help them teams using JAVA/J2EE. Dvlp RSMT XML ployer: Pactera Technologies, Inc. Job lo- to execute tasks in infotainment services format export for automatic tool generated cated in Redmond, WA. May be assigned testing &validation projects. Master, Soft- Scheduler. Participate in user training ses- to work at various unknown client loca- ware Engrg, Computer Science, Software sions about new enhancements deployed tions within the Seattle MSA.

Juniper Networks is recruiting for our Sunnyvale, CA office: Customer Focused Support Engi- Technical Support Engineer #19094: Software Engineer #54771: Design neer #30366: Design and implement Troubleshoot complicated h/w & s/w and develop Open Contrail solution computer and information networks, issues & replicate customer environ- for Container Orchestrators. Design such as local area networks (LAN), ments & network problems in the lab. and develop Open Contrail specific wide area networks (WAN), intranets, Technical Marketing Engineer #37124: solutions for vCenter orchestrator. extranets, and other data communi- Provide technical marketing knowl- Software Engineer #39528: Develop cations networks. Perform network edge to support the marketing & sales and test embedded software, review modeling, analysis, and planning. of Network Switches. Produce Imple- feature specifications, perform cod- Software Engineer #50994: Write test mentation app notes, design guide- ing and unit testing, conduct code plans, design test cases, and perform lines, & reference architectures for use reviews, drive technical discussions, manual and automated testing of Ju- by the field. perform tests like static analysis and niper’s Cloud Networking products. Technical Support Engineer #54403: scale and performance testing, and Develop automated test cases in Py- Troubleshoot complicated h/w & s/w document the results. thon and Ansible and fix bugs in auto- issues & replicate customer environ- mation framework. ments & network problems in the lab. IT Network Engineer #35377: Respon- Software Engineer #40066: Define, sible for requirement gathering, de- Systems Engineer #36621: Design and signing, deploying on different proj- design & implement key data plane implement networking solutions, in- features for data center including ects & troubleshooting the network. cluding detailed network designs, Perform technical analysis of the cur- MPLS, GRE tunneling, VxLAN, for architectural discussions, proof-of- routing and switching, family of prod- rent as-built network/infrastructure concept labs, configurations, and au- for designing & implementing any ucts. Participate in proof of concepts tomation. Automate Certification Lab & various customer engagements new IT network/infrastructure solu- Environments and integrate them into tion required. from technical side. Juniper Cloud Labs.

Mail single-sided resume with job code # to Juniper Networks Attn: MS A.4.410 1133 Innovation WaySunnyvale, CA 94089

SENIOR EMBEDDED SOFTWARE EN- innovation. Meet with UX design team for &maintenance of infotainment software GINEER, Warren, MI, General Motors. technical discussions. Master, Software embedded in vehicle components. Ana- Design, dvlp &debug real-time operating Engrg, Electronics Engrg, Engrg in Elec- lyze, root cause &provide fi xes/sugges- system (RTOS) embedded software in- tromagnetic Fields &Microwave Technol- tions for HMI framework subsystem, ap- cldg psgr vehicle Multi-Function Control- ogy, Electrical Engrg, or related. 12 mos plying knowledge of Android framework ler (MFC) feature to install new hardware exp as Engineer, dvlpg RTOS embedded system debugging &anlys, Linux C/C++, devices to control vehicle infotainment software incldg psger vehicle MFC fea- &Java programming. Interact with internal systems, using Java, C/C++ languages ture or related, in Android framework, us- customers, vendors, &external organiza- &multi-thread programming in Linux ing Linux, Java, C/C++ languages, or re- tions to refi ne understanding of reqmts &Android operating systems, in Android lated. Mail resume to Ref#2775-204, GM &identify opportunities for innovation. framework, &using Clear Case, Git, Ger- Global Mobility, 300 Renaissance Center, Meet with UX design team for technical rit, Jira, &RTC tools. Analyze &dvlp inno- MC:482-C32-C66, Detroit, MI 48265. discussions. Collaborate with GM global vative HMI framework features based on business leads &engrg specialists to new infotainment system reqmts. Use SENIOR EMBEDDED SOFTWARE ENGI- identify, capture, &refi ne system reqmts. Agile software dvlpmt methods to control NEER, Warren, MI, General Motors. De- Master, Computer Engrg, Software En- the project schedule &qlty of the deliv- sign &dvlp embedded Java &C/C++ soft- grg, Engrg (Circuits and Systems), Com- erables. Responsible for entire lifecycle ware incldg psgr vehicle Multi-Function puter Science, or related. 12 mos exp as of assigned domain, incldg reqmts defi - Controller (MFC) feature to install new Engineer, dvlpg embedded Java &C/C++ nition, system design, dvlpmt, review, hardware devices to control vehicle info- software incldg psgr vehicle MFC feature testing, deployment, &maintenance of tainment systems, using Java, C/C++ lan- or related, in Android framework, using infotainment software embedded in ve- guages in Linux &Android operating sys- Android, Linux, Java, C/C++ languages, hicle components. Analyze, root cause tems, in Android framework, &using Clear or related. Mail resume to Ref#2512, GM &provide “bug fi xes”/suggestions for Case, Git, Gerrit, Jira, &RTC tools. Analyze Global Mobility, 300 Renaissance Center, HMI framework subsystem, applying &dvlp HMI framework features based on MC:482-C32-C66, Detroit, MI 48265. knowledge of Android framework system new infotainment system reqmts. Use debugging &analyzing, using Linux, C/ Agile software dvlpmt methods to control BUSINESS PROCESS ANALYST, SALES- C++, &Java programming. Interact with project schedule &qlty of deliverables. FORCE Eversource Energy Service Com- internal customers, vendors, &external Responsible for lifecycle of assigned do- pany has a position available in Berlin, organizations to refi ne understanding main, incldg reqmts defi nition, system de- CT. Develop and maintain Sales, Service of reqmts &identify opportunities for sign, dvlpmt, review, testing, deployment, and Marketing solutions in the Gas Busi- ness Unit, including the 100+ user-base, Salesforce Enterprise version system. Provide and develop optimized solutions Help build the next generation of according to Salesforce best practices while using AppExchange products. Inte- systems behind WhatsApp’s gration using SOAP, Apex classes/Visual- products. force pages, and Apex Triggers. Establish specifi cations for business requirements, upgrades, and enhancements to improve Salesforce functionality and to improve WhatsApp, Inc. the Marketing, Sales and Sales Fulfi ll- ment process. Must possess current SalesForce Administrator and Sales- currently has multiple openings in Force Developer Certifi cations. Apply to: Menlo Park, CA Laurie Shuckerow, Job ID BPAS-RP, Ever- source Energy, P.O. Box 270, Hartford, CT (various levels/types): 06141-0270. Software Engineer (6288J) Create web and/or mobile applications that reach over one billion people, and build high-volume servers to support our content. Microsoft Corporation currently has the following opening in Ft. Lauderdale, FL.

Mail resume to: Program Manager II. Coordinate pro- WhatsApp, Inc. Attn: AA-USIM, gram development of computer software applications, systems or services, work- 1 Hacker Way, Menlo Park, ing with development and product plan- CA 94025. ning teams. https://jobs-microsoft.icims. com/jobs/11324/go/job Must reference job title & job code To view detailed job descriptions and min- shown above, when applying. imum requirements, and to apply, visit the website address listed. EOE.

www.computer.org/computingedge 59 CAREER OPPORTUNITIES

BLACKBERRY CORPORATION seeks apps &embedded software for psgr ve- embedded software for infotainment Sr. Software Test Engineer in San Diego, hicle infotainment ECUs for speech rec- ECUs or smartphones on Android OS CA who will be responsible for sw quality ognition, audio, media &projection on platform. Mail resume to Ref#2190-201, assurance & production troubleshoot- Android OS platform, using C/C++ &Java GM Global Mobility, 300 Renaissance ing. Refer to Req#BB3 & mail resume to programming languages, &Android SDK, Center, MC:482-C32-C66, Detroit, MI BlackBerry Corp., P.O. Box 141394, Irving, Android Studio, Eclipse, &Android Debug 48265. TX 75014. Bridge (adb) tools &Serial Terminal protocol. Design &dvlp Android Framework BLACKBERRY CORPORATION has the QUALITY ASSURANCE SYSTEMS ENGI- &HAL modules using Java, C &C++ pro- following openings in Mountain View, NEER Position available in New York, NY. gramming languages to support vehicle CA: Principal SW Test Eng (Req#BB4) Manage the testing of advanced financial features incldg speech recognition, au- Des. & dev. sw for secured mobile apps & services applications utilizing tools such dio, media &projection. Dvlp software, web-based enterprise services. Principal as Mocha, PhantomJS, and JavaScript. debug &fix issues in the Android Frame- SW Eng (Req#BB5) Des., dev., trouble- Create test plans, test data, and review work incldg AudioManager, AudioSer- shoot & debug sw enhancements & new and analyze results. Define test automa- vice, AudioPolicyManager, AudioFlinger, products. Refer to Req# & mail resume to tion strategy, create and execute regres- mediaserver &OpenSLES. Dvlp &fix is- BlackBerry Corp., P.O. Box 141394, Irving, sion tests, and perform, adhoc, security, sues in DomainServices for speech rec- TX 75014. user interface, and integration testing. ognition, audio &media as part of Android Prioritize quality assurance issues. De- Framework system server. Validate dvlpd velop and maintain quality assurance software performing unit test using JUnit SOFTWARE DEVELOPMENT ENGINEER guidelines and process documentation. >est to ensure high qlty bug free soft- IN TEST sought by Pactera, Inc. in Red- Direct applications to: Recruiting Oper- ware. Create &execute unit, integration, mond, WA. Design, develop, test and ations, Massachusetts Mutual Life Insur- system, load &acceptance test plans deploy computer applications, test auto- ance Company, 1295 State Street, Spring- &scripts. Use repo, git &gerrit configura- mation frameworks, databases, and ser- field, MA 01111; Please Reference Job ID: tion management tools for maintaining vices. Use SSIS technologies to extract 2362790 history &traceability of code changes. & compute data. Des. & dev. Multi. data- Bachelor, Computer Science, Computer bases using SSAS & OLAP technologies. SENIOR SOFTWARE ENGINEER, War- Science and Engrg, Computer Engrg, or Req BS+2 yrs exp. To apply send resume ren, MI, General Motors. Analyze reqmts, Electronics Engrg. 60 mos exp as Engi- to [email protected] and design, dvlp, validate &debug Android neer or Developer, designing &dvlpg reference Job# M0511039.50.

TECHNOLOGY SOFTWARE SOFTWARE HomeAway.com, Inc. HomeAway.com, Inc. Hotwire, Inc. has openings for following has openings for has openings for positions in Austin, TX: Software Engineers SOFTWARE SOFTWARE (Job ID#: 728.4961) ENGINEERS Design, implement, and debug software ENGINEERS TEST for computers including algorithms and (Job ID#: 728.5229) (Job ID#: 728.5613) data structures. in Bellevue, WA: in San Francisco, CA:

Staff ERP Database Engineers Design, implement, and debug Code moderately complex tests to (Job ID#: 728.4190) software for computers including implement a test design, set up test Implement and enhance the growing algorithms and data structures. automation and execution frame- Oracle applications footprint for the work. company. To apply, send resume to: Home- To apply, send resume to: Hotwire Away Recruiting, 333 108th Avenue To apply, send resume to: HomeAway Recruiting, 333 108th Avenue NE, NE, Bellevue, WA 98004. Must Recruiting, 333 108th Avenue NE, Bellevue, Bellevue, WA 98004. Must reference WA 98004. Must reference Job ID#. reference Job ID#. Job ID#.

60 ComputingEdge January 2019 CAREER OPPORTUNITIES

RESEARCHER, Warren, MI, General Engineer, researching &dvlpg algorithm dvlpmt on Jenkins. Mentor new mem- Motors. Research, dvlp &implement in- solutions to create psgr vehicle cloud bers of the team. Design &implement novative &creative algorithm solutions computing system architecture, us- embedded apps using C, C++ &debug to create new psgr vehicle cloud coming multi-modal data fusion &machine on target hardware. Master, Computer or puting system architecture, using multi- learning, for collected vehicle systems Electrical Engrg. 12 mos exp as Engineer, modal data fusion &machine learning, &active safety related apps incldg V2V Developer, or related, designing &dvlpg for collected vehicle systems &active &V2X in Android or Linux platforms. Mail telematics &infotainment OTA program- safety related apps incldg V2V &V2X resume to Ref#2606, GM Global Mobil- ming master framework for psgr vehicle in Android &Linux platforms, Java pro- ity, 300 Renaissance Center, MC:482- platforms using C/C++ on QNX &Linux gramming &MATLAB, Python &SPARK C32-C66, Detroit, MI 48265. platforms in CGM or TCP. Mail resume to tools. Design control algorithms &soft- Ref#3210-204, GM Global Mobility, 300 ware for complex systems. Create SOFTWARE ENGINEER, Warren, MI, Renaissance Center, MC:482-C32-C66, &utilize MATLAB models for algorithm General Motors. Design &dvlp telematics Detroit, MI 48265. definitions &utilize plant models for ver- &infotainment OTA programming master framework for global psgr vehicle plat- ification of algorithm &control features. FILEMAKER INC. has multiple positions Dvlp cloud computing system using forms using C/C++ on QNX &Linux plat- open for the following in Santa Clara, sensing &fusion of data from multi- forms in Communication Gateway Mod- CA: Software Engineer Applications modal vehicle sensors (speed, tire ro- ule (CGM) &Telematics Communication (REQ#AVE36M) Desgn & devlop e-com- tation, acceleration, &yaw rate) &smart Platform (TCP). Implement UDS diagnos- mrce SW. Refer REQ# & mail resume to: phone sensors (gyroscope &inertial) to tics over Ethernet (DoIP &Socket Adapter) FileMaker Inc., ATTN: CP, 5201 Patrick assess behavior of driver &the context &CAN using C/C++. Design &dvlp ECU Henry Drive, Santa Clara, CA 95054. File- of driving conditions. Research, dvlp updates &debug on target hardware. Maker is an EOE/AA m/f/disability/vets. &integrate multi-modal vehicle-based Perform static code analysis using Co- sensors incldg DSRC-based V2X. As- verity &Parasoft. Perform software con- sess situation awareness &collisions figuration management using Git, &code VLOCITY seeks Software Engineer, potential with objects. Dvlp reqmts, review using Gerrit. Integrate &test soft- Engineering Productivity in SF, CA to criteria, &evaluation methods for vehi- ware on vehicle bench. Provide solutions dev internal tools used by co. engineer- cle-based sensing technologies incldg to Ethernet network configuration incor- ing., testing, & build teams. Ref Job ID: cameras. PhD., Computer Science, Elec- porating vehicle architecture &vehicle AGM3N5 & send res to L. D. Kingsley at trical or Computer Engrg. 6 mos exp as IP network reqmts. Build automation [email protected]

TECHNICAL TECHNOLOGY CONSULTANT Oracle America, Inc. Oracle America, Inc. Oracle America, Inc. has openings for has openings for has openings for TECHNICAL APPLICATION CONSULTANT ANALYST DEVELOPER positions in Atlanta, GA. positions in Arlington, VA. positions in Bedford, MA. Job duties include: Analyze requirements Job duties include: Analyze, design, and deliver functional and technical Job duties include: Deliver solutions develop, troubleshoot and debug solutions. Travel to various unanticipated to the Oracle customer base while software programs for commercial or sites throughout the United States end-user applications. Write code, serving as an advocate for customer required. needs. complete programming and perform testing and debugging of applications. Apply by e-mailing resume to Apply by e-mailing resume to Apply by e-mailing resume to [email protected], [email protected], [email protected], referencing 385.21560. referencing 385.21488. referencing 385.22307. Oracle supports workforce diversity. Oracle supports workforce diversity. Oracle supports workforce diversity

www.computer.org/computingedge 61 CAREER OPPORTUNITIES

EPAM SYSTEMS, INC. seeks all levels Security Systems Engineer, design, im- Rhapsody &EA tools for design &devel- for the following positions in Newtown, prove ITIL/ITSM & SDLC (SCSY0818). opment, &Jenkins for automated back- PA and various long term unanticipated Mail resume to Marina Maizet, EPAM, 41 end continuous integration. Use Renesas worksites. Account Manager, strategize University Drive, Suite 202, Newtown, PA 32bit controller as hardware platform. progs to meet cl goals (AM0818); Applica- 18940. Must ref job code. Bachelor, Computer Science, Computer tion Support Analyst, create process docs Engrg, Computer Science &Engrg, or (ASA0818); Application Support Manager SENIOR EMBEDDED SOFTWARE ENGI- Electrical Engrg. 60 mos exp as Engineer, (ASM0818); Business Analyst, create NEER, Warren, MI, General Motors. Engr, Developer, or related, delivering audio, req’s & specs (BA0818); Business Anal- perform, dvlp, using C, C++, &Assembly calibrations, camera, system process ysis Manager (BAM0818); Data Anaylst languages, &execute VIP logic control- communication, &integrating system (DA0818); Database Developer, analyze, ler for psgr car, truck &SUV Center Stack &hardware peripherals, HMI framework, design, maintain DBs (DD0818); Delivery Module to meet nextgen automotive navigation, display, storage, &software Manager, deliver sw, arch. (DM0818); IT infotainment systems &in vehicle com- update functionality to dvlp embedded Manager, mng local IT prog (ITM0818); munications reqmts in compliance with software using Android, Java, &C/C++. Project Administrator (PA0818); Proj- ISO26262 AUTOSAR standards. Deliver Mail resume to Ref#5891, GM Global Mo- ect Manager, mg IT dev projs (PM0818); audio, tuner, diagnostics, calibrations, bility, 300 Renaissance Center, MC:482- Software Developer (SD0818); Software camera, vehicle components inter pro- C32-C66, Detroit, MI 48265. Engineer (SE0818); Software Engineer- cess &intra process communication, &in- ing Manager, proj coord. & integration tegrate system &hardware peripherals, SOFTWARE ENGINEER - AUTOMATED (SEM0818); Software Test Automation HMI framework &applications, telematics, DRIVING - ADVANCED PROJECTS, Engineer, create auto test plns (STA0818); vehicle data, navigation, display, chimes, Warren, MI, General Motors. Create, Software Test Automation Manager power mode &system states, storage, write, debug, &test software using C/ (STAM0818); Software Testing Engineer, memory, verified boot, software update C++, Python, MATLAB, Simulink, incldg test & QC sw (STE0818); Software Test- functionality to dvlp best in-class em- toolboxes, Integrated Development En- ing Manager (STM0818); Solution Archi- bedded software using Android, Java, C/ vironment (IDE), compilers, debuggers, tect, prod sol dev & arch. (SA0818); Sys- C++, &Kotlin. Engr, design, dvlp, release build system, libraries &frameworks, in tems Engineer, design sw prod environs &continuously improve psgr vehicle ECU Linux &Windows to ensure safe &proper (SYE0818); Experience Designer, user re- &software architecture reqmts &robust- functioning of safety critical autonomous search & testing (ED0818); Performance ness using DOORS for reqmts handling, vehicles incldg LIDAR, RADAR, high pre- Analyst, test sw performance (PFA0818); Vector DaVinci Configurator &Developer, cision camera, &IMU sensors. Work on complex problems involving Coordinate Transformations, Object Perception (in- SOFTWARE TECHNICAL cludes images, point clouds, segmentation, registration), Mapping &Localization, Oracle America, Inc. Oracle America, Inc. Path Planning, Obstacle Avoidance, Ob- ject Tracking, &Vehicle Control. Use Ag- has openings for has openings for ile dvlpmt utilizing code repository, code review, work &issue tracking tools including Git/Bitbucket and RTC. Study existing SOFTWARE technologies, tools, processes &algo- TECHNICAL rithms. Implement algorithms &software to fit use case &compare results to find DEVELOPER best approach to solve problems. Design positions in Seattle, WA. ANALYST test strategies, test &debug implemented software. Conduct code review to make positions in Orlando, FL. sure code follow coding standards. Col- Job duties include: Design, develop, trouble- lect required data for software dvlpmt. Study &create mathematical &machine shoot and/or test/QA software. As a learning model of system for anlys. Inte- member of the software engineering Job duties include: Deliver solutions to grate resulting software &model into advanced project dvlpmt vehicles, SIL &HIL division, apply knowledge of software the Oracle customer base while serving simulators. Master, Computer Engrg, architecture to perform tasks associated as an advocate for customer needs. Electrical Engrg, Computational Sciences &Robotics, or Computer Science. 12 mos with developing, debugging, or designing Offer strategic technical support to exp as Engineer, creating, debugging assure the highest level of customer &testing software using C/C++ &Python, software applications or operating systems incldg IDE, compilers, debuggers, build according to provided design specifications. satisfaction. system, libraries &frameworks, in Linux &Windows to ensure safe &proper functioning of safety critical robot or autono- Apply by e-mailing resume to Apply by e-mailing resume to mous vehicle with high precision camera, [email protected], or related. Mail resume to Ref#269, GM [email protected], Global Mobility, 300 Renaissance Center, referencing 385.22229. referencing 385.225444. MC:482-C32-C66, Detroit, MI 48265. Oracle supports workforce diversity. Oracle supports workforce diversity.

62 ComputingEdge January 2019 CAREER OPPORTUNITIES

IT ARCHITECT, General Motors, Detroit, stacks (REST &XML) &technologies incldg js, MongoDB, Redis, Memcached, and AWS MI. Design, dvlp, &validate IT systems on Java, Python, &Pega. Analyze &map user services such as EC2, VPC, Route53, S3, Enterprise Decision Management (EDM) reqmts to existing system components Elasticache, Work Steps, and Lambda func- business decisions platform using Pega &identify component updates necessary tions. Analyze, design, and review integration decisioning management system &Agile to fulfill user needs. Analyze existing data with third-party providers such as Milestone methodology. Create &plan current &fu- against new reqmts &identify gaps. Mas- VMS, S2, Salto, and Noke, using knowledge ture enterprise data analytics. Dvlp &im- ter, Computer Engrg, Computer Science, of programming languages including Ja- prove consumer customer experience in Electrical Engrg, or related. 6 mos exp as vascript, C#, Python, and PHP, in operating using distributed mobile apps (personal- Engineer, Systems Analyst, IT Architect, system environments including Linux and ized, smart driver, car sharing, vehicle to or related, dvlpg &architecting distrib- Windows. Requirements: Employer will ac- vehicle, vehicle to infrastructure, vehicle uted, component-based, service-oriented cept a Bachelor’s degree or equivalent in maintenance), component-based infotain- solutions for IT organization, or related, Computer Science, Technology, Computer ment features, &psgr vehicle telematics, applying SOA &web service stacks Engineering, or a related technical field, and advanced safety &autonomous vehicle (REST &XML) &technologies incldg Java 24 months of experience in the job offered features. Dvlp &validate IT application &Python. Mail resume to Ref#1890, GM or in a Software Engineer-related occupa- &infrastructure architectures applying Global Mobility, 300 Renaissance Center, tion. To Apply: You MUST mail resume to commercial &open-source web server MC:482-C32-C66, Detroit, MI 48265. 8308 Sterling Street, Irving, TX 75063, ref- application server platforms incldg Ora- erencing Job Code 6945-15. Include com- cle Enterprise Linux &Windows Enterprise SENIOR SOFTWARE ENGINEER sought plete contact information (including e-mail, Server. Dvlp &improve enterprise middle- by Petrocloud, LLC in Irving, TX. Duties day/evening phone, and mailing address) ware (w message oriented middleware) include: Lead research, development, vali- on resume/application. incldg Pivotal Cloud Foundry. Dvlp &im- dation and implementation efforts for next prove transaction processing monitors, generation technology, software and ap- FLEX LOGIX TECHNOLOGIES, INC. is re- calculating infrastructure reqmts, &using plications for machine-to-machine (“M2M” cruiting for our Mountain View, CA office: Cassandra &Hyperion database archi- or “Internet of Things”) devices enabling Hardware Design Engineer: Perform dig- tectures, &security systems incldg single remote automation, control and monitoring ital & circuit design of EFLX (embedded sign-ons. Contribute to OOAD. Dvlp &ar- for energy and critical infrastructure cus- FPGA) cores in 40nm, 28nm, 16nm,14nm, chitect distributed, component-based, tomers. Develop web-based software and & 7nm. Mail resume w/job code #38960 service-oriented solutions for GCCX/) applications, interfaces, and endpoints to to Flex, Attn.: HR, 2465 Latham St., Ste. organization, applying SOA &web service provide a complete solution using Node. 100, Mountain View, CA 94040.

TECHNOLOGY TECHNOLOGY SOFTWARE Oracle America, Inc. Oracle America, Inc. Oracle America, Inc. has openings for has openings for has openings for USER EXPERIENCE IT SOFTWARE DEVELOPERS MANAGER DEVELOPER positions in Redwood Shores, CA. positions in Austin, TX. positions in Bellevue, WA.

Job duties include: Create, evaluate, and Job duties include: Define, design, and Job duties include: Design, develop, modify design and prototypes to support implement network communications and troubleshoot and/or test/QA software. evolving hardware and software applica- solutions within a fast-paced, leading edge Travel to various unanticipated sites tion development. Travel to various database/applications company. Perform throughout the United States required. unanticipated sites throughout the performance trend analysis and manage the United States required. server/network capacity. Apply by e-mailing resume to Apply by e-mailing resume to [email protected], Apply by e-mailing resume to [email protected], referencing 385.22197. [email protected], referencing 385.21310. referencing 385.21578. Oracle supports workforce diversity. Oracle supports workforce diversity. Oracle supports workforce diversity.

www.computer.org/computingedge 63 CAREER OPPORTUNITIES

TECHNOLOGY Intuit Inc. has openings in Mountain View, California for Product Managers (Job code: I-443) Design and bring to market revenue generating, customer-driven software products and services that deliver great customer experiences.

Intuit Inc. has openings in Reno, NV for Senior Software Engineers (Job code: I-4370) Exercise senior level knowledge in selecting methods and techniques to design, implement, modify and support a variety of software products.

Intuit Inc. has openings in Plano, TX for Software Engineers (Job code: I-3102) Apply software development practices to design, implement, and support individual software projects.

To apply, submit resume to Intuit Inc., Attn: Olivia Sawyer, J203-6, 2800 E. Commerce Center Place, Tucson, AZ 85706. You must include the job code on your resume/cover letter. Intuit supports workforce diversity.

TECHNOLOGY TECHNICAL CONSULTANT Oracle America, Inc. Oracle America, Inc. has openings for Oracle America, Inc. has openings for SYSTEMS has openings for ANALYST TECHNICAL CONSULTANT (Oracle Technical Support) WRITER positions in Atlanta, GA. positions in Redwood Shores, CA. positions in Santa Clara, CA. Job duties include: Provide technical support Job duties include: Analyze requirements to client personnel who are diagnosing, Job duties include: Create, develop, plan, and deliver functional and technical write, and edit operational, instructional, troubleshooting, repairing and debugging solutions. Implement products and Oracle Engineered Systems, computer maintenance, test or user manuals for systems, complex software, or networked paper, multimedia or web-based publica- technologies to meet post-sale customer and/or wireless systems. Travel to various tions. needs. unanticipated sites throughout the United States required. May telecommute from Apply by e-mailing resume to Apply by e-mailing resume to home. [email protected], [email protected], Apply by e-mailing resume to referencing 385.20061. referencing 385.21777. [email protected], referencing 385.22309. Oracle supports workforce diversity. Oracle supports workforce diversity. Oracle supports workforce diversity.

64 ComputingEdge January 2019 CAREER OPPORTUNITIES

TECHNOLOGY LinkedIn Corp. has openings in our Mountain View, CA location for: Software Engineer (All Levels/Types) (SWE1218MV) Design, develop & integrate cutting-edge software technologies.

LinkedIn Corp. has openings in our Sunnyvale, CA location for: Software Engineer (All Levels/Types) (SWE1218SV) Design, develop & integrate cutting-edge software technologies; Data Scientist (6597.943) Work with a team of high-performing analytics, data science professionals, & cross-functional teams to identify business opportunities & optimize product performance or go-to-market strategy; Data Scientist (6597.2702) Manipulate massive-scale structured & unstructured data to drive actionable insights. Systems Architect, Global Sales Comp. Reporting & Insights (6597.2975) Work with business stakeholders & IT to plan, design, code, test, & roll-out an enterprise scale Data Warehouse & Business Intelligence Solution; Site Reliability Engineer (6597.2543) Ensure that LinkedIn’s complex, web-scale systems are healthy, monitored, automated, & designed to scale.

LinkedIn Corp. has openings in our San Francisco, CA location for: Software Engineer (All Levels/Types) (SWE1218SF) Design, develop & integrate cutting-edge software technologies; Staff Network Engineer (6597.1289) Design, engineer, & deploy massive, & scalable datacenter & large-scale backbone & edge-pop infrastructure.

LinkedIn Corp. has openings in our New York, NY location for: Software Engineer (All Levels/Types) (SWE1218NY) Design, develop & integrate cutting-edge software technologies.

Please email resume to: 6597@linkedin.com. Must ref. job code above when applying.

TECHNOLOGY Oracle America, Inc. has openings for APPLICATIONS DEVELOPER From the analytical engine to the positions in Arlington, VA. supercomputer, from Pascal to von Neumann, IEEE Annals of the History of Job duties include: Analyze, design, Computing covers the breadth of computer develop, troubleshoot and debug history. e quarterly publication is software programs for commercial or end-user applications. May telecommute an active center for the collection and from home. dissemination of information on historical projects and organizations, oral history Apply by e-mailing resume to activities, and international conferences. [email protected], referencing 385.22342.

www.computer.org/annals Oracle supports workforce diversity.

www.computer.org/computingedge 65 CAREER OPPORTUNITIES

Apple Inc. has multiple positions available in Cupertino, CA. Operations Engineering Pro- fnctalty of iOS dvcs. Travel Req deploy data warehouse & bus gram Coordinator (REQ#AZ- 20%. intelligence analytics solutions ESVB) Implmnt Mnfctrng Dsgn for mult bus group. Software Engineer Systems for new Apple prdcts by wrkng (REQ#AHX35N) Authr sensr IST Technical Project Coor- with Prdct Dsgn Engineering tech instrmntation engg req dinator (REQ#AUY7WS) Co- Team. Travel Req’d 33%. specfctions. ordnate project teams to create Software Engineer Applications new prodcts & release prodct Engineering Project Specialist (REQ#AR4P5X) Supprt big data enhancements. (REQ#AJW3RE) Implmnt var- operations & engineering for ious IT Srvice (ITSM) modules Software Development Engineer Apple. like incident, problem, request, (REQ#9SYTRP) Dsgn & dev Systems Design Engineer change, & SLA. web & iOS apps aimed at im- (REQ#AEAW26) Design HW prvng SW quality at Apple. Software Quality Assurance En- test flow & coverge for Apple’s gineer (REQ#AG6QBG) Assist Software Development Engineer Macbook prdcts including HW, & guide in qualifyng frmewrk (REQ#9WXR67) Rsrch, dev & SW & mech design. & dvlping automatn usng this test Comp Vision SW systems. Software Engineer Applications frmwrk. Software Development Engineer (REQ#ANU289) Dsgn & dvlop Software Development Engineer (REQ#AGFTEC) Plan, design & highly scalble server applctns & (REQ#AVZMY8) Rspnsble for develop SW prototypes. srvcs. dvlpg the framewk that enables Software Development Engineer Software Development Engi- home accessories, iOS devices, (REQ#AHWSVS) Dsgn & im- neer (REQ#AQL3S9) Dvlp SW & apps to wk together to bld an plmntn of grphcs drvr SW. for Input/Output tech. Apple exp in the Home. Software Development Engineer Software Development Engi- (REQ#9UA4F9) Rsrch, dsgn, neer (REQ#ATK8UP) Dsgn & dvlp, implement, & debug com- dev tests & frmewrks to validate Refer to REQ# & pilers targeting current & future functnlty, stability & perfrmnce mail resume to: Apple Inc., GPUs. of HomeKit Access’ries w/ iOS. ATT N: D.W., Software Development Manager Software Development Engi- 1 Infinite Loop 104-1GM, (REQ#9P2VRW) Dsgn & dev neer (REQ#AHWRKU) Apply Cupertino, CA 95014. SW for mobile devices & desk- knowledge of AI to apps like Siri Apple is an EOE/AA m/f/ top comps. & search. disability/vets. Machine Learning Engineer Software Engineer Applications (REQ#AZN3JG) Design & de- (REQ#AYXRVB) Resp for Data velop search solution for maps. Vol Mgmt for lrg SAP entrprs sys. Hardware Development Engi- Apple Inc. has neer (REQ#ANC5TE) Dsgn, Hardware Development Engi- validate, & characterize high neer (REQ#9WXRHH) Rsrch, multiple positions speed board dsgns. prototype & implmnt algrthms available in for Cmptr Vision, fcsing on Software Engineer Applications multi-modal (sensor fusion) 2D (REQ#ACY24F) Desgn & dvlop Cupertino, CA. & 3D calibration, object detec- SW for big data analytics sys. tion, & multi-object trckng. Firmware Engineer (REQ# Systems Design Engineer AAPTDH) Dsgn & dvlp SW for Software Engineer Applications (REQ#APP487) Tst cllulr tlphny real time embd systms for wrlss (REQ#AT9T9T) Dsgn, dvlp, & comnctns.

66 ComputingEdge January 2019 CAREER OPPORTUNITIES

Systems Design Engineer (REQ# Software Engineer Applications ADRNWN) Design antennas Apple Inc. has (REQ#AUU37Y) Anlyze, de- for mobile communication de- multiple positions sign, devlop, test & implment vices. Travel Req 10 % next gen Apple custm built en- available in Elk terprise HR Capital Manage- Software Development Engineer ment Sys to meet Apple’s HR (REQ#9UD4HR) Design, devlp, Grove, CA: global bus sys reqs. implmnt & debug statistical & deterministic natural language Unix System Administrator ASIC Design Engineer (REQ# processing SW for novel Chinese (REQ#ACAUM7) Install & co- AHHS6G) Rsponsible for block textual input systms. ordinate Apple’s manufctring level place & route of a dsgn to critical systms. meet all timing, area, power, & Hardware Development Engi- other constraints. neer (REQ#9TU2C5) Dsgn & dev novel sensing tech for Ap- Hardware Development Engi- ple’s consumer electronics port- neer (REQ#AN5S8M) Respon- folio spanning proof of concept Refer to REQ# & sible for AC/DC power supply eval thru to mass production. mail resume to: Apple dsgn & development. Travel Req 15%. Inc., ATTN: D.W., Machine Learning Engineer 1 Infinite Loop 104-1GM, ASIC Design Engineer (REQ# (REQ#A4RQCN) Conceive & Cupertino, CA 95014. AFYTZ5) Collab w/ architec- design end to end data science Apple is an EOE/AA m/f/ ture team to define, analyze, & solutions to support Apl’s busi- disability/vets. doc micro-architecture specs of ness units. the HW dsgn blocks. Software Development Engineer (REQ#9U3673) Design and dev core operating sys functionality. Apple Inc. has Refer to REQ# & Software Development Manager mail resume to: Apple Inc., (REQ#9QYPAC) Mng, schdle & multiple positions ATT N: D.W., delegate work to direct reports. 1 Infinite Loop 104-1GM, available in Austin, Cupertino, CA 95014. TX: Apple is an EOE/AA m/f/ disability/vets. ASIC Design Engineer (REQ# Refer to REQ# & mail AGNTJT) Prpose & dsgn in- resume & transcript(s) to: novative solutions to debug & Apple Inc., ATTN: D.W., prform verification of complex 1 Infinite Loop 104-1GM, HW chips. Cupertino, CA 95014. ASIC Design Engineer (REQ# Apple is an EOE/AA m/f/ AYU6CN) Rspn for all aspcts of disability/vets. timing incldng wrkng w/dsgners for timing chnges, helpng con- strct/modfy flows, timing anly- sis & timing closure. ASIC Design Engineer (REQ# AW75KV) Collab w/ architecture team to define, analyze, & doc micro-architecture specs of the HW dsgn blocks.

www.computer.org/computingedge 67 CAREER OPPORTUNITIES

Help build the next generation of systems behind Facebook’s products. Facebook, Inc. currently has multiple openings in Menlo Park, CA (various levels/types):

Software Engineer (13195J, 12742J, 5555J) Create web and/or mobile applications that reach over one billion people, and build high volume servers to support content. Audio Software Engineer (135239J) Design, develop, and test audio capture and render pipeline that may span across DSP and Host processor. Optical Component Engineer (134499J) Create test and validation plans for optical modules in Facebook network hardware. Manager, Reliability Engineering (10938J) Responsible for overseeing the reliability of all virtual reality products and hardware. Position requires some international travel. Engineering Manager (3446J) Man- age engineers working with the engineering teams to build products and/or infrastructure required to support the products at scale. Sys- tems Engineering Manager (6519J) Manage engineers working with the engineering teams to build, scale, and secure Facebook’s corporate infrastructure and systems software. Technical Program Manager (135644J) Manage cross-functional infrastructure security engineering projects in a flat, matrix organization. Data Engineering Manager (134781J) Proactively drive the vision for the BI and Data Warehousing across a product vertical, and define and execute on a plan to achieve that vision. Data Center Infrastructure Management Engineer (135132J) Monitor and analyze Facebook’s global data center footprint and underlying facility operations network, which is critical to ensuring we can monitor the power, cooling, and environmental meters. Domestic travel required. Technical Program Manager (12904J) Contribute to projects and cross-functional technical programs by working with development teams, product teams, and external partners. Domestic travel required. Network Engineer (11952J) Deploy and support Facebook’s global production and corporate network including edge, backbone, datacenter, campus, and content delivery network infrastructure. Domestic and international travel required. Computer Vision Engineer (9994J) Develop creative computer vision and track software for a variety of Oculus products. IP Integrity Analyst (136109J) Apply your expertise in quantitative business analysis, data mining, and the presentation of data to identify trends and opportunities to combat abuse in the IP (Intellectual Property) space. Edge Strategy Manager (133559J) Lead network planning efforts with ISPs in Latin America for interconnection and caching, including building long- term relationships and capacity plans. Position requires at least 25% international travel. Must be fluent in Spanish.

Openings in Seattle, WA (multiple openings, various levels/types):

Data Engineer (135140J) Design, build, and launch data pipelines to move data across systems and build the next generation of data tools that generate business insights for a product. Research Scientist (12885J, 9736J) Research, design, and develop new optimization algorithms and techniques to improve the efficiency and performance of Facebook’s platforms.Software Engineer (4840J) Create web and/or mobile applications that reach over one billion people, and build high volume servers to support content.

Openings in New York, NY (multiple openings, various levels/types):

Research Scientist (133884J) Research, design, and develop new optimization algorithms and techniques to improve the efficiency and performance of Facebook’s platforms.

Openings in Austin, TX (multiple openings, various levels/types): Business Integrity Associate (6818J) Investigate and define clear business problems and prioritize solutions using data-driven analytics, developing success metrics to measure project performance and alignment with team mission and goals.

Mail resume to: Facebook, Inc. Attn: AA-USIM, 1 Hacker Way, Menlo Park, CA 94025. Must reference job title & job code shown above, when applying.

68 ComputingEdge January 2019 CAREER OPPORTUNITIES

TECHNOLOGY Expedia, Inc. has openings for the following positions in Bellevue, WA (various/levels/types): Software Engineers (Job ID#: SW1218): Design, implement, Faculty Position and debug software for computers including algorithms and data structures. Technical Compliance Managers (Job ID#: The Electrical and Computer Engineering Department 728.4942): Handle full lifecycle of medium to complex cross of Baylor University seeks faculty applicants for a Clinical Assistant/Associate Professor. Any area of functional IT and security risk/compliance projects. Database expertise in ECE will be considered. Applicants must Administrators (Job ID#: 728.2592): Provide Database Admin- possess an earned masters or doctorate degree and extensive industry experience, and demonstrate istrator support in a growing, and challenging mission critical potential for excellent teaching; applicants for OLTP eCommerce database environment. Security Engineers Clinical Associate Professor must additionally present evidence of deeper industry experience and (Job ID#: 728.2016): Provide cyber security expertise in the achievement in teaching commensurate with the analysis, assessment, development, and evaluation of security desired rank. The ECE department offers B.S., M.S., M.E. and Ph.D. degrees and is rapidly expanding its solutions and architectures. faculty size. Facilities include the Baylor Research and Innovation Collaborative (BRIC), a newly-established To apply, send resume to: Expedia Recruiting, research park minutes from the main campus. 333 108th Avenue NE, Bellevue, WA 98004. Chartered in 1845 by the Republic of Texas, Baylor Must reference Job ID#. University is the oldest university in Texas. Baylor has an enrollment of over 15,000 students and is a member of the Big XII Conference. Baylor’s mission is to educate men and women for worldwide leadership and Cisco Systems, Inc. service by integrating academic excellence and is accepting resumes for the following Christian commitment within a caring community. The department seeks to hire faculty with an active positions: Christian faith; applicants are encouraged to read about Baylor’s vision for the integration of faith and NEW YORK, NY: Systems Engineer (Ref. #NY110D): learning at www.baylor.edu/profuturis/. Provide business-level guidance to the account team or Applications will be considered on a rolling basis operation on technology trends and competitive threats, starting January 1, 2019. Applications must include: both at a technical and business level. Telecommuting 1) a letter of interest that identifies the applicant’s Permitted. anticipated rank, SAN FRANCISCO, CA: CNG Staff (SF470D): 2) a complete CV, Charter and implement the Strategic Buffer Program for all hardware critical components. CNG Management 3) a concise statement of teaching interests, (SF472D): Facilitate the team’s ability to work on customer 4) the names and contact information for at least cases from initial creation to resolution. three professional references. SAN JOSE/MILPITAS/SANTA CLARA, CA: Additional information is available at www.ecs. baylor.edu. Should you have any questions on the Hardware Engineer (Ref.# SJ050D): Responsible for position, feel free to contact the search chair, Dr. Keith the specification, design, development, test, enhancement, Schubert at [email protected]. Upload and sustaining of networking hardware. materials via Baylor’s iApply system accessible at www.baylor.edu/facultypositions PLEASE MAIL RESUMES WITH REFERENCE NUMBER TO CISCO SYSTEMS, INC., ATTN: G51G, 170 W. Tasman Baylor University is affiliated with the Baptist General Drive, Mail Stop: SJC 5/1/4, San Jose, CA 95134. No Convention of Texas. As an Affirmative Action/ Equal Employment Opportunity employer, Baylor phone calls please. Must be legally authorized to work encourages candidates of the Christian faith who in the U.S. without sponsorship. EOE. are minorities, women, veterans, and persons with www.cisco.com disabilities to apply.

www.computer.org/computingedge 69 TECHNOLOGY Visa U.S.A. Inc., a Visa Inc. company, currently has openings in our Foster City, CA location for: Directors (Job# REF14173A) to be responsible for delivering Visa’s consumer credit products and Visa’s credit market knowledge and insights to our clients as part of the Consumer Credit Product team. Travel to various unanticipated sites within the US required. Marketing Managers (Job# REF14176W) to be responsible for the overall digital marketing analytics process, organizing data which delivers information, insight, and recommendations to monitor, measure, and optimize our online marketing efforts. Some domestic and international travel (up to 30%) may be required to work on projects at various, unanticipated sites. Staff Software Engineers (Job# REF14272I) to engage in technical design of solution that is based on use cases and business requirements. Senior Directors, Compliance (Job# REF14266O) to work across multiple functions such as Compliance, Human Resources, Internal Audit, Legal, Risk Management, and Corporate Relations to support ethical business decisions consistent with Visa’s Code and company values. Sr. Product Managers (Job# REF14537J) responsible for overall global program management and implementation of the CTA (Central Travel Account) program across all geographies. Ensure Issuers and Clients working with internal and external teams can meet critical CTA business needs. Some travel (10-20%), may be required to work on projects throughout the United States. Project Management / Program management certification required.

Openings in our San Francisco, CA location: Staff Designers (Job# REF14284C) to conceive and design experiences that are delightful and engaging. Some travel (less than 50%) required to work on projects at various, unanticipated sites throughout the United States.

To apply, please reference Job#s above when mailing resume to: LJ, Visa, Inc., MS: M1-12 SW, 900 Metro Center Blvd., Foster City, CA 94404. EOE

CyberSource Corporation, a Visa Inc. company, currently has openings in our Austin, TX location for: Staff Software Engineers (Job# REF14166A) to be responsible for the application development deliverables that include architecting, designing, coding, and unit testing. Staff Software Engineers (Job# REF14175U) to design, enhance and build transactional processing engine in an agile development environment. Staff Software Engineers (Job# REF14237A) to collaborate with business and technical staff to understand business requirements for use in designing appropriate solutions. Senior Software Engineers (Job# REF13472R) to be responsible for defining requirements, architecting, and implementing software solutions to support business requests. Senior Software Engineers (Job# REF14163M) to design, enhance and build solutions for customers in an agile, fast-paced environment. Senior Software Test Engineers (Job# REF14507A) Design and implement test solutions for the Merchant & Acquiring Processing department. Implement test solutions through to production with high quality and complete all testing phases following an Agile methodology.

Openings in our Bellevue, WA location: Senior Software Engineers (Job# REF13430W) to write software code that complies with design specifications and meets security and .Net best practices. To apply, please reference Job#s above when mailing resume to: LJ, Visa, Inc., MS: M1-12 SW, 900 Metro Center Blvd., Foster City, CA 94404. EOE

70 ComputingEdge January 2019 TECHNOLOGY Visa Technology & Operations LLC, a Visa Inc. company, currently has openings in our Austin, TX location for: Staff Software Engineers (Job# REF13509J) to participate in the design and development of new programs and subprograms, as well as enhancements, modifications, and corrections to existing software. Senior Software Engineers (Job# REF14274D) to work with product owners to understand the desired application capabilities and testing scenarios. Lead Cybersecurity Engineers (Job# REF14304A) to secure employee access to our core applications in the most efficient, compliant and user-friendly manner, innovatively using the latest industry standards and tools. Staff Software Engineers (Job# REF14302I) to work as a member of the Marketing Systems Team and responsible for creating the best user experience. Senior Software Engineers (Job# REF14484E) to design, code, document and implement new applications as well as existing programs. Contrib- ute to enhancements, and corrections to existing B2B applications & Visa Business Portal codebase. Information/Cybersecurity Analysts (Job# REF1234IT) to establish strong build and release management practices as part of a devOps team. Perform stand- alone Ant/Maven Builds. Senior Software Engineers (Job# REF14502C) to design, code, document and implement new applications as well as existing programs. Contribute to enhancements, and corrections to existing B2B applications codebase. Certifica- tion in Java Programming required. Business Analysts (Job# REF14503Y) to create internal benchmark indices and lead efficiency strategy based O&I programs. Break down O&I budget into services and identify Total Cost of Ownership of these services.

Openings in our Ashburn, VA location: Systems Administrators, Storage Operations (Job# REF14479K) to support, certify, and document, as well as serve as a critical issue point for, existing infrastructure. Interact with Systems, Database, Engineering, Asset Management, Product Development, Operations, Audit and Risk, and various groups within Technology.

Openings in our Palo Alto, CA location: Senior Software Engineers (Job# REF13476V) to design and develop data administration web interfaces for data management & Access Administration. Senior Software Engineers (Job# REF13487F) to be responsible for the maintenance of the Data Services platform, design and development of various platform components to enable users to rollout new products and services to Visa clients and for internal use. Staff Software Engineers (Job# REF14482U) to serve as domain lead and develop new capabilities on Visa's Global Merchant Repository platform (GMR). Develop complex algorithms using a combination of Natural Language Processing and Machine Learning techniques to improve the quality and accuracy of Merchant Data.

Openings in our Foster City, CA location: Sr. SW Engineers (Job# REF14483N) to work on the design and coding of complex mission-critical systems. Collaborate with other engineers to develop flexible cost effective solutions to tactical and strategic business requirements. Sr. Info. Security Analysts (Job# REF14480J) to be responsible for developing and evaluating all staff security training as well as role-based training for specific organizational roles, such as executive-level or technology employees. Some travel (up to 10%) may be required to work on projects at various, unanticipated sites throughout the United States, India, Singapore, and the United Kingdom.

To apply, please reference Job#s above when mailing resume to: LJ, Visa, Inc., MS: M1-12 SW, 900 Metro Center Blvd., Foster City, CA 94404. EOE

www.computer.org/computingedge 71 IEEE Letters of the Computer Society (LOCS) is a rigorously peer- reviewed forum for rapid publication of brief articles describing high- impact results in all areas of interest to the IEEE Computer Society.

Topics include, but are not limited to: EDITOR IN CHIEF • software engineering and design; Darrell Long - University of California, Santa Cruz • information technology; • software for IoT, embedded, and cyberphysical ASSOCIATE EDITORS systems; Dan Feng, Huazhong University of Science and • cybersecurity and secure computing; Technology • autonomous systems; • machine intelligence; Gary Grider - Los Alamos National Laboratory • parallel and distributed software and Kanchi Gopinath - Indian Institute of Science (IISc), algorithms; Bangalore • programming environments and languages; Katia Obraczka - University of California, Santa • computer graphics and visualization; Cruz • services computing; Thomas Johannes Emil Schwarz - Marquette • databases and data-intensive computing; University • cloud computing and enterprise systems; Marc Shapiro - Sorbonne-Université–LIP6 & Inria • hardware and software test technology. Kwang Mong Sim - Shenzhen University

LOCS offers open access options for authors. Learn more about IEEE open access publishing: Learn more about LOCS,

www.ieee.org/open-access submit your paper, or become a subscriber today: www.computer.org/locs

72 ComputingEdge January 2019

r9p10.indd 10 9/10/18 1:52 PM Looking for the BEST Tech Job for You? Come to the Computer Society Jobs Board to meet the best employers in the industry—Apple, Google, Intel, NSA, Cisco, US Army Research, Oracle, Juniper...

Take advantage of the special resources for job seekers— job alerts, career advice, webinars, templates, and resumes viewed by top employers.

www.computer.org/jobs

r1cov4.indd 4 12/29/17 1:10 PM IEEE TRANSACTIONS ON SUBMIT TODAY BIG DATA

SCOPE

The IEEE Transactions on Big Data (TBD) publishes peer reviewed articles with big data as the main focus. The articles provide cross disciplinary innovative research ideas and applications results for big data including novel theory, algorithms and applications. Research areas for big data include, but are not restricted to, big data analytics, big data visualization, big data curation and management, big data semantics, big data infrastructure, big data standards, big data performance analyses, intelligence from big data, scientific discovery from big data security, privacy, and legal issues specific to big data. Applications of big data in the fields of endeavor where massive data is generated are of particular interest.

SUBSCRIBE AND SUBMIT For more information on paper submission, featured articles, calls for papers, and subscription links visit: www.computer.org/tbd

Blockchain &gt; Internet of Things &gt; Internet

Blockchain > Internet of Things > Internet