Of Artificial Intelligence 8,000 Users

Software Machine Learning Multimedia Quantum Computing

SEPTEMBER 2020 www.computer.org IEEE Computer Society Has You Covered!

WORLDCLASS CONFERENCES Stay ahead of the curve by attending one of our 200+ globally recognized conferences.

DIGITAL LIBRARY Easily access over 780k articles covering world-class peer-reviewed content in the IEEE Computer Society Digital Library.

CALLS FOR PAPERS Discover opportunities to write and present your ground-breaking accomplishments.

EDUCATION Strengthen your resume with the IEEE Computer Society Course Catalog and its range of offerings.

ADVANCE YOUR CAREER Search the new positions posted in the IEEE Computer Society Jobs Board.

NETWORK Make connections that count by participating in local Region, Section, and Chapter activities.

Explore all of the member beneﬁ ts at www.computer.org today! IEEE COMPUTER SOCIETY computer.org

STAFF

Editor Publications Portfolio Managers Cathy Martin Carrie Clark, Kimberly Sperka

Publications Operations Project Specialist Publisher Christine Anthony Robin Baldwin

Production & Design Artist Senior Advertising Coordinator Carmen Flores-Garvey Debbie Sims

Circulation: ComputingEdge (ISSN 2469-7087) is published monthly by the IEEE Computer Society. IEEE Headquarters, Three Park Avenue, 17th Floor, New York, NY 10016-5997; IEEE Computer Society Publications Office, 10662 Los Vaqueros Circle, Los Alamitos, CA 90720; voice +1 714 821 8380; fax +1 714 821 4010; IEEE Computer Society Headquarters, 2001 L Street NW, Suite 700, Washington, DC 20036. Postmaster: Send address changes to ComputingEdge-IEEE Membership Processing Dept., 445 Hoes Lane, Piscataway, NJ 08855. Periodicals Postage Paid at New York, New York, and at additional mailing offices. Printed in USA. Editorial: Unless otherwise stated, bylined articles, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in ComputingEdge does not necessarily constitute endorsement by the IEEE or the Computer Society. All submissions are subject to editing for style, clarity, and space. Reuse Rights and Reprint Permissions: Educational or personal use of this material is permitted without fee, provided such use: 1) is not made for profit; 2) includes this notice and a full citation to the original work on the first page of the copy; and 3) does not imply IEEE endorsement of any third- party products or services. Authors and their companies are permitted to post the accepted version of IEEE-copyrighted material on their own Web servers without permission, provided that the IEEE copyright notice and a full citation to the original work appear on the first screen of the posted copy. An accepted manuscript is a version which has been revised by the author to incorporate review suggestions, but not the published version with copy- editing, proofreading, and formatting added by IEEE. For more information, please go to: http://www.ieee.org/publications_standards/publications /rights/paperversionpolicy.html. Permission to reprint/republish this material for commercial, advertising, or promotional purposes or for creating new collective works for resale or redistribution must be obtained from IEEE by writing to the IEEE Intellectual Property Rights Office, 445 Hoes Lane, Piscataway, NJ 08854-4141 or [email protected]. Copyright © 2020 IEEE. All rights reserved. Abstracting and Library Use: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy for private use of patrons, provided the per-copy fee indicated in the code at the bottom of the first page is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923. Unsubscribe: If you no longer wish to receive this ComputingEdge mailing, please email IEEE Computer Society Customer Service at help@ computer.org and type “unsubscribe ComputingEdge” in your subject line. IEEE prohibits discrimination, harassment, and bullying. For more information, visit www.ieee.org/web/aboutus/whatis/policies/p9-26.html.

IEEE Computer Society Magazine Editors in Chief

Computer IEEE Intelligent Systems IEEE Pervasive Computing Jeff Voas, NIST V.S. Subrahmanian, Marc Langheinrich, Università Dartmouth College della Svizzera italiana Computing in Science & Engineering IEEE Internet Computing IEEE Security & Privacy Lorena A. Barba (Interim), George Pallis, University David Nicol, University George Washington University of Cyprus of Illinois at Urbana-Champaign IEEE Annals of the History IEEE Micro of Computing Lizy Kurian John, University IEEE Software Gerardo Con Diaz, University of Texas at Austin Ipek Ozkaya, Software of California, Davis Engineering Institute IEEE MultiMedia IEEE Computer Graphics Shu-Ching Chen, Florida IT Professional and Applications International University Irena Bojanova, NIST Torsten Möller, Universität Wien

18 24 30 Code Mining When Learning to Multimedia Synthesize and Meets Fashion Manipulate Natural Images Software 8 The Rode0day to Less-Buggy Programs ANDREW FASANO, TIM LEEK, BRENDAN DOLAN-GAVITT, AND JOSH BUNDT 14 BetterCode Reviews With Design by Contract GEORGE FAIRBANKS

Machine Learning 18 Code Mining GERARD J. HOLZMANN 24 When Multimedia Meets Fashion SIJIE SONG AND TAO MEI

Multimedia 30 Learning to Synthesize and Manipulate Natural Images JUN-YAN ZHU 40 Multimedia for Disaster Information Management SHU-CHING CHEN

Quantum Computing 42 The Quantum Moonshot ERIK P. DEBENEDICTIS 48 The National Quantum Initiative Will Also Benefit Classical Computers ERIK P. DEBENEDICTIS AND MICHAEL P. FRANK

Departments 4 Magazine Roundup 7 Editor’s Note: Improving Code Reviews 64 Conference Calendar

Subscribe to ComputingEdge for free at www.computer.org/computingedge. Magazine Roundup

he IEEE Computer Society’s lineup of 12 peer-reviewed technical magazines covers cutting-edge topics rang- Ting from software design and computer graphics to Internet computing and security, from scientific applications and machine intelligence to visualization and microchip design. Here are highlights from recent issues.

stereoscopic head-mounted dis- final position at Xerox, she licensed plays to place users into simu- high-quality typeface designs from Data-Centric Edge Computing lated and realistic 3D digital envi- Mergenthaler Linotype for use on to Defend Power Grids ronments, unlocking significant Xerox’s high-performance 9700 Against IoT-Based Attacks benefits from the seamless inte- laser printer. In the early 1980s, gration of digital information with the Adobe cofounder John War- The Internet of Things (IoT) intro- the healthcare practitioner and nock recruited Crews to Adobe duces new attack surfaces to patient’s experience. This arti- Systems. While at Adobe, she was power grids through Wi-Fi-enabled cle from the May/June 2020 issue responsible for bringing high-qual- high-wattage appliances, render- of Computing in Science & Engi- ity typography to the Adobe Post- ing security mechanisms ineffec- neering explores some of the cur- Script page description language, tive. The authors of this article rent and emerging technologies running corporate marketing, hir- from the May 2020 issue of Com- and applications in surgery; their ing graphics designers, and put- puter propose a data-centric edge- benefits and challenges around ting Adobe on a firm footing with computing infrastructure to host immersion, spatial awareness, and professional designers and print- defend mechanisms in IoT clouds cognition; and their reported and ers, greatly enhancing the value by integrating physical states in projected use in learning environ- of their product. After Adobe, she decentralized power-grid regions. ments, procedure planning, and helped found Electronics for Imag- perioperative contexts in the sur- ing and, after some consulting, gical theatre. retired in 1993.

Augmented and Virtual Reality in Surgery The Advent of Digital High-Fidelity Point-Based Augmented and virtual reality Typography Rendering of Large-Scale 3-D are transforming the practice of Scan Datasets healthcare by providing powerful This article from the January– and intuitive methods of explor- March 2020 issue of IEEE Annals Digitalization of 3D objects and ing and interacting with digital of the History of Computing pres- scenes using modern depth sen- medical data, as well as integrat- ents an interview conducted with sors and high-resolution RGB ing data into the physical world to Liz Bond Crews. Crews began her cameras enables the preserva- create natural and interactive vir- career as a systems analyst for tion of human cultural artifacts at tual experiences. These immer- RCA. She soon joined the Xerox an unprecedented level of detail. sive technologies use lightweight Electro-Optical Division. In her Interactive visualization of these

4 September 2020 Published by the IEEE Computer Society 2469-7087/20 © 2020 IEEE large datasets, however, is chal- 2020 issue of IEEE Intelligent Sys- sensor-based continuous authen- lenging without degradation in tems examine and compare dif- tication system leveraging the visual fidelity. A common solution ferent effects of analyst attitude accelerometer, gyroscope, and is to fit the dataset into available and crowd sentiment on stock magnetometer on smartphones to video memory by downsampling prices with data from CSMAR. capture users’ behavioral patterns. and compression. The achievable By estimating a multivariate lin- To improve the authentication per- reproduction accuracy is thereby ear regression model, they find formance and enhance system limited for interactive scenarios, that although the wisdom of both reliability, they utilize two feature such as immersive exploration in experts and crowds has impact on fusion strategies of serial feature virtual reality. This degradation in stock prices, the latter’s impact fusion and parallel feature fusion visual realism ultimately hinders on stock prices prevails. They also to combine the designed features the effective communication of adopt LightGBM, a novel machine- from the three sensors in the fea- human cultural knowledge. This learning model, to predict stock ture extraction module. Based article from the May/June 2020 trends based on empirical results. on the trained one-class support issue of IEEE Computer Graph- Portfolio returns of different mod- vector domain description classi- ics and Applications presents a els also suggest that crowd wis- fier, they evaluate the authentica- method to render 3D scan data- dom is more valuable for creating tion performance of FusionAuth sets with minimal loss of visual investment strategy than expert in terms of impact of window size fidelity. A point-based rendering wisdom. and user size, and accuracy on dif- approach visualizes scan data as ferent users. a dense splat cloud. For improved surface approximation of thin and sparsely sampled objects, the Using Feature Fusion authors propose oriented 3D ellip- Strategies in Continuous RTX on—The NVIDIA Turing soids as rendering primitives. Authentication on GPU Smartphones NVIDIA’s latest processor family, With the increasing prevalence the Turing GPU, was designed to Stock Selection Model Based of mobile devices, people prefer realize a vision for next-generation on Machine Learning with to use smartphones to make pay- graphics, combining rasterization, Wisdom of Experts and ments, take photos, and collect ray tracing, and deep learning. It Crowds personal vital information. Due includes fundamental advance- to the high possibility of smart- ments in several key areas: stream- Both stock recommendations phone illegal access, the secu- ing multiprocessor efficiency, a from sell-side analysts and online rity and privacy of the devices Tensor Core for accelerated AI user-generated content from become more important. In this inferencing, and an RTCore for crowds have great significance in article from the March/April 2020 accelerated ray tracing. With these the stock market. The authors of issue of IEEE Internet Computing, innovations, Turing unlocks both this article from the March/April the authors present FusionAuth, a real-time ray-tracing performance www.computer.org/computingedge 5 MAGAZINE ROUNDUP

and deep-learning inference in responsible. This paper from the from the May/June 2020 issue of consumer, professional, and data- January–March 2020 issue of IEEE IEEE Software, consisting of 106 center solutions. Read more in this Pervasive Computing explores interviews at a multinational com- article from the March/April 2020 some of the reasons that the ethi- pany operating in a DevOps-at- issue of IEEE Micro. cal implications of IoT for children scale environment, identifies a are difficult to contain, and sug- number of innovative practices. gests several ways that design might make progress toward Glasses-Free 3-D and socially responsible IoT. Augmented Reality Display Food for Thought: Fighting Advances: From Theory to Fake News and Online Implementation Disinformation StopGuessing: Using Guessed In this article from the January– Passwords to Thwart Online Fake news is becoming a grow- March 2020 issue of IEEE MultiMe- Password Guessing ing concern for several indus- dia, the authors describe how four trial sectors and society at large. of the most viable glasses-free 3D Practitioners who are seeking In this article from the March/ display methods were chosen for to defend against online pass- April 2020 issue of IT Professional, their research at NTU’s Advanced word-guessing attacks usually the authors explain why peo- Displays Laboratory. The authors find a shortage of techniques. The ple are susceptible to fake news developed three spatio-temporally authors of this article from the and what types of impact fake multiplexed super multi-view dis- May/June 2020 issue of IEEE Secu- news can cause, taking exam- play systems based on liquid crys- rity & Privacy present an open ples from agri-food and other sec- tal displays, organic light-emitting source system to defend against tors. By providing a concise yet diodes, and fast projectors. This password-guessing attacks using insightful overview in the field, the article explains the background, information ignored previously: authors discuss how advances status of the prototypes, and their the guessed passwords them- in machine learning and seman- performance. selves. They examine passwords tic technologies can be utilized to detect login failures caused by to detect fake news and mitigate users mistyping their passwords. online disinformation.

Accounting for Dynamic Diversity Among Child Users Innovative Practices for of IoT Knowledge Sharing in Large- Join the IEEE Scale DevOps Computer As IoT becomes increasingly per- Society vasive, children are more regu- Agile development methods and computer.org/join larly encountering IoT. The recent DevOps require adaptation during GDPR legislation in Europe goes implementation to meet the needs some of the way toward protect- of a constantly changing software ing children as they make use of development environment. The these IoT technologies, but there emergence of knowledge-sharing remain significant challenges in practices for large-scale DevOps ensuring that the IoT that per- has not been the subject of much vades children’s lives is socially research. This in-depth case study

6 ComputingEdge September 2020 Editor’s Note

Improving Code Reviews

anual peer code review “Better Code Reviews With Design realism while editing photo- Mis the software indus- by Contract” details the design- graphs. “Multimedia for Disaster try’s standard method for catch- by-contract method and argues Information Management,” from ing bugs and design flaws. But that it not only makes the code IEEE MultiMedia, posits that too often, mistakes slip through. review process smoother but also machine learning-based anal- Recently, software practitioners helps developers improve their ysis of multimedia big data can have turned to new techniques for design skills. lead to more effective disaster reviewing code. This issue of Com- Machine learning can also be response. putingEdge presents two ways to employed to improve software. This ComputingEdge issue improve code reviews—one that IEEE Software’s “Code Mining” closes with two Computer articles incorporates automated tech- considers whether machine learn- about the relationship between niques and another that inserts ing could be used to evaluate the classical and quantum comput- more structure and objectivity quality of program code. Mean- ing and the evolving definition of into the process. while, IEEE MultiMedia’s “When “computer.” “The Quantum Moon- IEEE Security & Privacy’s “The Multimedia Meets Fashion” shows shot” explores the shifts in edu- Rode0day to Less-Buggy Pro- how machine learning is contrib- cation, training, and research grams” discusses the advantages uting to better software in the that the computer science field of automated bug-finding tools. fashion industry. must undergo to leverage quan- The authors describe a monthly Machine learning has many tum computing. “The National competition in which teams use uses in multimedia applications. Quantum Initiative Will Also Ben- these bug finders to catch errors The authors of “Learning to Syn- efit Classical Computers” argues in a corpus of programs, with thesize and Manipulate Natu- that rethinking computer design the goals of evaluating the tools’ ral Images,” from IEEE Computer to accommodate quantum will effectiveness and identifying the Graphics and Applications, use lead to advances that affect all of best approaches. IEEE Software’s machine learning to preserve computing.

2469-7087/20 © 2020 IEEE Published by the IEEE Computer Society September 2020 7 EDITORS: D. Balzarotti, [email protected] W. Enck, [email protected] T. Holz, [email protected] This article originally appeared in A. Stavrou, [email protected]

vol. 17, no. 6, 2019 DEPARTMENT: SYSTEMS ATTACKS AND DEFENSES

The Rode0day to Less-Buggy Programs

Andrew Fasano, MIT Lincoln Laboratory and Northeastern University Tim Leek, MIT Lincoln Laboratory Brendan Dolan-Gavitt, New York University Josh Bundt, Army Cyber Institute and Northeastern University

espite their best efforts, computer program- (http://lcamtuf.coredump.cx/afl/) often succeeds at mers consistently fail to build the safe and finding bugs. D reliable programs they imagine; rather, they But as scientists and practitioners, we would like typically build systems that work well until something to know not just that a tool like AFL works, but how unexpected happens. When presented with unex- well it works. We want to gauge bug-finding merit and pected inputs, systems may reveal bugs in their code, improvement to guide the use and development of which cause incorrect behavior or crashes. Every pro- these tools. Is AFL better than KLEE, which uses sym- grammer knows that bugs are a persistent and costly bolic execution and Satisfiability Modulo Theory solv- issue in computer programs, but the path to getting rid ing to find bugs? Is this new version of AFL better than of them is less clear. the last one? To answer such questions, we propose Straightforward approaches such as manual code the following approach to testing bug-finding systems: review are difficult to scale and often unsuccessful. Increasingly, developers are turning to automated 1. Create a corpus of buggy programs. bug-finding tools. ›› A corpus should contain multiple real-world Static analyzers may automatically flag suspicious- programs. looking code patterns or use more sophisticated ›› Each of the programs should contain diverse analyses to discover potential vulnerabilities in large and realistic bugs. codebases. Another approach is to find bugs through ›› Each bug should manifest at a known dynamic testing (i.e., passing a large number of location. diverse inputs to a program and fixing any bugs that ›› An input to trigger each bug should be are discovered). This technique is overwhelmingly generated. favored for both offensive and defensive security 2. Conduct an evaluation of bug finders. research, because this method is easy to perform ›› At the start of an evaluation, all bug finders with off-the-shelf tools, and it gives concrete results should be provided with the buggy programs without false positives. Anyone can download an from the same bug corpus. Other data in industrial-strength, coverage-guided fuzzer such as the corpus (e.g., triggering inputs and so on) “American fuzzy lop” (AFL), and it will likely be able to should not be provided. Prior to the evalu- test his or her programs. More importantly, this type ation, this corpus must not be available in of testing absolutely works; despite its easy setup, AFL any form to developers of systems being evaluated. ›› Each bug finder should be evaluated based Digital Object Identifier 10.1109/MSEC.2019.2933682 upon its ability to find inputs that trigger Date of current version: 29 October 2019 distinct bugs in the corpus.

3. After each evaluation, the full corpus of buggy Apocalypse employs symbolic execution and con- programs should be released to the public with straint solving as well as program synthesis to achieve a a list of bugs, where they manifest, and the similar end. The bugs inserted by Apocalypse are fewer triggering inputs for each. in number but far more complicated and stateful than 4. Evaluations should be run repeatedly and are the LAVA ones. Although these bugs are harder frequently. to generate and fewer in quantity, they may better reflect the complexity of some bugs in real software. Over the past year, we have been running Rode- 0day [a portmanteau of a rodeo and a 0-day vulnerabil- RODE0DAY ity, pronounced “ro-dee-oh-day” (http://rode0day.mit With access to LAVA and Apocalypse, we decided .edu)], a monthly bug-finding competition and our first to launch a recurring bug-finding competition for attempt at this kind of precise bug-finder evaluation. which we would release a corpus of buggy programs We think this competition satisfies all of our criteria, and challenge teams to find as many of the injected with the possible exception of realistic bugs. bugs as they could. Each competition would run for a month, during which time teams would try their BUG INJECTION best to find bugs. At the end of each competition, we To evaluate bug-finding systems, we first need buggy would declare a winner and release inputs to trigger programs. In fact, we need tons of buggy programs for each of the injected bugs. Our vision of how this com- every evaluation. Manually creating these programs petition improves bug finding is shown in Figure 1. clearly can’t scale to the demand, but fortunately, At the start of each Rode0day, teams are given recent research has shown that buggy programs a collection of Linux programs, sample inputs, and can be created automatically. Previously, we created instructions about how to run each program on the two systems to do just that: LAVA (which stands for sample inputs. Teams use their bug-finding skills to “large-scale automated vulnerability addition”) and generate inputs that cause target programs to crash. Apocalypse,3 which both automatically inject bugs When a team believes they have generated such an into C programs. input, they submit it to the Rode0day API and are LAVA uses a dynamic taint analysis to identify scored immediately. For each new bug a team finds, program variables controllable by input data. New they are awarded 10 points. If a team is the first to code is injected into the program, which, depending find a given bug, they are awarded a single bonus on what those variables are set to, can cause a crash. point. This encourages teams to submit their gener- The taint analysis reveals how values in the input file ated inputs as soon as they find them. As teams sub- affect program variables; this information is used to mit new inputs to the API, their scores are updated generate input files that likely trigger each bug. LAVA in real time on a scoreboard on the Rode0day web- combines this analysis with an empirical test to see site. Although we conceal from competitors the if an input actually causes a program to crash. LAVA total number of bugs that were injected into a given was presented at the IEEE Symposium on Security challenge, the scoreboard does show the number of and Privacy in 2016, and the corpora from this article, unique bugs found by all the teams, thus providing a LAVA-1 and LAVA-M, were released later the same lower bound. year. At that time, the combined detection rate for a In May 2018, we ran a closed beta Rode0day state-of-the-art fuzzer and symbolic execution bug competition. Using LAVA, we injected 52 bugs into finder was 2%. Since then, 47% of the bug-finding two simple C programs we had written: a trivial file systems presented at the top four academic computer parser and a simple key/value store. On average, our security conferences have self-evaluated against nine competitors found slightly more than half of the LAVA-M corpus. However, the expiration date of the injected bugs. The first-place team, itszn, found LAVA-M seems to have come and gone, as evidenced every injected bug in roughly 5 h, an indication that by the Angora fuzzer1 finding all of the labeled bugs in future competitions should feature more substantial this corpus (and a few unlabeled ones, too). programs. www.computer.org/computingedge 9 SYSTEMS ATTACKS AND DEFENSES

Bug Finding Repeat

bug Fix Found Analyze Select Program Program Bugs Program bug Program to bug for Analysis bug Find Bugs bug

Select or Modify a Bug Finder

Rode0day Repeat

bug bug bug bug bug Bug Injection Rode0day Select Programs bug bug Learn Programs With LAVA/ bug Challengesbug Run Competition for Competition bug bug bug What Works Apocalypse bug bug bug bug

Evaluate Bug Finders

Evaluate bug bug bug bug Approach 1: 13% Found Beer Rode0day Diﬀerences bug Understanding of Challengebug Approach 2: 10% Found Between Bug-Finding bug Bug-Finding bug bug Approaches Approach 3: 27% Found Approaches

Study Bugs

bug bug Rode0day Bug Found by Teams 1 and 3 Evaluate bug Diﬀerences Beer bugChallenge Bug Found by All Teams Understanding bug bug Between bug bug of Bugs bug Bug Found by No Teams Bugs

FIGURE 1. An example of how Rode0day improves bug finding.

We launched our first official Rode0day competi- experiments by introducing different types of bugs, tion in July 2018 and have been running new compe- varying the size of programs, and using different titions (nearly) every month since. At the time of this bug-injection strategies. Although we haven’t yet con- writing, we have run 10 competitions with 44 buggy ducted a full analysis of the data we’ve collected in the programs containing a total of 2,103 bugs. These pro- past year, in this article, we provide a sampling of some grams are generated from the real software displayed of our more interesting results. in Table 1. A total of 28 teams have scored points in One obvious question is: Which team is the best? these competitions. Answering this question is not completely straightforward, because not every team has participated in RESULTS every competition. To assess the relative performance Administering Rode0day provides us with a wealth of of different teams, we used a version of the Elo rating empirical data about what works and what doesn’t for system, which was developed to rank chess players. finding bugs. It also allows us to conduct controlled Specifically, we use the multiplayer variant of Elo

10 ComputingEdge September 2020 SYSTEMS ATTACKS AND DEFENSES

TABLE 1. The target programs used in Rode0day competitions. TABLE 2. The overall rankings for top Rode0day competitors after 10 competitions. Bugs Bugs Name DescriptionC-SLOC* injected discovered Place Elo scoreTeam name File File parser 15,671 657431 11,087aﬂ-lazy LibYAMLYAML parser 10,324 231216 21,069 itszn LibjpegJPEG parser 20,316 268268 31,027 H3ku Bzip2File compressor 5,8202828 41,017 REDQUEEN DuktapeJavaScript engine 83,793 18 15 51,062 NU-AFL-QSYM Grep Regular expression matcher 70,030 80 78

Jq JSON parser 16,897 332210

Pcre2grepRegular expression matcher 83,113 184180 discovered bug. The color of each square indicates the time it took to Sqlite Database 12,902 56 24 find a bug: green squares indicate TinyexprMath engine 1,2414437 bugs found within 24 h, light blue

*Lines of C code as measured by David A. Wheeler’s SLOCCount. SLOC: source lines of code. squares indicate bugs that took just over a day, and darker blue squares took longer (up to 25 days). described by Tom Kerrigan at http://www.tckerrigan Of these discovered bugs, some are clearly easier to .com/Misc/Multiplayer_Elo. In the Elo system, every find than others, as evidenced by multiple teams find- player starts out with the same skill rating. When two ing the same bugs within a day. players play a match against each other, the differ- Of the 34 discovered bugs shown, the rightmost ence in their scores is used to calculate the expected column represents a bug found by a single team after outcome. The difference between the expected and 25 days of searching. To understand some of the chal- actual outcome is then used to update their ratings. lenges involved in finding this bug, let’s walk through The rankings for our first year are shown in Table 2. what needs to occur for it to be triggered. The bug The top-ranked team, afl-lazy, played in the last five requires three separate program variables (i.e., A, B, competitions and won four of them. Despite being the and C) to be set to values copied from separate por- best team to participate thus far, afl-lazy has found tions of the input file. If the values of A, B, and C satisfy fewer than half the injected bugs across all the Rode- the equation ((A * B) – C) == 0xf14ec3, an index into 0day competitions they participated in. Finding these an array is offset by the value from B, so it is possible injected bugs is clearly not a solved problem. (though difficult) to generate an input that will lead In addition to comparing team performances, to an out-of-bounds array access. The input file also we’re also interested in studying what makes some determines the path taken by the program, and this bugs easy to find and others difficult. Figure 2 shows path must visit the locations in the code where A, B, which bugs were found in a version of a file used in the and C are assigned values from the input. However, the November 2018 Rode0day. For this challenge, the 59 sample input provided to competitors did not execute LAVA bugs injected were of two types: simple bugs, in this code for A and C, nor did it reach the potentially which a memory safety violation occurs if a value from invalid array access. Competitors had to find an input the input matches a constant; and complex bugs, in that could reach these parts of the program. which a memory safety violation occurs if three values But these challenges are not unique to this bug; from the input satisfy a particular equation. Fifty per- other bugs required solving similar equations and cent of the 36 injected simple bugs and 70% of the 23 executing new parts of the program. So why was this injected complex bugs were found by the six compet- bug so much harder to find than others? And why could ing teams. Each column in Figure 2 corresponds to a NU_AFL_QSYM find it and other teams could not? As www.computer.org/computingedge 11 SYSTEMS ATTACKS AND DEFENSES

Relative Hours Until Solved Bugs Found in ﬁle2_S2 0 623

Bug Type Team Name Simple Bug Complex Bug H3ku NU-AFL-QSYM NU_AFL REDQUEEN aﬂ-lazy slazar0_A

FIGURE 2. A visualization of when teams found bugs in file2_S2 during the November 2018 Rode0day. Green indicates the bugs found within 24 h of a team’s first score. The stars denote the first team to find a bug.

we continue to run our Rode0day competitions, we hope positives and because different tools may reasonably to collect enough data to answer questions like these. disagree about the location of a bug in the program. By finding a way to quantify how difficult a bug is For instance, if a pointer is corrupted on one line of a to find with a given approach to bug finding, we hope program but does not cause a crash until it is derefer- to gain an understanding of how different bug-finding enced on another line, which line contains the bug? We strategies can be used in combination or how new think these challenges can be overcome with careful bug-finding systems can be developed to find and fix scoring design and some empirical examination of how more bugs in real programs. current static analyzers report bugs. Finally, for the time being, we have intentionally he first year of Rode0day has been a success, but made no attempt to place computational restrictions Twe see several areas where it can be improved. on competitors. This means that a well-funded team Most pressingly, we have yet to ensure that the bugs could potentially perform better by dedicating thou- used in the competition are representative of the sands of cores to bug finding. It also makes it more dif- naturally occurring bugs we’d like our bug finders ficult to compare the performance of different tools. to uncover. We plan to do this by 1) looking at many To make more direct, apples-to-apples comparisons real-world bugs and trying to get LAVA to match between different tools, we plan to create a yearly, their properties in ways that matter and 2) validating computation-limited competition. For this event, that the systems that perform well in our competi- competitors would submit a Docker container or vir- tion are also successful in the real world. We would tual machine containing their bug-finding system; we also like to open up Rode0day to other bug-injection would then allocate some fixed amount of resources systems aside from LAVA and Apocalypse, such as on a standard cloud platform like Amazon EC2 to run hand-created bug corpora. This would allow others to the bug-finding systems and declare a winner. contribute bugs that they think are more realistic or As we continue running Rode0day, we hope to that cover new bug classes (e.g., use after free) that we keep improving our competitions to provide the best haven’t had the time to implement. possible evaluation of bug-finding systems. Along the Another limitation is that most static analysis way, we’ll keep collecting, publishing, and analyzing tools cannot currently participate because they do our data in the hope of giving our community a better not provide a triggering input that demonstrates they understanding of bugs and bug finding. have found a given bug. For each of the injected bugs, however, we know exactly where their “root cause” is, ACKNOWLEDGMENTS and because our bugs are injected into source code, This material is based upon work supported by the we can, in principle, evaluate static analyzers as well. Under Secretary of Defense for Research and Engineer- This task is slightly more complicated than evaluating under U.S. Air Force contract FA8702-15-D-0001. ing fuzzers, both because of the possibility of false Any opinions, findings, conclusions or recommendations

12 ComputingEdge September 2020 SYSTEMS ATTACKS AND DEFENSES

expressed in this material are those of the author(s) Ph.D. student at Northeastern University, Boston, Massa- and do not necessarily reflect the views of the Under chusetts. Fasano received a bachelor’s degree from Rens- Secretary of Defense for Research and Engineering. selaer Polytechnic Institute, Troy, New York. Contact him at [email protected]. REFERENCES 1. P. Chen and H. Chen “Angora: Efficient fuzzing by TIM LEEK is a member of the research staff at the MIT Lin- principled search,” in Proc. IEEE Symp. Security and coln Laboratory, Lexington, Massachusetts. Leek received a Privacy, 2018, pp. 711–725. master’s degree from the University of California San Diego. 2. B. Dolan-Gavitt et al., “LAVA: Large-scale automated He is a member of USENIX and the Association for Comput- vulnerability addition,” in Proc. IEEE Symp. Security and ing Machinery. Contact him at [email protected]. Privacy, 2016, pp. 110–121. 3. S. Roy, A. Pandey, B. Dolan-Gavitt, and Y. Hu, “Bug BRENDAN DOLAN-GAVITT is an assistant professor at New synthesis: Challenging bug-finding tools with deep York University. Dolan-Gavitt received a Ph.D. from the Geor- faults,” in Proc. ACM Joint Meeting European Software gia Institute of Technology, Atlanta. He is a Member of the Engineering Conf. Symp. Foundations of Software IEEE. Contact him at [email protected]. Engineering, 2018, pp. 224–234. JOSH BUNDT is a Ph.D. student at Northeastern University, Boston, Massachusetts. Bundt received a master’s degree ANDREW FASANO is a member of the research staff at the from the Naval Postgraduate School, Monterey, California. MIT Lincoln Laboratory, Lexington, Massachusetts, and a He is a member of USENIX. Contact him at [email protected].

ADVERTISER INFORMATION

Advertising Coordinator Central US, Northwest US, Southeast US, Asia/Pacific: Eric Kincaid Debbie Sims Email: [email protected] Email: [email protected] Phone: +1 214-553-8513 | Fax: +1 888-886-8599 Phone: +1 714-816-2138 | Fax: +1 714-821-4010 Cell: +1 214-673-3742

Midwest US: Advertising Sales Contacts Dave Jones Email: [email protected] Mid-Atlantic US: Phone: +1 708-442-5633 Fax: +1 888-886-8599 Dawn Scoda Cell: +1 708-624-9901 Email: [email protected] Phone: +1 732-772-0160 Cell: +1 732-685-6068 | Fax: +1 732-772-0164 Jobs Board (West Coast and Asia), Classified Line Ads

Southwest US, California: Heather Bounadies Mike Hughes Email: [email protected] Email: [email protected] Phone: +1 623-233-6575 Cell: +1 805-208-5882

Northeast, Europe, the Middle East and Africa: Jobs Board (East Coast and Europe), SE Radio Podcast David Schissler Email: [email protected] Marie Thompson Phone: +1 508-394-4026 Email: [email protected] Phone: +1 714-813-5094

www.computer.org/computingedge 13 EDITOR: George Fairbanks, [email protected] This article originally appeared in DEPARTMENT: THE PRAGMATIC DESIGNER

vol. 36, no. 6, 2019 Better Code Reviews With Design by Contract

George Fairbanks

esign by contract (DBC) is a technique that not whether it works but if it’s well designed. Clean improves the quality of your team’s code. It code usually has a contractual nature while spaghetti D yields code with both a logical and a proce- code does not, even if its author is not consciously fol- dural nature, where the contracts state declaratively lowing DBC. Who has not sighed in frustration when what will happen, and the implementations procedur- encountering something like “void process() {/* 1kLOC ally cause the desired effect. The team can reason elided */}”? either logically, by using the contracts, or procedurally, DBC and code reviews are a great combination. by following the code line by line, but the former allows Reviewers may themselves write clean code but strug- them to reason about far larger programs. It also cre- gle to guide others to do the same. They can say how ates conditions for deliberate practice so developers they would write the code but not articulate why that using DBC grow their design skills faster. is better. Such advice can devolve into a battle of opin- Teams that are looking for ways to improve their ions. The opportunity with DBC is that when both the code should seriously consider DBC. It is a technique author and reviewer agree to a goal of writing code with for designing software in which each method has a clear contracts, they can look out for the DBC practices contract, much like a legal document, stating what being followed (or not) in the code being reviewed. A the caller is responsible for and what the method body reviewer can point the author to the relevant DBC prac- must do. It was introduced as a term and adapted tice and, fingers crossed, help the author improve the to object-oriented design in the 1980s by Bertrand design. This article includes a list of DBC practices that Meyer1 and traces its roots to the late 1960s with the you can add to your code review style guide. work of Robert Floyd, Tony Hoare, and Edsger Dijkstra on reasoning logically about procedural programs. HUNGRY FOR CONTRACTS DBC is not a magic elixir that guarantees great pro- What are contracts exactly? Let’s introduce this con- grams. It’s more like standard sentence mechanics in cept using an example that’s familiar to everyone— an essay. When sentences are awkward, it’s hard for a buying a sandwich—and apply some practices to paragraph or essay to succeed. There are authors who decide on a contract. A sandwich seller might tell us have written great essays while breaking grammar and that buying a sandwich can be broken down into a style rules, but they have done so after they’ve mas- series of steps: take payment, give change, collect the tered them. DBC ensures that methods and functions ingredients (bread, peanut butter, and jelly), spread are simple and easy to understand and are therefore the peanut butter on one piece of bread using a knife, great building blocks for a whole program. spread jelly on the other piece using a knife, assemble Many teams do code reviews so that all code the sandwich, and deliver the sandwich. Let’s call those changes are reviewed by a teammate. Before being steps the implementation of the buySandwich method. sent for review, however, the author has already tested DBC asks us to add a contract to that implementa- the code and knows it works. So, code reviews check tion, for example (given in U.S. dollar): If I give you $5, you will give me a sandwich. That contract states the buyer’s responsibility and what the seller will provide in Digital Object Identifier 10.1109/MS.2019.2934192 return. We can make that a bit clearer by saying “buyer” Date of current version: 22 October 2019 and “seller” instead of “me” and “you.” We can also

state what you probably assumed, buySandwich i.e., that the seller takes the money. Precondition: The buyer has $5. • Take payment. Here’s an improved version of that • Give change. contract: If the buyer gives the • Collect the ingredients (bread, peanut buer, and jelly). • Spread the peanut buer on one piece of bread using a knife. seller $5, the seller will give the • Spread jelly on the other piece using a knife. buyer a sandwich and keep the $5. • Assemble the sandwich. If you had read that contract as • Deliver the sandwich. Postcondition: The seller has $5, and the buyer has a sandwich. a code comment above the method buySandwich, you’d be able to understand that method fairly well FIGURE 1. An example of contract and implementation. without reading the code body. Notice that the contract isn’t a translation of the procedure into natural language. The con- could convince yourself that any of these details are tract doesn’t talk about how the sandwich is made (e.g., something a caller may depend on. If so, how will we using a knife) or the sequence of its operations. Instead, it ever fix bugs or change the implementation to run states what happens before and after the method. faster? Stating the contract removes the guesswork. You may have noticed that the contract talks about As an implementer, writing a contract leads you to what happens using procedural language: The seller think about what the caller can rely on and separate gives the buyer $5. We can change that to declara- that from how the method is implemented. As a caller, tive language like this: The buyer has $5. Now that it’s a contract tells you what’s safe to depend on. stated declaratively, we can use it as the precondition for the buySandwich method. The postcondition becomes: The seller has the buyer’s $5, and the buyer CLEAN CODE USUALLY HAS A has a sandwich. The contract and procedure are CONTRACTUAL NATURE WHILE shown in Figure 1. SPAGHETTI CODE DOES NOT, EVEN Perhaps you are thinking that this is just a mild IF ITS AUTHOR IS NOT CONSCIOUSLY improvement, as there could have been a comment FOLLOWING DBC. on the buySandwich method saying something similar. Besides, everyone knows how buying a sandwich works. What is different is that callers know what they LOGICAL REASONING can depend on. Consider a few details of the imple- Without the contract, you can reason through a mentation that do not appear in the contract: method procedurally, animating the source code line by line in your head like a little machine, and draw con- ›› Making change: The method body says that the clusions about how it will behave. When methods have buyer will get change, but the contract doesn’t contracts, you can still use procedural reasoning if you guarantee that. You have probably seen similar want to, but you can also apply logical reasoning. best efforts, say, from a vending machine that In this buySandwich example, you know that, might not have exact change or a public bus. before calling the method, you are rich and hungry, and ›› Assembly with a knife afterward, you are poor and full. That’s consistent with ›› Peanut butter before the jelly reasoning procedurally about the implementation, but ›› Sandwich type: I’m sure there are readers who it’s different. It lets you employ formal logic, which is were already questioning whether peanut butter why contracts are used in automated program analy- and jelly is an acceptable sandwich at any price. sis, such as when your IDE warns you that a value in your program might be null. Without the contract, you could read the method Contracts also let humans reason informally with name and implementation and then guess at the logic, and we do that all the time. Imagine an implemen- contract, but it’s easy to infer the wrong things. You tation of buySandwich that uses two helper methods: www.computer.org/computingedge 15 THE PRAGMATIC DESIGNER

• Contracts state what must be true about the inputs. looking for ways to guide authors • Contracts state what will be true about the outputs. • Contracts use declarative, not procedural, language. in advance, not just during the • Contracts omit implementation choices, including sequence. review. Authors can shoot at a • Contracts state when a subset of a type is used. • Contracts state how nulls are handled. known target as they write code • Contracts state what inputs or states trigger predictable failures. by using a list of DBC practices and • Contracts identify any side eﬀects. the overall guiding metaphor of a • Callers can understand the contract without reading the implementation. • State ubiquitous conditions as invariants. contract that is usable by callers. • Prefer simple contracts over complex ones. I was unable to find a checklist • Omit contracts only when caller cannot infer the wrong contract. • Align contracts with the contours of the problem. suitable for use in code reviews, so • Separate predicates, queries, and commands. I created the list shown in Figure 2, and I think it’s consistent with FIGURE 2. The design-by-contract practices. DBC literature, such as in Mitchell and McKim.2 You can think of the list as an extension of the team’s collectMoney and makeSandwich. Does that work? coding style. Like any style guidance, you should dis- Your logical intuition says yes. But consider differ- cuss and tweak it for your project. ent helper methods blandly named A and B. Now your I find that once developers start thinking about intuition is less sure. You actually don’t have any more DBC, it changes how they write every method. There information about collectMoney than you do about A, are always choices about how to decompose a prob- but your mind inferred a logical contract such as “At lem, and they will gravitate toward methods where the contract is easy to state. When introducing DBC to an existing team or codebase, it’s better to do it gradually ONCE DEVELOPERS START THINKING and skip contracts that you think callers can reliably ABOUT DBC, IT CHANGES HOW THEY guess. Overall, state contracts as terse comments that WRITE EVERY METHOD. help callers. DBC is something that you can practice on your own but it’s even better if the whole team adopts it. Low-hanging fruit is the easiest to pick and the end of the method collectMoney, the seller has the coworkers are unlikely to protest, so I suggest starting $5.” You couldn’t have been reasoning procedurally with methods that represent predicates, such as isAc- through the implementation because there isn’t one. tive or hasAddress. Contracts for these methods can We all reason through programs procedurally, be stated as one-line comments of the form “Returns but there’s a size limit, and it’s not big. Can you keep true iff … ,” where “iff” is short for “if and only if.” If the 10,000 lines in your head and reason about it? Using codebase already has methods like these, the change procedural reasoning, that’s shaky, but using logical is just stating the contract, but if the methods don’t reasoning it’s pretty easy. Consider this: I bet you recall exist, then you also benefit from making the code read the postcondition for buySandwich, but do you recall better by reducing in-line logic. every step in the implementation? Next, turn your attention to query methods, like To me, the ability to scale our reasoning is the great getStatus or getAddress. Because a comment saying benefit of DBC. When you structure your program with “Returns the status” is unhelpful noise, think about clear contracts on methods, you can always fall back the corner cases and write a contract if you discover to procedural reasoning, but you also unlock your abil- anything interesting, i.e., how it handles null, whether ity to reason logically and can keep larger programs in the method checks for invalid data, or whether the val- your head. ues are only a subset of the declared type (especially for primitives such as string or integer). Also consider HOW TO GET STARTED whether the accessors should exist, as they could Every code review starts with the author thinking that be coupling the caller to the current implementation the proposed change is a good idea, so we should be unnecessarily.

16 ComputingEdge September 2020 THE PRAGMATIC DESIGNER

When you write contracts for commands (i.e., BC is a technique taught to computer science transactional methods that are impure), look for Dundergraduates at many universities, including opportunities to split the method. Broad methods the Massachusetts Institute of Technology in 6.031 such as updateCustomer tend to have long contracts and Carnegie Mellon University in 17-241. It encour- that cover all of the corner cases. It can be easier to ages designs where you know what must be true when write the contract for a narrower method that does a method completes, rather than designs where a less, and it’s easier for clients to understand. A set of method does a bunch of stuff and you squint to infer commands likely has an obligation to maintain invari- what exactly it means. ants on the data structures it manipulates, so make If you are a technical lead or manager who wants sure those invariants are clearly stated. to improve your system’s code, you could just wait several years until the team has more experience. If you DELIBERATE PRACTICE want to do something today, however, there are only As they grow, developers must learn to detect vague- a few techniques that are easy to teach and offer the ness, incompleteness, and clumsiness in their designs. benefits that DBC does. DBC helps if just one person Years on the job will eventually give them those skills, on the team applies it, and it helps more with each but mundane experience is less effective than deliber- additional person. ate practice. One nice thing about DBC is that it can There are other ways to arrive at elegant designs, turn routine programming into deliberate practice. but DBC is a particularly good fit for code reviews To understand how, let’s look at deliberate practice in because a reviewer can point the author to a practice another field. that the code does not yet follow. What’s more, DBC William Zinsser, an English composition teacher, leads the team to think about the abstractions that says that in his writing classes, he would not cross out the contracts refer to, so it’s a gateway to other helpful unclear or unnecessary parts of students’ sentences techniques like precise modeling. but would instead put square brackets around those Having contracts on methods is like having an parts. Rather than simply telling the students what he owner’s manual in your car’s glove box. The most loved considered the right answer, his technique encour- owner’s manuals are the ones that are never opened aged them to wrestle with it themselves. His experi- because the design is simple and obvious. Everyone ence was that, by the end of the semester, they had wants software that is simple and probable, but wish- learned to write terse prose.3 ing does not make it so. DBC provides an early warning By encouraging his students to wrestle with their about awkward designs, shows where complexity still work, he created the conditions for deliberate prac- lives, and often leads to methods that callers under- tice. I see the same thing happening with DBC. When stand without reading the contract. I’m just grinding out code, I’m not deliberately practic- ing. But when I force myself to state the contracts for REFERENCES each method, I notice when an idea is fuzzy, when the 1. B. Meyer, Object-Oriented Software Construction. contract is rambling, or when my code makes unstated Reading, MA: Addison-Wesley, 1994. assumptions about a data structure. 2. R. Mitchell and J. McKim, Design by Contract, by The act of stating contracts creates the condi- Example. Reading, MA: Addison-Wesley, 2001. tions for deliberate practice. It makes the unstated 3. W. Zinsser, On Writing Well: The Classic Guide to Writing visible, like the square brackets that direct attention Nonfiction. New York: Harper Perennial, 2016. in an essay. The contracts let me see my code from a different perspective, revealing design flaws, and nudging me toward clearer designs. If DBC is able to accelerate the careers of developers by helping them GEORGE FAIRBANKS is a software engineer at learn to detect vagueness, incompleteness, and Google. Contact him at gf@georgefairbanks clumsiness in their designs, it is worth trying for that .com. reason alone. www.computer.org/computingedge 17 EDITOR: Gerard J. Holzmann, Nimble Research, [email protected] This article originally appeared in DEPARTMENT: RELIABLE CODE

vol. 36, no. 2, 2019 Code Mining

Gerard J. Holzmann

achine-learning techniques are starting to achieve some impressive feats. The suc- M cesses are due, in part, to the availability of astonishingly large amounts of data that can be used for training. Show a suitably equipped system a billion images of cats and a billion images of lawn mow- ers, and then see whether it can figure out what the common patterns are. If this succeeds, we can then show the same system a picture of our own cat or lawn mower, and maybe it can tell which is which. Of course, if you show the same system an image of a house, it wouldn’t have a clue beyond saying that it’s neither a cat nor a lawn mower. Basically, what we’re leveraging are statistics, not intelligence, but, nonetheless, the results can be impressive. FIGURE 1. A picture of a smiling cat that, with some luck, a Could we somehow use the same type of approach machine-learning system could be trained to recognize. My in software analysis to find likely bugs? Just superfi- daughter, Tessa, drew this when she was 10 years old, devi- cially, you can imagine an experiment in which we show ously adding orange splotches that she must have known some “suitably equipped” system a billion examples of could complicate recognition. good programs and ask it to deduce common patterns. Then we show it a billion examples of programs known to be buggy and again ask it to spot some common char- MINING CATS acteristics among these. Next, we show the now-learned How do we know, or, how could our learned system system our own code and ask it to classify the code as know, that we’re looking at a picture of a cat like in Fig- either good or bad. Finding the input sets would not ure 1? Clearly there are clues in the image. The out- be very difficult. There are large enough databases of line of the cat is one such hint. Other clues are the two reasonably “good” programs available. I’ll conveniently eyes, the pointy ears, and other characteristics com- sidestep the landmine here of trying to give examples, mon to cats. Both we and our systems can learn to rec- but we can likely all think of a few that could fit the bill. ognize those patterns. The patterns in the image can, We also have substantial repositories of examples of of course, be deduced with relatively simple image bad code, such as the fragments of code collected processing techniques until we get what is basically in the ever-growing database of common weakness a fingerprint or signature of each image. I’m not a enumerations that you can mine at https://cwe.mitre machine-learning expert, but I can imagine that some .org. I think you’ll agree that this approach is unlikely to clever hashing techniques could be used to group simi- work. I’ll try to explain why I think so, although I would lar images, which can then be used to gain a confidence be delighted if someone could prove me wrong. rating of the closeness of a match. Figure 1 should then score a lot better against our fingerprints of cat images than against those of lawn mower images. Digital Object Identifier 10.1109/MS.2018.2884867 But how would that process work if we wanted to Date of publication: 22 February 2019 distinguish good from bad software? Program code is

rife with patterns that are obvious to a human reader. “for” statement in the C language. It doesn’t take much But are they also easy to discover by a learning algo- imagination to note that if the condition part of the rithm? A few examples can illustrate the degree of for-statement contains a < or <= operator, then the difficulty. increment part of the statement will most likely con- A person may realize, for instance, that we don’t see tain an increment operator, such as ++ or +=. Similarly, many appearances of the word “goto” in well-written if the condition part contains a > or >= operator, then code, but he or she may also notice that, whenever we the increment part will most likely contain a < or <= do see it, it is most often followed by a name, and that operator. Intuitively, this is fairly obvious. We should same name appears somewhere else in the same code be on our guard now, because sometimes things that followed by a colon. That name, finally, will rarely be fol- seem obvious are actually not true. So let’s make sure. lowed by an equal sign in the same fragment of code. Of I don’t have a machine-learning algorithm handy course, the name in question is quite arbitrary, which that can give me statistics for large code bases, but means that, although the pattern is real enough, it might it is easy enough to run some tests, say, on a couple be very hard for an algorithm to discover it. Furthermore, of million lines of source code from one of the Linux we should add that even if our algorithm would dis- distributions. I’ll use the 14.9 million lines of code from cover it and could detect deviations from the pattern the Linux 4.3 distribution for this check. in bad code, that ability is not too useful because every Figure 2 shows the correlation of operators used halfway decent compiler can do the same. in the condition part of for-loops with operators used in the increment part of the same loop. So, here, we’re CODE AS TEXT ignoring everything else that might appear within the It would be a little more useful if our algorithm could control part and just look at the operators. It’s fair deduce that a name, when it is followed by an open to say that there is, indeed, a correlation of the type round brace and preceded by a word other than “void” we suspected, and it’s a pretty strong one at that. (in “good” programs, e.g., in the definition of a func- The most common way to write a for-loop is to use a tion), will almost never be preceded by either a colon < operator in the condition and a ++ operator for the or a semicolon when it appears elsewhere in the code loop increment. Table 1 shows the most interesting (again, followed by an open round brace). That is, the combinations in more detail. value returned by the function is effectively used by The more unexpected combinations, for instance, the caller in an expression or an assignment. the use of a < or <= operator and a decrement opera- As you can see, trying to describe these code pat- tor, are quite rare but curiously nonzero. Those cases, terns as just textual coincidences leads to some tor- being the statistical outliers, could well warrant more tured language, but the statistics should definitely bear scrutiny and may contain either mistakes or the use out these types of correlations. Whether they are also of obscure code that may be prone to misunderstand- discoverable by a machine-learning algorithm without ing. A good example of the former could be this line further guidance is another matter. I have my doubts. of code in file ./drivers/net/ethernet/qlogic/netxen Unfortunately, it is generally easy to record patterns /netxen_nic_hw.c, line 2,335: that are present but much harder to record patterns that are rare or absent. An example of a well-known for (k = 0; k < read_cnt; k--) { pattern that is absent from well-written English prose, for instance, is that a preposition is rarely followed by a and an example of the latter could be this line of code period. Our teachers made sure that we all follow that in file mm/memtest.c, line 108: rule, at least most of the time. for (i = memtest_pattern - 1; i < UINT_MAX; --i) { CODE FORENSICS Although the patterns I mentioned so far are some- The good thing is that, once we know the statistics, what dubious, it is not too hard to think of correlations the number of outliers is usually sufficiently small that that could be worth mining. For example, consider a we can inspect each one in detail and form an opinion www.computer.org/computingedge 19 RELIABLE CODE

FIGURE 2. A graphic showing the correlation between operators used in the condition part of a for-loop and the operators used in the increment part of the same for-loop. The peak of 48,138 cases is for the combination of operator < in the condition part and operator ++ in the increment part of the same for-loop. The statistics are for 14.9 million lines of the Linux 4.3 distribution. Symbols at the base of the graphic represent operators. Numbers along the left side represent number of cases.

TABLE 1. Some of the most frequent operator and then filter out the operators in the control part combinations used in for-loops. that follows in round braces. Often, the complete control part of the for-loop appears on the same line as the initial keyword but not always. There are enough exceptions to the one-line rule here that we may want to try another way. A code-browsing tool comes in handy at this point, especially if it allows us to specify a pattern of interest as a regular expression over lexical tokens rather than plain characters. The tool I’ve used for my check about the quality of that code. A static analyzer could is called Cobra (http://spinroot.com/cobra), and it sup- take advantage of these types of patterns, but the ports such a pattern-matching language. So when we question is still: How can we systematically discover say, for instance, them with machine-learning techniques or otherwise? $ cobra -pat “for ( .* ) ” *.c CODE EXPLORATIONS I haven’t said yet how I generated the data shown in we’re not restricted to matches that appear on a single Figure 2 and Table 1. You can, of course, try to use sim- line, since the match is on the token sequence, which ple text processing tools to extract the code frag- is insensitive to white space. ments of interest. For example, we can try to use the Unfortunately, this still doesn’t give us the statis- tool grep to find all occurrences of the keyword “for” tics we need. We will have to postprocess the code to

20 ComputingEdge September 2020 RELIABLE CODE

extract the two groups of operators. Yet even though round brace can be interrogated as a set. So the third the input to the pattern matcher above isn’t sensitive command checks that set and narrows the selections to white space, the output (the matching lines) will still to only those sets that contain tokens matching the contain all those line breaks. So, in postprocessing, we regular expression shown. The regular expression must still deal with that circumstance. matches only on token texts that start with < and may I found it easier to enact the entire process with a contain a subsequent = character and nothing else. sequence of code-querying commands, issued inter- Now we have all for-loops that contain either a < or actively in a code-querying session over all of the 14.9 a <= operator. Next, we check whether those same million lines of code. I used those commands to home for-loops contain a decrement operator, which is in on the various parts of each for-loop and then write either a - or a -= token, again, with a regular expression. it out in two groups of operators per line, one for the We refine the set of matches a little more by next also operators used in the condition part and the other saying that the same clause does not also contain the for operators used in the increment part. A few small increment operator ++, and we then display the small postprocessing steps then gave me the data for Figure set of remaining matches. 2. But we can do more. The whole sequence can be abbreviated by using For example, the following is a sequence of Cobra single-letter shorthand for each query command, commands I used to find a relatively small set of devia- and we can pass the entire command sequence in a tions from the general rule that we can find in the code command-line invocation of the tool rather than typing base. The pound sign (#) is a comment delimiter of the them in one at a time in an interactive session, but you query-command language: get the idea. Of course, this whole process of querying the code base would be of little use if it took more than mark for ( # mark all keywords “for” followed a few seconds, so speed is important. by a round brace next # move the mark forward to the FAST APPROXIMATIONS token following “for” We can get the tool to work fast because the match- contains/ˆ<=?$ # keep only those marks where the ing algorithm is simple and, therefore, trivial to par- range of tokens that is allelize. It can, in principle, be executed indepen- # enclosed in the round braces dently on each separate file, so given enough CPUs, contains an < or <= operator the 21,987.c files in the Linux distribution we looked at contains/ˆ-[-=]$ # keep only those marks where the could be scanned within a fraction of a second. On my range of tokens also own 32-core system, the processing takes roughly 20 # contains at least one decrement s, which includes approximately 9 s for preparing the operator lexical token stream itself. I should add that this last contains no ++ # and no increment operators code-mining method I used is not precise. For exam- = “nr matches:” # report the number of matches ple, I did not bother to distinguish among the three dif- display # and display them ferent parts of the for-loop. The final set of matches was small enough, even for a code base of this size, The first command in this set marks all places in that I could afford to overshoot the final set by a small the code where we find the keyword “for” followed amount and then peruse the matches for the little by an open round brace. Remember that white space gems I was really seeking. doesn’t matter since we’re matching on lexical tokens This brings to mind a quote from the late John here. The matches will point at the locations in the Tukey, a Bell Labs mathematician who, in the late 1940s code where the “for” itself appears. With the next com- and 1950s, worked with both John von Neumann and mand, we now move those match points forward to Claude Shannon. Tukey wrote in 1962: the open round brace. That open round brace is part of a pair that the tool knows about, and the range of The most important maxim for data analysis to tokens in between the matching open and closing heed, and one which many statisticians seem to www.computer.org/computingedge 21 RELIABLE CODE

have shunned, is this: “Far better an approximate of a regular expression. The initial star is escaped so answer to the right question, which is often that it cannot be misinterpreted as a metasymbol. The vague, than an exact answer to the wrong ques- dot matches any token text, and the combination.* tion, which can always be made precise.”1 is used to indicate an arbitrary sequence of tokens. Cobra makes sure that the round braces in the pattern The observation was meant as a critique of the match, which forces the text in between the braces to cookie-cutter approach that some of Tukey’s col- refer to a single expression. In an interactive session leagues were using to perform data analysis at the with the Linux source code, matches to this query time: spending little or no time carefully checking pattern are found in roughly 1 s on a multicore system. whether the underlying assumptions for those meth- If you’re curious, there are approximately 6,000 such ods were valid in each case at hand. Perhaps the matches in the 14.9 million lines of Linux code. Here’s same can be said about our use of static source-code the first one, found in file mm/slub.c, online 247: analysis techniques today. Although most tools allow the user to customize the properties checked for to a return *(void **)(object + s−>offset); given code base, these capabilities are rarely used. Several obstacles stand in the way of the routine It is, of course, not much harder to now home in on use of custom properties in the most commonly used more egregious types of pointer arithmetic that use static source-code analyzers. One is the difficulty a sequence of additions, subtractions, and, perhaps, of specifying new types of queries. A person has to even multiplications. Once we have the initial set of be very determined to learn the internal details of matches, those refinements are simple to add interac- the tool, which are, of course, different for each one. tively by querying the range of tokens contained in the Then the average query resolution time for new types braced expression, similar to what we did before when of static analysis can be substantial if the code base querying for-loops. If we do so, we are rewarded with grows to a few million lines. The long waits discourage roughly 70 matches. The following cast, which con- experimentation and targeted approximation to home tains an addition, a subtraction, and a multiplication, in on cases of interest. is found in file drivers/video/console/fbcon.c on lines 2697–2698 (note that the pattern spans multiple lines): CASTING DOUBT Query approximations that can be quickly executed return (u16 *) (vc−>vc_origin + offset - can have real value. Consider how we may try to deter- softback_lines * vc−>vc_size_row); mine how often different types of pointer arithmetic are used in a large code base. If you’re brave enough oth the ease with which we can phrase these to try to express this in one of the customization lan- Bqueries and the speed with which they can be guages of the leading static analysis tools, you are a resolved make a difference. So far though, it’s most better person than I. Here’s how I would do it with a helpful to those old-fashioned types of learning sys- Cobra query that can both be written and executed in tems: us. a few seconds: REFERENCE $ cobra −pat “\* ) ( .* + .* ) ” *.c 1. J. W. Tukey, “The future of data analysis,” Ann. Math. Statist., vol. 33, no. 1, pp. 1–67, Mar. 1962. The pattern I used attempts to match type casts, enclosed in round braces, that end with a star operator, indicating that whatever follows is cast to a pointer. GERARD J. HOLZMANN works on developing Next, the pattern checks that the expression in round stronger methods for the design and analysis braces that follows contains a plus operator, indicating of safety-critical software as a consultant that an addition is being performed. The syntax for the and researcher at Nimble Research. Contact pattern expression we used here is a simplified version him at [email protected].

22 ComputingEdge September 2020 PURPOSE: The IEEE Computer Society is the world’s largest EXECUTIVE COMMITTEE association of computing professionals and is the leading provider President: Leila De Floriani of technical information in the field. President-Elect: Forrest Shull MEMBERSHIP: Members receive the monthly magazine Past President: Cecilia Metra Computer, discounts, and opportunities to serve (all activities First VP: Riccardo Mariani; Second VP: Sy‐Yen Kuo are led by volunteer members). Membership is open to all IEEE Secretary: Dimitrios Serpanos; Treasurer: David Lomet members, affiliate society members, and others interested in the VP, Membership & Geographic Activities: Yervant Zorian computer field. VP, Professional & Educational Activities: Sy-Yen Kuo VP, Publications: Fabrizio Lombardi COMPUTER SOCIETY WEBSITE: www.computer.org VP, Standards Activities: Riccardo Mariani OMBUDSMAN: Direct unresolved complaints to VP, Technical & Conference Activities: William D. Gropp [email protected]. 2019–2020 IEEE Division VIII Director: Elizabeth L. Burd CHAPTERS: Regular and student chapters worldwide provide the 2020-2021 IEEE Division V Director: Thomas M. Conte opportunity to interact with colleagues, hear technical experts, 2020 IEEE Division VIII Director-Elect: Christina M. Schober and serve the local professional community. AVAILABLE INFORMATION: To check membership status, report BOARD OF GOVERNORS an address change, or obtain more information on any of the Term Expiring 2020: Andy T. Chen, John D. Johnson, following, email Customer Service at [email protected] or call Sy-Yen Kuo, David Lomet, Dimitrios Serpanos, +1 714 821 8380 (international) or our toll-free number, Hayato Yamana +1 800 272 6657 (US): Term Expiring 2021: M. Brian Blake, Fred Douglis, • Membership applications Carlos E. Jimenez-Gomez, Ramalatha Marimuthu, • Publications catalog Erik Jan Marinissen, Kunio Uchiyama • Draft standards and order forms Term Expiring 2022: Nils Aschenbruck, • Technical committee list Ernesto Cuadros‐Vargas, David S. Ebert, William Gropp, • Technical committee application Grace Lewis, Stefano Zanero • Chapter start-up procedures • Student scholarship information • Volunteer leaders/staff directory EXECUTIVE STAFF • IEEE senior member grade application (requires 10 years Executive Director: Melissa A. Russell practice and significant performance in five of those 10) Director, Governance & Associate Executive Director: Anne Marie Kelly PUBLICATIONS AND ACTIVITIES Director, Finance & Accounting: Sunny Hwang Director, Information Technology & Services: Sumit Kacker Computer: The flagship publication of the IEEE Computer Society, Director, Marketing & Sales: Michelle Tubb Computer publishes peer-reviewed technical content that covers Director, Membership Development: Eric Berkowitz all aspects of computer science, computer engineering, technology, and applications. COMPUTER SOCIETY OFFICES Periodicals: The society publishes 12 magazines and 18 journals. Washington, D.C.: 2001 L St., Ste. 700, Washington, D.C. Refer to membership application or request information as noted 20036-4928; Phone: +1 202 371 0101; Fax: +1 202 728 9614; above. Email: [email protected] Conference Proceedings & Books: Conference Publishing Los Alamitos: 10662 Los Vaqueros Cir., Los Alamitos, CA 90720; Services publishes more than 275 titles every year. Phone: +1 714 821 8380; Email: [email protected] Standards Working Groups: More than 150 groups produce IEEE MEMBERSHIP & PUBLICATION ORDERS standards used throughout the world. Phone: +1 800 678 4333; Fax: +1 714 821 4641; Technical Committees: TCs provide professional interaction in Email: [email protected] more than 30 technical areas and directly influence computer engineering conferences and publications. IEEE BOARD OF DIRECTORS Conferences/Education: The society holds about 200 conferences President: Toshio Fukuda each year and sponsors many educational activities, including President-Elect: Susan K. “Kathy” Land computing science accreditation. Past President: José M.F. Moura Certifications: The society offers three software developer Secretary: Kathleen A. Kramer credentials. For more information, visit Treasurer: Joseph V. Lillie www.computer.org/certification. Director & President, IEEE-USA: Jim Conrad Director & President, Standards Association: Robert S. Fish BOARD OF GOVERNORS MEETING Director & VP, Educational Activities: Stephen Phillips Director & VP, Membership & Geographic Activities: 24 – 25 September 2020 in McLean, Virginia, USA Kukjin Chun Director & VP, Publication Services & Products: Tapan Sarkar Director & VP, Technical Activities: Kazuhiro Kosuge

revised 1 May 2020 DEPARTMENT: STARTUPS This article originally appeared in

vol. 25, no. 3, 2018 When Multimedia Meets Fashion

Sijie Song, Peking University Tao Mei, JD.com

Through transforming visual fashion into computational images, multimedia technologies are transforming fashion industry at a faster pace than before. The huge demand for fashion analytics triggers promising applications and theoretical research in this area. The future of fashion is being reshaped by multimedia, and researchers are working toward computational fashion.

ven without realizing that artificial intel- In recent years, fashion industry has developed ligence (AI) is no longer a prerogative in mature pipeline from the process of design to produc- E research laboratories of computer science, tion. The rapid increase in fast fashion brands such it is gradually getting well prepared for our everyday as Zara, H&M, and the development in e-commerce lives in a wide range of fields, including health-care, giants, such as Amazon, JD.com, etc., have led the transportation, education, and even fashion. Multime- fashion industry to find tools to accurately locate dia, as an important part in AI, provides powerful tools market orientation and spot potential customers. For for analyzing, understanding, and predicting fashion. these merchandisers, they are dedicated to predict- With rich data posted in the fashion communities and ing the fashion trend to guide their production and online stores from the Internet, multimedia technolo- advertisement spending. They need to know what gies are trying to teach machine to rethink fashion, consumers want to wear, before the consumers even cultivate fashion sense, and creativity. The collision of know themselves. With the technologies in multi- fashion and multimedia technologies would definitely media, the accurate prediction could be achieved bring a new revolution to the fashion industry. through collecting and analyzing large amounts of digitized fashion-related data. The customer's brows- DEMANDS AND STARTUPS ing and shopping history are rich sources for retailers IN FASHION to align the product and demand. On the other hand, Fashion is an integral part of our everyday life. It is merchandisers pay more attention to provide person- about not only what people wear, but also a mirror of alized and diverse services to consumers by analyz- people's attitude toward life, reflections of culture, ing their shopping preference. JD.com, the leading arts, and even economics. Fashion is really a huge e-commerce platform, foresees the fashion future industry of which the total global value is up to 3T dol- powered by multimedia technology. It could provide lars, about 2% of the world's gross domestic product.1 a personalized webpage layout in the app according In general, there are usually three roles involved in the to users’ preference and fashion style. An online retail fashion industry, including merchandisers, design- assistant was also launched to offer professional ers, and customers. With the demands and business style suggestions to customers, as an attempt to digi- opportunities from the three roles, fashion gradu- tize the retail world. ally becomes a hot topic, not to mention the increas- Designers, the creator in the fashion chain, need ing attention from many startup companies. We will to sense the emerging trends from numerous fash- next demonstrate the needs from various aspects and ion shows. They are always supposed to remember some promising startups in the fashion industry. insights from thousands of pictures or videos and,

Functions Item Item Color Style Visual Outﬁt Fashion Industry detection aribute analysis classiﬁcation search recommendation Trend

Markable √√ √ √

Visenze √ √ √ √ √ √

ScopeMedia √ √ √ √ √ √

SenseTime √ √ √ √

ProductAI √ √ √ √

Vue √ √ √ √

Alibaba √ √ √ √ √

JD.com √ √ √ √ √

*Markable: https://markable.ai/; Visenze: https://www.visenze.com/; ScopeMedia: https://scopemedia.com/; SenseTime: https://www .sensetime.com/; ProductAI: https://www.productai.cn/home; Vue: https://vue.ai/; Alibaba: https://www.alibabacloud.com/zh/campaign /fashionai; JD.com: https://www.jd.com/.

then, discover trending elements in their fashion shopping websites, it is much easier for multimedia collections. The collaboration of multimedia and algorithms to learn the rules of outfit matching from fashion design is expected to explore how multimedia the large amounts of online fashion data. Amazon and technologies can take inspirations from the fashion JD.com have provided intelligent speakers to evaluate market as early as possible, and inject creativity into different outfits captured by snapshots, and they are the designer's daily work. For example, the multimedia able to return a result indicating which outfit won out technology can replace the repetitive and low-level over the other. In addition, consistently improving the design works for the designers to improve their pro- shopping experiences is another promising direction in ductivities. The team consisted of students from the fashion industry. Markable, ViSenze, etc., the startup Fashion Institute of Technology and researchers from companies for visual e-commerce, deliver intelligent IBM AI Research managed to use multimedia algo- visual recognition solutions, enabling shoppers to rithms to combine shapes, colors, and patterns as a search and discover products they are already inspired fashion basis for future design.2 to buy. Besides, JD.com has respectively launched Consumers care more about what to wear at a spe- an AR Styling Station and a three-dimensional (3-D) cific occasion and how to be in a fashionable outfit. try-on fitting room to help consumers test makeup The key to this problem lies in the harmonious cloth- or clothes virtually with the convenience of online ing matching. It reminds the startups of the poten- shopping. tial market in personal stylist. With fashion images Overall, startups are trying their best to occupy from online social media such as Pinterest, or online the fashion market. To give an overview of the fashion www.computer.org/computingedge 25 STARTUPS

industry, we list several top companies and the fash- a jumpsuit look like, or what kind of outfits belong to ion functions they are working toward more practical the style of hipster. When the machine learns to rec- applications in Table 1. ognize what is in the image, they are taught to parse the image and understand the content. We expect the TOWARD COMPUTATIONAL machine to achieve finer tasks such as marking the FASHION human body in pixel level, or even locate where the From catwalk to street style, fashion is basically con- hands or feet are. Of course, each pixel in the training veyed and perceived by vision. Thus, it is an excel- images is supposed to be annotated to indicate if it is lent domain for applying technologies from computer in the human body or in the background. To go a further vision and multimedia, where how to convert fashion step, automatic relation discovery is also an important into computational data is the key issue. Essentially, course for the deep learning machines to take. For fashion images are 2-D arrays of numbers know as pix- instance, machines have seen too many images that els. The first step toward computational fashion is to a shirt always go with pants, so that they would not interpret these basic pixel values, and then analyze match two pairs of pants as an outfit. them to achieve further tasks. Besides, vision is not the only way for humans to At the early stage, images are extracted by sense the environment. Other sources from audi- hand-crafted descriptors, which are designed by tion, olfactory, and text contribute a lot to provide humans, such as image gradients, edges, or other cues in the colorful world. Therefore, it might not be image statistics calculated based on the pixel values. enough to equip multimedia technology with fashion Later on, more complex feature descriptors such as sense solely from fashion images. In the fashion local binary pattern, histogram of oriented gradient, area, currently another accessible information is and scale-invariant feature transform were developed text, which can be easily obtained from many fashion to better capture the essential components in images, blogs. By relating text and images, it gives extra guid- regardless of the variation in viewpoints, scale, and ance to better understand images from semantic even brightness. With lots of features from images in perspectives. a specific category, various tools are employed to discover the pattern for recognition. Typical tools in pat- INSPIRING RESEARCH tern recognition include K-nearest neighbors, support The development in multimedia has provided vector machine, and so on. researchers with basic tools to deal with the problems In the recent years, researchers started to train in fashion. However, there are still particular issues in machines to learn the world as humans do. Human fashion domain, including the detection of clothes perceive images through the retina in the eye made attributes and landmark, as well as how to utilize them of millions of individual cells. Our brain assembles all to benefit further fashion analysis. Here, we would like these points and processes the inherent signals and to introduce the progress in fashion research, which structures. For example, after looking lots of blouse would likely accelerate the conversion from laboratory images, our brain is able to conclude the pattern of to startups in the fashion industry, and even impact blouses, which usually consist of a collar, two sleeves, people's lifestyle in the near future. and body. Thus, we are able to recognize the given The topics of fashion research in the literature of image as a blouse or not. Researchers design deep multimedia are summarized in Figure 1. One of the neural networks to simulate densely interconnected branches in fashion research is low-level pixel com- cells, imitating how our brain recognizes patterns and putation, which aims to label each pixel in the scene makes decisions. The models are learning all by itself, (i.e., fashion parsing, landmark detection, and pose just like a brain. That is the wave of deep learning. estimation). Supported by the low-level pixel com- Deep learning methods need to be trained with putation, mid-level fashion understanding emerges, enough learning examples. Thus, content tagging is including style classification, item detection, and required to provide guidance to teach the machine to attribute prediction. More recently, high-level fashion see and sense fashion. We have to tell machine what analysis blossoms in recommendation (including outfit

26 ComputingEdge September 2020 STARTUPS

matching and retrieval), fashion synthesis, and fashion trend prediction, which lead us a step closer to a fashion intelligent assistant.

Low-Level Pixel Computation Fashion parsing refers to generate pixel-level labels on the image, including hair, head, upper clothes, pants, etc., which under- stands images in a finer granular- ity. It is a very challenging problem, since the number of garment FIGURE 1. The research topics in fashion (model images are from DeepFashion types, the variation in configura- Dataset3 and FashionStyle14 Dataset5). tion, and appearance are enormous. Currently, fashion parsing is considered to a restricted domain, where the poten- Then attribute prediction can be applied to resolve tial labels are predefined. the detected fashion items. The attributes predefined Besides, due to the clothes structure, clothes key by human usually include category, texture, fabric, points are helpful to extract discriminative repre- shape, part, and style, which are associated with a sentations to understand fashion images. These key local patch in the images. Thus, the attribute predic- points are referred as fashion landmarks, which were tion depends on the localization of landmarks or first introduced by the research team in the Chinese human skeletal joints. Guided by the key points, infor- University of Hong Kong with a well-annotated fashion mation in the local patches is explicitly or implicitly landmark dataset DeepFashion.3 Landmark detection extracted. Combined with the deep neural network,3 is to localize these key points located at functional we are able to infer predefined attributes with lots of regions. Regression models are usually employed to annotated data. However, annotating attributes is achieve landmark detection. quite a tedious work that costs much time and efforts. Another topic in low-level pixel understanding lies Recently, automatic attribute discovery has attracted in pose estimation. Different from landmark detec- researcher's attention. These methods usually explore tion, human pose estimation is the process of inferring the visual semantic relationships from the fashion 2-D or 3-D human body part positions, such as head, image and corresponding text. The visual attributes torso, shoulders, etc. The results of pose estimation are discovered according to the neural activations.6 and fashion could benefit each other and be refined In addition, style classification plays a key role in iteratively.4 mid-level fashion understanding. The clothes people wear reveal the personal style, bohemia or pinup, Mid-Level Fashion Understanding Goth or preppy. The pipeline for style classification is Based on pixelwise labels, fashion images can be extracting features from images and employing clas- understood at a higher level, making machine to pre- sifiers to identify the specific style.5 How to discover dict what is in the fashion images. First, the machine elements that define a style becomes the key issue. needs to locate the fashion items. Item detection aims to detect various fashion items that a person in High-Level Fashion Analysis the image is wearing or carrying. To achieve this task, When it comes to fashion analysis, many people first item proposals are generated and classified with many think of understanding people's preference to benefit state-of-the-art object detection methods including personalized services.7 With the help of mid-level fash- Faster-RCNN, SSD, YOLO, etc. ion understanding, it is possible to realize customized www.computer.org/computingedge 27 STARTUPS

have gradually shown their power in the fashion industry. Merchan- disers, designers, and customers could all work alongside technology. Statistical models have been developed using large amounts of data to help merchandisers make FIGURE 2. Fancy works from GAN. (a) Sketch to image from Pix2Pix.9 (b) Pose to decisions in marketing. Several image from DAGAN.10 applications from startups have been seamlessly embedded in the products for customers. However, recommendation. There are mainly two aspects in rec- the demanding of designers is far from being satisfied. ommendation: outfit matching and retrieval. Outfit Intelligent design is still a challenge so far. Fortunately, matching aims to find compatible candidates when there have been successful examples showing that a particular item is given. For example, jeans are the line between technology and creativity is gradu- more desired if a white T-shirt is provided. Currently, ally blurring. Machines today can write poems, com- some researchers deal with the task of outfit match- pose music, and create artwork that no one can tell ing by searching visual semantic embedding space,8 is machine generated. So the fashion design is surely where compatible items are supposed to stay closely. not beyond the capabilities of multimedia. Fashion For retrieval, earlier research works focus on similar designers armed with multimedia technology will be retrieval. The returned item is only required to be simi- able to transfer their radical new ideas into products lar to the query in color or style. Exact retrieval to find more efficiently. Intelligent design will amplify design- the exact clothing item is also well developed recently. er's creativity rather than replace it. It will become an This task is more challenging but also more demand- indispensable tool for designers, ultimately improving in practice. ing and fueling the creative processes. To realize In the meantime, the success of generative adver- technology-assisted fashion design, here we list some sarial network (GAN) encourages researchers to promising research directions that might be helpful. devote themselves for fashion synthesis. As shown in Figure 2, GAN is like an amazing brush that could ›› Fashion generation conditioned on text: transfer desired textures from arts or other sources Automatic synthesis of realistic images from to the target items such as coats, shoes, and bags,9 or text would be interesting and important in the even generate images according to text or skeleton.10 design process. Although recent years have The application of GAN in fashion field could benefit seen the progress in this area, the generation virtual try-on, product–human transfer, and ultimately results are only satisfying in limited objects fashion design. such as birds, flowers, etc. The diverse Fashion trend analysis is another rising research attributes of fashion images in color, pattern, topic. With the big data from Internet, researchers are style, and even fabric make it challenging to able to investigate the world's fashion landscape from generate realistic fashion images from text. For a geographic perspective,11 or predict the future of example, the conditional text could be “long fashion from temporal analysis.12 Relying on the statis- sleeve supple leather jacket in light blue with tics of social media, they analyze the visual elements zip closure at front.” Research works on how to shared across images. Finally, researchers track the handle such complex conditions as well as data styles’ spatial or temporal trajectories and provide the sources should be inspired. future trends. ›› Fashion assessment: Currently the evaluation metric for fashion generation is based on incep- FUTURE DIRECTIONS tion score or human preference score. However, We are excited to see that multimedia technologies inception score focuses more on the image

28 ComputingEdge September 2020 STARTUPS

quality, regardless of the aesthetic factors. 4. K. Yamaguchi, et al., “Parsing clothing in fashion Human preference score obtained from a small photographs,” in Proc. IEEE Conf. Comput. Vis. Pattern group can be easily influenced by the users’ Recognit., 2012, pp. 3570–3577. personal preference or the environment. Thus, 5. M. Takagi, et al., “What makes a style: Experimental it is eager to build a novel fashion assessment analysis of fashion prediction,” in Proc. IEEE Int. Conf. metric that is objective and robust. Comput. Vis. Workshop, 2017, pp. 2247–2253. ›› Unsupervised or weakly supervised fashion 6. S. Vittayakorn, et al., “Automatic attribute discovery research: Many fashion tasks rely heavily on with neural activations,” in Proc. Eur. Conf. Comput. annotated data, such as fashion parsing, pose Vis., 2016, pp. 252–268. estimation, landmark detection, item detec- 7. S. Liu, L. Liu, and S. Yan, “Fashion analysis: Current tion, etc. But densely labelling is a tedious and techniques and future directions,” IEEE Multimedia, time-consuming process. More efforts on vol. 21, no. 2, pp. 72–79, Apr.–Jun. 2014. unsupervised or weakly supervised learning 8. X. Song, et al., “Neurostylist: Neural compatibility mod- in fashion domain are necessary to relieve the eling for clothing matching,” in Proc. ACM Multimedia labelling burden. One possible solution is to Conf., 2017, pp. 753–761. utilize text information along with the visual 9. P. Isola, et al., “Image-to-image translation with data. conditional adversarial networks,” in Proc. IEEE Conf. ›› 2D/3D virtual try-on: Although there are much Comput. Vis. Pattern Recognit., 2017, pp. 1125–1134. advancement in fashion virtual try-on, current 10. S. Ma, J. Fu, C. W. Chen, and M. Tao, “DA-GAN: methods are still far from this goal. On the one Instance-level image translation by deep attention hand, clothing is much more difficult to render generative adversarial networks,” in Proc. IEEE Conf. due to deformation and occlusion. On the other Comput. Vis. Pattern Recognit., 2018, pp. 5657–5666. hand, 3-D human body modeling for arbitrary 11. Y. T. Chang, et al., “Fashion world map: Understand- people is still challenging. The virtual try-on ing cities through streetwear fashion,” in Proc. ACM is particularly important for real applications. Multimedia Conf., 2017, pp. 91–99. New models and learning methods need to be 12. Z. Al-Halah, R. Stiefelhagen, and K. Grauman, “Fashion developed to capture personalized shape detail forward: Forecasting visual style in fashion,” in Proc. and clothing geometry. IEEE Int. Conf. Comput. Vis., 2017, pp. 388–397.

The collision of multimedia technologies and fashion is still in its infancy. There is a long way to go. But SIJIE SONG is currently working toward the PhD degree the sparkles have lighted up our passion to trigger the with Peking University, Beijing, China. Her research inter- revolution of fashion. We are looking forward to the ests include computer vision and image processing. She transforming future of fashion and multimedia. received the BS degree in computer science. Contact her at [email protected]. REFERENCES 1. “Global fashion industry statistics—International TAO MEI is a technical vice president with JD.com and the apparel,” 2018. [Online]. Available: https: deputy managing director of JD AI Research, where he also //fashionunited.com/global-fashion-industry serves as the director of Vision and Multimedia Lab. His lab -statistics is focusing on the research, development, and innovation of 2. A. Joseph, “IBM Watson+Tommy Hilfige +FIT: The AI computer vision and multimedia, with a broad range of appli- upgraded designers,” Jan. 26, 2018. [Online]. Available: cations in retail, logistics, finance, cloud computing, as well https://futur404.com/elevated-designer/ as fashion and design. Prior to joining JD.com in 2018, he was 3. Z. Liu, et al., “Deepfashion: Powering robust clothes a senior research manager with Microsoft Research Asia in recognition and retrieval with rich annotations,” in Beijing, China. He received the BE and PhD degrees from the Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, University of Science and Technology of China, Hefei, China. pp. 1096–1104. Contact him at [email protected]. www.computer.org/computingedge 29 EDITOR: Jim Foley, [email protected] This article originally appeared in DEPARTMENT: DISSERTATION IMPACT

vol. 39, no. 2, 2019 Learning to Synthesize and Manipulate Natural Images

Jun-Yan Zhu, Massachusetts Institute of Technology

EDITOR’S NOTE:

Jun-Yan Zhu received the 2018 SIGGRAPH Outstanding Doctoral Dissertation Award.

Humans are avid consumers of visual content. Every day, people watch videos, play games, and share photos on social media. However, there is an asymmetry—while everybody is able to consume visual data, only a chosen few are talented enough to express themselves visually. For the rest of us, most attempts at creating realistic visual content end up quickly “falling off” what we could consider to be natural images. In this thesis, we investigate several machine learning approaches for preserving visual realism while creating and manipulating photographs. We use these methods as training wheels for visual content creation. These methods not only help users easily synthesize realistic photos but also enable previously not possible visual effects.

very day, people consume astounding object detection, or even into a human-like sentence amounts of visual content, as they watch vid- such as “A group of people riding motorcycles on a E eos, play digital games, and share photos on busy city street” via image captioning. However, there social networks. For example, Facebook alone reports is another side of visual intelligence: visual synthesis. 3 million photo uploads per day, and YouTube sees 300 It operates in the opposite direction, from compact hours of video uploaded every single minute. As of concepts back into visual data. In this dissertation,1 I this writing, an estimated 4.7 trillion photos have been would like to teach machines to imagine the realistic taken since the invention of photography, of which visual world from both low-level visual concepts such around 20% are from the past 12 months. as texture and shape, as well as high-level semantic The availability of this big visual data has enabled concepts like objects and scenes. researchers to develop powerful visual understanding But why would visual synthesis be useful? Indeed, methods, which aim at compressing visual data, such it can help address the one-sided nature of communi- as images or videos, into abstract representations. For cation between humans and vast amounts of visual example, a machine can convert an image into a single data. While we all perceive information in the visual word “street” via scene classification, into multiple form through photographs, paintings, sculpture, words “pedestrians,” “motorcycles,” and “buildings” via videos, etc., only a chosen few are talented enough to effectively express themselves visually. This imbal- ance manifests itself even in the most mundane tasks. Digital Object Identifier 10.1109/MCG.2019.2891309 Consider an online shopping scenario. A user looking Date of current version 22 March 2019. for shoes has found a pair that mostly suits them, but

perhaps they would like them to be a little taller, or designed and optimized separately. To take advan- wider, or in a different color. How can the user commu- tage of end-to-end learning, we combine the realism nicate their preference to the shopping website? How classifier and image synthesis program into a single can we recreate the vivid visual world representing the image-to-image translation pipeline. Inspired by a user's mental picture? If the user is an artist, then a few recently proposed method known as generative adver- minutes with an image editing program will allow them sarial networks (GANs),3 we train an image generation to transform the shoe into the desired one, and then network to translate inputs (such as user sketches) a simple image-based search can find it. However, directly to output results, while simultaneously teach- for most of us, even a simple image manipulation in ing a realism classifier to distinguish the synthesized Photoshop presents insurmountable difficulties. One results from natural photographs. Through many reason is the lack of “safety wheels” in visual synthesis qualitative results and human perceptual studies, we and editing: any less-than-perfect edit immediately demonstrate that our proposed methods help users makes the image look completely unrealistic. To put it easily synthesize more visually appealing photos, com- another way, the classic visual synthesis and manipu- pared to the previous state-of-the-art. We also show lation paradigm does not prevent the user from “falling that our methods enable many new visual effects not off” the set of natural images. possible before, such as turning a running horse video In this dissertation, I investigated several data- into a zebra video, generating real photographs from driven visual synthesis approaches for preserving Impressionist paintings, and converting an image cap- visual realism while creating and manipulating photo- tured at night into day images with different types of graphs. Most prior works rely on low-level visual cues, lighting, sky, and clouds. such as color and texture, for modeling visual realism or hand-crafted engineering to reduce artifacts for DISCRIMINATIVE LEARNING OF individual applications. Unlike these methods, we pro- VISUAL REALISM AND AESTHETICS pose to model visual realism directly from large-scale A key open problem in data-driven image synthesis is collections of natural images. We then use the learned how to make sure that the synthesized image looks models as training wheels for visual content creation. realistic, i.e., if it lies in the set of natural images. The We define a class of image synthesis and manipulation human ability to very quickly decide whether a given operations, constraining their outputs to look realistic image is “realistic” is impressive. In contrast, charac- according to the learned visual realism models. terizing this in closed form for a computer is very dif- We first build our methods on two classic machine ficult, and this is precisely what makes good com- learning paradigms: discriminative learning and puter graphics and photographic editing so difficult. generative modeling. For discriminative learning, we So many things must be “just right” for a human train a classifier to predict visual realism and aesthet- to perceive an image as realistic, while a single thing ics. The classifier can then be used for choosing the going wrong will likely hurtle the image down the best-looking photos from a collection, as well as com- Uncanny Valley. puting the optimal parameters of an image editing pro- So what makes an image appear realistic? This is gram. For generative modeling, we directly enforce the one of the most long-standing problems in computer synthesized results to look realistic, as characterized vision. Back in 1999, Huang and Mumford6 studied by the learned image generation models, while satis- the local statistics of images, ranging from a simple fying user constraints. We explore several variants histogram of intensity values to the joint distribution of this idea, from a 19th-century-old image averaging of texture features, such as wavelets. But until now, no model2 to a state-of-the-art deep generative model.3 existing machine learning method has been shown to We present real-time applications such as visual data reliably tell whether a given image looks natural. This exploration and image editing. is because the spectrum of unrealistic images is much In the above-mentioned methods, visual real- larger than the spectrum of natural ones. If this were ism modeling and image synthesis algorithms serve not the case, photorealistic computer graphics would as two independent system components which are have been solved long ago. www.computer.org/computingedge 31 DISSERTATION IMPACT

FIGURE 1. Learning to predict visual realism and photo aesthetics. (a) CNNs are trained to predict if an image composite looks realistic or not.4 (b) A support vector machine (SVM) learns to predict the attractiveness of facial expressions for a given subject.5 This model can automatically mine attractive photos (top row) from a personal photo collection.

FIGURE 2. Our model RealismCNN4 can help create more realistic composites. Our model can (a) improve color compatibility between an object and its background and (b) choose the best object that fits a given background.

Predicting and Improving by its ability to rank image composites by their per- Visual Realism ceived realism. Our model mainly characterizes In our work, RealismCNN,4 we took a discriminative visual realism regarding color, lighting, and texture approach to address a particular instance of the compatibility. above problem, training a high-capacity discrimina- Moreover, we demonstrated that our learned tive model, a convolutional neural network (CNN),7 model can be used as a tool for creating better image to estimate the realism of spliced image composites. composites automatically via simple color adjust- Since it is difficult to obtain enough human-labeled ment. Given a low-dimensional color mapping func- training data to learn what looks realistic, we instead tion, we directly optimized the visual realism score learned to classify between real images and automat- predicted by our CNN model. Our method outperforms ically generated composites. Surprisingly, the result- previous color adjustment methods on a large-scale ing classifier can actually predict how realistic a new human subject study (see Figure 2(a)). We also used composite would look to a human, as shown in Fig- our model to choose an object from a category that ure 1(a). Interestingly, the model appears to be pick- best fits a given background at a specific location (see ing up on cues about visual realism, as demonstrated Figure 2(b)).

32 ComputingEdge September 2020 DISSERTATION IMPACT

FIGURE 3. Examples of interactively discovered modes in the data using AverageExplorer.8 See our interactive interface and more vision and graphics applications in our supplemental video.

Modeling and Improving with a classifier. Therefore, the running time is expo- Photo Aesthetics nential to the number of the parameters in an image Along the same line of thought, we developed a discrim- editing program, which makes it computationally pro- inative learning approach for modeling and improv- hibitive for real-time graphics applications. We can ing the photograph aesthetics of portraits.5 We used perhaps search for the best parameters if the image crowdsourcing to score the portraits for their attrac- editing program is differentiable, but the search space tiveness. We then used these scores to train an SVM is still huge. that can automatically predict attractiveness of dif- To address the issues above, we directly model ferent expressions of a given person. As shown in Fig- the natural image prior via generative models and ure 1(b), the model can evaluate the attractiveness of constrain the output of an image manipulation tool to facial expressions and can automatically mine attrac- look realistic according to the learned prior. Genera- tive photos from personal photo collections. Further- tive models can directly synthesize a realistic image more, we demonstrated a training app that helps peo- given a low-dimensional vector as input, often in ple learn how to mimic their best expressions. See our real-time. This advantage allows us to build interactive training demo5 for more details. data-driven exploration and editing interfaces.

GENERATIVE MODELING FOR Visual Exploration via Image Averaging VISUAL EXPLORATION AND We first studied a simple and old image generation SYNTHESIS model: image averaging, pioneered by Sir Francis Gal- We have demonstrated that discriminative classifiers ton2 in 1878. The image averaging model constrains can learn to accurately predict the photorealism of the the output result to be a convex combination of the result regarding a single factor, such as color compat- database images. Building on this simple model, we ibility4 or facial expressions.5 However, many factors presented AverageExplorer, an interactive frame- play a role in the perception of realism. While a learned work8 that allows a user to rapidly explore and visual- classifier picks up on one visual cue such as color, oth- ize a large image collection using the medium of aver- ers such as lighting, semantics, scene layout, perspec- age images. Our interactive, real-time system provides tive, etc., are also important. Learning a classifier for a way to summarize large amounts of visual data by all the factors is challenging due to the lack of train- weighted averages of an image collection, with the ing data. Besides, to produce a single result, we have to weights reflecting user-indicated importance. As first generate many candidate results and rank them shown in Figure 3, we not only captured the mean of www.computer.org/computingedge 33 DISSERTATION IMPACT

FIGURE 4. Interactive image editing with GANs3,9: we first project an original photo (a) onto a low-dimensional representation (b) by regenerating it using GAN. A user can then modify the color and shape of the generated image (d) using various brush tools (c) (for example, dragging the top of the shoe). Finally, our edit transfer algorithm applies the same amount of geometric and color changes to the original photo to achieve the final result (e). See our interactive image editing video for more details.

the distribution, but also a set of modes, discovered model.9 However, two reasons prevent these advances via interactive exploration. We posed this explora- from being useful in practical applications at this time. tion in terms of a user interactively “editing” the aver- First, the generated images, while impressive, are still age image using various types of strokes, brushes, and not quite photo-realistic. Second, more importantly, warps, with each user interaction providing a new con- these generative models are set up to produce images straint for updating the average. Together, these tools by sampling a vector, typically at random. So, these allow the user to simultaneously perform two funda- methods are not able to create or manipulate visual mental operations on visual data, user-guided cluster- content in a user-controlled fashion. ing and user-guided alignment, within the same frame- To improve the quality and user control, we used work. The supplemental video shows that our system the GANs to learn the prior of natural images, but is useful for various computer vision and graphics we did not actually employ it for image generation. applications. Instead, we used it as a constraint on the output of various image manipulation operations, to make sure the Visual Manipulation With results follow the learned prior at all times. This idea Deep Generative Models enables us to reformulate several editing operations, Simple image averaging models2 can serve as an intu- specifically color and shape manipulations, in a natural itive and artistic medium for visual data exploration. and data-driven way. The model automatically adjusts However, the generated results often look blurry and the output keeping all edits as realistic as possible (see far from realistic, even with the alignment algorithm Figure 4). The supplemental video shows three image implemented in AverageExplorer.8 Furthermore, novel editing applications based on our proposed interface results cannot always be represented as linear com- iGAN10: (1) manipulating an existing photo based on binations of dataset images. To produce more natural an underlying generative model to achieve a different and diverse results, we turned to recently developed look (shape and color), (2) “generative transformation” deep generative models due to the quality and com- of one image to look more like another, (3) generating a plexity of their samples. In the last two years, there has new image from scratch based on user's scribbles and been rapid advancement, fueled mainly by the devel- warping UI. opment of the GANs.3 In particular, recent work has Figure 4 illustrates our approach. Given a real shown visually impressive samples drawn from the photo, we first projected it onto a low-dimensional

34 ComputingEdge September 2020 DISSERTATION IMPACT

FIGURE 5. Given any two unordered image collections X and Y, our algorithm Cycle-Consistent Adversarial Networks (Cycle- GAN)11 learns to automatically “translate” an image from one into the other and vice versa. Example application (bottom): using a collection of paintings of a famous artist, learn to render a user’s photograph into their style.

representation parameterized by GANs. Then, we Unpaired Image-to-Image Translation presented a real-time method for generating a Training a pix2pix12 model requires paired training desired image that both satisfies the user's edits and data, which is often difficult and expensive to col- stays close to the natural image prior. Unfortunately, lect. For example, only a couple of small datasets the generative model usually loses low-level details exist for tasks like semantic segmentation. Obtaining of the input image. Therefore, we proposed an image input-output pairs for graphics tasks such as artistic alignment method that can transfer the edits from stylization can be even more difficult since the desired generated image to the original photo and produce output often requires artistic authoring. To address the final result. this issue, we proposed a model, CycleGAN,11 for learning to translate an image x from a source domain to an IMAGE-TO-IMAGE TRANSLATION image y in a target domain, in the absence of paired In the methods above, the modeling of visual realism examples. Our goal is to learn a mapping G: x y, such and the algorithms for image synthesis are designed that the distribution of images from G(x) is indistin- and optimized separately. To solve such coupled guishable from the data distribution of target images problems jointly, my coauthors and I proposed a y using an adversarial loss. Because this mapping is general-purpose image-to-image translation frame- highly under-constrained, we enforced the additional work, pix2pix,12 where we learned a graphics program constraint that translation should be “cycle consis- to translate inputs (e.g., user sketches) directly to out- tent,” in the sense that if we translate, e.g., a sentence put results, while simultaneously teaching a realism from English to French, and then translate it back from classifier to distinguish the synthesized results from French to English, we should arrive back at the origi- real photos. This method can achieve visually appeal- nal sentence. In practice, we coupled it with an inverse ing results for many problems that traditionally would mapping, F: y x, and introduce a cycle consistency require very different formulations, such as synthesiz- loss to push F(G(x)) ≈ x (and vice versa). Figure 5 shows ing photos from label maps, turning sketches into pic- qualitative results on several tasks where paired data tures, and colorizing images. does not exist, including collection style transfer, www.computer.org/computingedge 35 DISSERTATION IMPACT

FIGURE 6. Multimodal image-to-image translation13: given an input image from one domain (night image of a scene), we can model a distribution of potential outputs in the target domain (corresponding day images), producing both realistic and diverse results.

object transfiguration, season transfer, and photo IMPACT AND APPLICATIONS enhancement. We have described a few data-driven approaches for learning visual realism directly from large-scale image Multimodal Image-to- collections. We used the learned visual realism mod- Image Translation els for realistic image synthesis and editing. The pre- Our methods11,12 can only produce one plausi- sented approaches not only produce more visually ble result given an input image. However, many appealing results compared to previous methods that image-to-image translation problems are inher- rely on hand-crafted engineering or heuristics, but ently ambiguous, as a single input image may corre- also enable several new visual effects, unachievable spond to multiple possible outputs. To address the by any prior work. issue above, my coauthors and I proposed a conditional generative modeling framework13 to model a Research distribution of possible outputs. The ambiguity of Several of our algorithms above, most notably Cycle- the mapping is distilled into a low-dimensional vec- GAN,11 have been used in different fields including tor, which can be randomly sampled at test time. A modifying the style of images, three-dimensional (3-D) generator learns to map the given input, combined models, and animation (graphics), generating syn- with this vector, to the output. We explicitly encour- thetic training data (vision), adapting policy from sim- aged the connection between output and the vec- ulations to the real world (robotics), converting MRIs tor to be invertible. This objective helps prevent a into CT scans (medical imaging), synthesizing voice many-to-one mapping from the vector to the out- and music (audio processing), and changing the sen- put during training, also known as the problem of timent of text (NLP). Even as authors, we are consis- mode collapse and produces diverse results. Our tently surprised by these creative applications, many proposed method encourages bijective consistency of which we never considered ourselves. These individ- between the encoding vector and output modes. Fig- ual applications demonstrate the utility of our work as ure 6 shows an example result of using our method general-purpose unsupervised learning methods for to generate a day image from a night one. Given the transforming data across different domains. same input night image, our method can synthesize day images with different sky color, various light- Education ing effects on the ground, and different density and CycleGAN11 and pix2pix12 have been taught in sev- shape of clouds. eral universities worldwide such as Stanford, MIT, UC

36 ComputingEdge September 2020 DISSERTATION IMPACT

FIGURE 7. Image search with a mental picture: our image synthesis system could be potentially used as an image search interface for efficient browsing.

Berkeley, University of Toronto, as well as in leading it is humans, not machines, who are overwhelmed MOOC platforms such as Udacity and Fast.ai. Cycle- by trillions of photos on the Internet. How can we GAN has also been used as homework and projects in help humans better understand this vast visual the courses above. CycleGAN and pix2pix have also space, to see what is out there? While keywords appeared or will appear in several textbooks. are often used as a proxy for indexing visual content, the visual world is much richer than what can Software be named by words; and yet this process is inevita- We have open-sourced many of our research projects bly lossy and noisy. Fortunately, generative models and online demos for public use. As a result, many have the potential to provide a useful human-data researchers and practitioners have built new models interface. If users can synthesize their mental pic- and practical applications based on our work. Among tures using generative models, systems can under- them, three projects were listed as the top 30 machine stand the intent and retrieve results quickly and learning projects in 2017 (out of 8800 projects) accord- accurately. We performed a preliminary study with ing to a recent survey.14 One of my favorite applica- the AverageExplorer system.8 As shown in Figure 7, tion is cat and dog transformation, where an Internet the user may start with a set of black high heels (left), user from Japan transformed a dog into a cat using and then decide on a different color. Simply drawing CycleGAN. a stroke with the desired red color on the generated image (center) retrieves the available red shoes of Art similar styles. If the user then wants a heel with more A number of our tools have been used widely not just support, adding an edge stroke at the bottom of the by researchers and developers, but also by visual art- shoe switches the heel style (right). In the future, ists. For example, many professional artists and ama- generative modeling will be a promising way to con- teurs have used CycleGAN to perform collection artis- nect a user's mental picture directly to large-scale tic style transfer, including turning a face portrait into datasets. anime art, transforming a real landscape photo into stained glass art, or rerendering everyday drink and Learning Graphics for Building food pictures with flowers sketches.15 These results Autonomous Systems demonstrate exciting, artistic collaborations between Historically, graphics research has mainly focused humans, machines, and algorithms. on creating content for humans. However, computer graphics can also be used to create richly annotated FUTURE DIRECTIONS virtual environments for training autonomous sys- Below, I discuss several potential future directions, tems. Unfortunately, the performance of such systems building on our current image synthesis and manipu- can suffer if the virtual environments used in train- lation algorithms. ing look different from the real-world environments in which the systems are used. To bridge this gap, our Communication Interfaces Between recent work17 learns to adapt virtual CG inputs to the Humans and Visual Data appearance of a real-world test domain (see Figure 8), We are living in an age of big visual data. In the end, significantly improving the model's generalizability. www.computer.org/computingedge 37 DISSERTATION IMPACT

FIGURE 8. CycleGAN11 can learn to bridge the gap between the synthetic virtual world such as the Grand Theft Auto (GTA5) and real-world computer vision datasets such as Cityscapes dataset.16

Moving forward, I plan to build differentiable graph- ACKNOWLEDGEMENT ics pipelines that can easily adapt these methods to a I would like to thank my thesis committee mem- wide range of new datasets and tasks. The parameters bers: A. A. Efros, J. Malik, R. Ng, and M. DeWeese for of the graphics engine will be learned given an objec- their guidance. Special thanks to my collaborators as tive, rather than specified by humans. For example, the well as students, faculty, and postdocs at UC Berke- model would learn to produce training images opti- ley, CMU, and MIT who provided tremendous support mized for a particular vision or robotics application for my research. This work was supported in part by a while matching the appearance of a target domain. Facebook Graduate Fellowship. This paper has supple- mentary downloadable material at http://ieeexplore Beyond 2-D Image Generation .ieee. org, provided by the author. Deep generative models such as GANs3 have been able to produce visually appealing images, enabling REFERENCES computer vision and graphics applications as men- 1. J. Zhu, Learning to Synthesize and Manipulate Natural tioned in this paper.11,12 However, most of the recent Images. Berkeley, CA, USA: UC Berkeley, 2017. learning-based methods focus on generating 2-D 2. F. Galton, “Composite portraits made by combining images. Consider a generative model that can syn- those of many different persons into a single figure,” thesize a car. But what will this car look like from a Nature, vol. 18, pp. 97–100, 1878. new viewpoint? Can we transfer this car's look to a 3. I. Goodfellow et al., “Generative adversarial nets,” in different 3-D CAD model? What does this car look NIPS Proc., 2014. like when it drives on the road and interacts with 4. J.-Y. Zhu, P. Krahenbuhl, E. Shechtman, and A. A. Efros, other cars and pedestrian? None of the above ques- “Learning a discriminative model for the perception tions can be answered with the current 2-D image of realism in composite images,” in Proc. Comput. Vis. generation models. Besides, it is hard to deploy these Pattern Recognit., 2015, pp. 3943–3951. 2-D models to many real-world applications such as 5. J.-Y. Zhu, A. Agarwala, A. A. Efros, E. Shechtman, and J. robotics, virtual reality, and gaming. Therefore, to Wang, “Mirror mirror: Crowdsourcing better portraits,” build intelligent machines that can recreate and ACM Trans. Graph., vol. 33, p. 234, 2014. imagine our visual world, we have to go beyond 2-D 6. J. Huang and D. Mumford, “Statistics of natural images image generation. We need to model the 3-D nature and models,” in Proc. IEEE Comput. Soc. Conf. Comput. of our visual world as well as recreate its dynamics. Vis. Pattern Recognit., 1999, vol. 1, pp. 541–547. We have studied simplified forms of the above prob- 7. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet lem statements such as synthesizing 3-D textured classification with deep convolutional neural net- objects,18 generating realistic videos conditional works,” in NIPS, 2012. on input videos,19 and synthesizing and interpolat- 8. J.-Y. Zhu, Y. J. Lee, and A. A. Efros, “AverageExplorer: ing light field videos.20 These methods still work in Interactive exploration and alignment of visual data restricted domains or require additional inputs as collections,” ACM Trans. Graph., vol. 33, p. 160, 2014. guidance. In the future, a full generative model that 9. A. Radford, L. Metz, and S. Chintala, “Unsupervised can synthesize 3-D or 4-D continuous visual experi- representation learning with deep convolutional ence would be desirable. generative adversarial networks,” in ICLR, 2016.

38 ComputingEdge September 2020 DISSERTATION IMPACT

10. J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. 18. J.-Y. Zhu et al., “Visual object networks: Image genera- Efros, “Generative visual manipulation on the natural tion with disentangled 3D representations,” in NIPS, image manifold,” in Proc. Euro. Conf. Comput. Vis., 2016, 2018. pp. 597–613. 19. T.-C. Wang et al., “Video-to-video synthesis,” in NIPS, 11. J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired 2018. image-to-image translation using cycle-consistent 20. T.-C. Wang, J.-Y. Zhu, N. K. Kalantari, A. A. Efros, and adversarial networks,” in Proc. ICCV, 2017, pp. 2223–2232. R. Ramamoorthi, “Light field video capture using a 12. P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, learning-based hybrid imaging system,” ACM Trans. “Image-to-image translation with conditional adversar- Graph., vol. 36, 2017, Art. no. 133. ial networks,” in Proc. Comput. Vis. Pattern Recognit., 2017, pp. 1125–1134. 13. J.-Y. Zhu et al., “Toward multimodal image-to-image JUN-YAN ZHU is a Postdoctoral Researcher with Massa- translation,” in NIPS, 2017. chusetts Institute of Technology, Cambridge, MA, USA. His 14. Mybridge. 30 Amazing Machine Learning Projects for research interests include graphics, computer vision, and the Past Year, 2018. machine learning. He received the Ph.D. degree in EECS from 15. H. Sarin, “Playing a game of GANstruction,” 2018. UC Berkeley, Berkeley, CA, USA, in 2017, and the B.E. degree 16. M. Cordts et al., “The cityscapes dataset for semantic from Tsinghua University, Beijing, China, in 2012. He received urban scene understanding,” in Proc. Comput. Vis. a Facebook Fellowship and the 2017-18 David J. Sakrisson Pattern Recognit., 2016, pp. 3213–3223. Memorial Prize. Contact him at [email protected]. 17. J. Hoffman et al., “CyCADA: Cycle-consistent adversarial domain adaptation,” in Proc. Int. Conf. Mach. Contact department editor Jim Foley at [email protected] Learn., 2018. .edu.

IEEE COMPUTER SOCIETY Write for the IEEE Computer Society’s authoritative Call for Papers computing publications and conferences.

GET PUBLISHED www.computer.org/cfp

www.computer.org/computingedge 39 COLUMN: EIC MESSAGE This article originally appeared in

vol. 25, no. 3, 2018 Multimedia for Disaster Information Management

Shu-Ching Chen, Florida International University

he global cost of natural disasters in 2017 are presented in recent years to overcome the follow- increased by almost twice from the ones in ing challenges.2–4 T 2016. In particular, disastrous events such as wildfires, flood, hurricanes, and earthquakes caused ›› Time sensitivity and constraint. the economic loss of $306 billion in 2017, which is ›› The heterogeneous nature of data types and much higher than the average 10-year loss of $190 bil- sources including static and dynamic data. lion.1 Due to these significant devastations and losses, ›› Trustworthiness of data sources and security. there is an urgent need for the development of disaster ›› Speed of access to data. information management systems. With the proliferation of social media and smart devices, a considerable amount of multimedia data WHAT IS THE ROLE OF can be collected before, during, and after a disaster. MULTIMEDIA AND SOCIAL MEDIA The disaster-based multimedia data include visual, IN DISASTER RESPONSE AND audio, textual, geographical, sensor, and many other RECOVERY? data types extracted from social networks, mobile Multimedia big data analysis is a key aspect of disas- phones, apps, and emergency websites. This valuable ter information management systems, which effec- information is considered as “big data,” which can be tively manages very large-scale data generated in further integrated and utilized by the community resi- disaster-based circumstances. Techniques such as dents and Emergency Management (EM) responders machine learning and data mining together with multi- for better and faster decision-making and for more media and big data solutions can be utilized to address reliable and accurate situational awareness. the aforementioned challenges. In general, the steps of disaster data analysis can be classified into prepro- CHALLENGES cessing, data selection and summarization, predictive Despite the enormous opportunities provided by mul- analytics, and postprocessing.4 timedia big data for different real-world applications, Nowadays, social media is becoming a big part we are still far from having a comprehensive disaster of everyone's life. Over the last decade, the impact information management system. This system should of social media has been greatly shown in different be able to handle the heterogeneous and large-scale applications from politics to businesses. Now, the data efficiently, analyze the information automatically, question arises is how social media can enhance the summarize and integrate the critical information, and existing disaster management systems. Obviously, it finally deliver fast responses to residents and emer- can be considered one of the most valuable sources gency responders. of information for disaster preparedness, response, The reliable and timely interactions of emergency and recovery.5 It provides amazing opportunities for services are required to diminish the human and prop- training, real-time communication, and data collec- erty losses in a disaster recovery situation. Therefore, tion and analysis. Also, the location-based services in integrated communication and information systems social media can assist emergency responders to save

40 September 2020 Published by the IEEE Computer Society 2469-7087/20 © 2020 IEEE human lives in catastrophic situations and victims to -natural-disasters-and-climate-change-effects get help in a faster and more convenient way. -cost-the-world-306-billion/ 2. T. Li et al., “Data-driven techniques in disaster informa- WHAT IS NEXT? tion management,” ACM Comput. Surv., vol. 50, no. 1, Existing multimedia-aided disaster information man- Mar. 2017, Art. no. 1. agement systems utilize few sources of information 3. V. Hristidis, S.-C. Chen, T. Li, S. Luis, and Y. Deng, (e.g., textual, Geographic Information System (GIS), “Survey of data management and analysis in disaster sensors, etc.). It is essential to collect all critical infor- situations,” J. Syst. Softw., vol. 83, no. 10, pp. 1701–1714, mation from various sources, summarize, and inte- 2010. grate them to adequately assist the EM. However, 4. M. E. P. Reyes, S. Pouyanfar, H. C. Zheng, H.-Y. Ha, and multimodal data models for disaster information man- S.-C. Chen, “Multimedia data management for disaster agement are still in its early stages and more efforts situation awareness,” in Proc. Int. Symp. Sens. Netw., are required to achieve this goal. Syst. Secur., Springer, 2017, pp. 137–146. It is possible to generate interactive models for 5. C. H. Sherwood, “How social media impacts disaster disaster monitoring and situational awareness using response,” ZDNet, 2011. [Online]. Available: https: visualization techniques. Currently, multimedia tech- //www.zdnet.com/article/how-social-media-impacts niques6 and GIS-based models7 have been proposed -disaster-response/ to serve this purpose. However, such systems mostly 6. Y. Yang et al., “MADIS: A multimedia-aided disaster rely on two-dimensional visualizations, which may information integration system for emergency not be easily interpretable by the nonexpert users. management,” in Proc. 8th IEEE Int. Conf. Collaborative Therefore, we can utilize advanced three-dimensional Comput. Netw., Appl. Worksharing, Oct. 14–17, 2012, visualization of real-life GIS data8 to help residents pp. 233–241. and the community for better preparedness and 7. GeoVISTA, Pennsylvania State University, University decision-making. Park, PA, USA, 2010. [Online]. Available: http://www Today, artificial intelligence and machine learning .geovista.psu.edu have been extensively used in real-world applications. 8. M. E. P. Reyes and S.-C. Chen, “A 3D virtual environ- New technologies and tools such as drones, sensors, ment for storm surge flooding animation,” in Proc. 3rd and robots can accurately collect data (damaged IEEE Int. Conf. Multimedia Big Data, Apr. 19–21, 2017, buildings, trapped victims, etc.) and provide informa- pp. 244–245. tion that cannot be easily gathered by the humans. 9. X. Song, R. Shibasaki, N. J. Yuan, X. Xie, T. Li, and R. Also, new machine learning techniques such as deep Adachi, “DeepMob: Learning deep knowledge of learning can be leveraged for understanding the deep human emergency behavior and mobility from big and knowledge of data for planning effective disaster heterogeneous data,” ACM Trans. Inf. Syst., vol. 35, information management and humanitarian relief.9 no. 4, Aug. 2017, Art. no. 41.

REFERENCES 1. A. Rathi, “Global disasters in 2017 cost $306 billion— SHU-CHING CHEN is an eminent scholar chaired professor nearly double the cost of 2016,” Quartz, 2017. [Online]. with Florida International University. Contact him at chens Available: https://qz.com/1162791/2017 @cs.fiu.edu. www.computer.org/computingedge 41 EDITOR: Erik P. DeBenedictis, Zettaflops, LLC, [email protected]

This article originally appeared in DEPARTMENT: REBOOTING COMPUTING

vol. 53, no. 4, 2020

The Quantum Moonshot

Erik P. DeBenedictis, Zettaflops, LLC

This article reviews the IEEE Computer Society’s opportunity and responsibility in the information revolution.

t looks like the world is embarking on a quantum middle of a continuum bounded by CMOS and DNA. moonshot. So I’ll make a projection of how quan- Quantum circuits look like classical circuits, with I tum will partially slip into the technology pipeline gate symbols connected by lines and time going left as a replacement for the transistor, but the effect will to right. This will allow tools for designing circuits grow over time and lead to a new type of computer for and compilers to be upgraded and the users familiar solving problems that are difficult to fully anticipate with existing tools to employ the new ones with little today. Society is in the middle of an information revolu- retraining. While the Boolean logic values of false and tion where humanity is developing the explicit means true can be applied to (some) quantum gates, yielding to control information using not only computers but classical reversible logic, qubits have phase and other also DNA, the Internet, proteins, and other physical properties that go beyond just the bit value. These carriers and processors of information. additional properties change the meaning of what The exponential growth in computing over the look like normal circuits to give them the essential last half-century led humanity to create a pipeline properties that make quantum computers especially for commercializing increasingly powerful comput- powerful for some problems. ers. The pipeline had been fed by smaller and smaller There are efforts worldwide to see how quantum transistors. Yet transistors are close to their limits, and can fit into the existing pipeline that commercializes the pipeline needs a new technology to fuel progress. computers, but I think this is destined to be a story with Still, quantum is not a drop-in replacement for the multiple chapters. The efforts are sometimes called transistor. moonshots, so let’s review the actual moonshot. There There are well-founded beliefs that there are were six Apollo moon landings between 1969 and 1972 profoundly different ways to compute, but building a but none since. I’ll argue that the Apollo moonshot was biological computer, for example, would require inven- a huge success nonetheless. tion of a new design hierarchy to replace gates, soft- The June 1964 issue of IEEE Spectrum1 was a spe- ware, and so on. The key point for the IEEE Computer cial issue with nine articles focused on semiconductor Society is that the innovators in biological computing integrated circuit R&D, which was funded from the call themselves bioengineers instead of computer economic success of discrete transistor avionics and scientists. equipment for the Apollo space program. The business Quantum computers seem destined to fall in the opportunity was leading industry to search for a way to scale integrated circuits, resulting in the 1965 scaling concept later called Moore’s law. Moore’s law powered Digital Object Identifier 10.1109/MC.2020.2972949 an information revolution whose ongoing progress Date of current version: 9 April 2020 supports many IEEE Members, the IEEE Computer

42 September 2020 Published by the IEEE Computer Society 2469-7087/20 © 2020 IEEE Society, and many of IEEE’s publishing and conference simplified educational materials for specialties, such endeavors. as users of applications and frameworks, component Although there were no additional moonshots suppliers, and so forth. after 1972, the Apollo moonshot had spinoffs that sig- There are examples of these mathematically sim- nificantly benefitted the Computer Society and IEEE plified abstractions today, yet they must be invented as a whole. By analogy, we should be prepared for the quantum moonshot to produce an as-yet-unimagined spinoff product that looks as much like current WE SHOULD BE PREPARED FOR THE quantum computers as a smartphone looks like the QUANTUM MOONSHOT TO PRODUCE computer in the Apollo lunar lander. While quantum AN AS-YET-UNIMAGINED SPINOFF computers can perform general-purpose computing, PRODUCT THAT LOOKS AS MUCH LIKE recent efforts in quantum benchmarking and suprem- CURRENT QUANTUM COMPUTERS acy have shown that quantum computers can address AS A SMARTPHONE LOOKS LIKE THE new applications and problems at a greater scale, such COMPUTER IN THE APOLLO LUNAR as the chemical simulation of larger molecules, or per- LANDER. form additional functions for some existing applications, such as optimization. and validated by professor-level experts with a deep THE IEEE COMPUTER SOCIETY IS A understanding of both the math and physics. I’m pro- PEOPLE-PROCESSING MECHANISM jecting that this will be a new and rigorous aspect of I’m suggesting that quantum, if successful, will ulti- computer science and engineering in the future. mately change the interaction between the Computer Professors are created from tenure-track faculty Society and technically oriented individuals over their who compete on the number of peer-reviewed articles lifetimes from middle school onward. Young people they write, which, in turn, depends on organizations become interested in computing based on their affin- that create and run conferences and journals that ity for building things with well-defined structure and handle peer-reviewed research. As a large professional moderate amounts of math. The Computer Society society with integral conference and publications should motivate people with the specific concepts businesses, this puts the IEEE Computer Society at the behind both classical and quantum information, and top of the supply chain. this will mean moving beyond Boolean logic and Tur- ing machines. SCIENCE, TECHNOLOGY, Students, young or old, will need classes with ENGINEERING, AND MATHEMATICS textbooks and labs on quantum. To the extent that EDUCATION AND IMMEDIATE these materials exist today, they are for advanced WORKFORCE AUGMENTATION undergraduates and graduate students in physics and There is an immediate need to train a portion of require an extensive mathematical background. The the workforce to be effective in performing the quantum industry will specialize as it grows, which quantum-related R&D that is being funded world- should make it possible to create mathematically wide. Over time, this immediate need will become an www.computer.org/computingedge 43 REBOOTING COMPUTING

are compatible with their interests and abilities. Figure 1(a) is a smartphone game called Hello Quantum2 from IBM, showing the states of two qubits as eight white, blue, or black circles. Pushbuttons implement gate operations, causing some of the circles to change color. The object of the game is to duplicate the pattern under “Target” along the left edge by push- ing the correct sequence of buttons. It is entertaining in the same sense as a Rubik’s Cube but conveys valid information about how qubits differ from bits and how quantum gates work. Many people have created quantum abstractions, such as a baby book,3 musical notes,4 Powerball tickets,5 and others, but it remains to be seen which ones are effective at both teaching quantum concepts and making quantum interesting and approachable to newcomers.

APPLYING CLASSICAL COMPUTING KNOWLEDGE TO QUANTUM Physicists invented the concepts in quantum computing from the properties of elementary particles and (a) their interactions with one another and their environment. The math is complex, and the interactions with the environment increase the number of equations and variables. However, David Deutsch6 reinterpreted key (b) aspects of particle physics to the terminology of classical reversible logic, leading to something like a quantum logic family. Figure 1(b) shows an example circuit, with horizontal lines conveying information = left to right. Figure 1(c) explains the gate symbol for (c) the three-input, three-output Toffoli gate, after Tom- maso Toffoli. The two-input conditional not, or CNOT, FIGURE 1. (a) The Hello Quantum application for teaching a gate is a Toffoli gate with one control input removed. simple quantum concept to a newcomer to the field. (Source: While quantum gates are intrinsically analog, the logic IBM; used with permission.) (b) A three-qubit quantum circuit family abstraction applies well enough to simplify in conventional digital circuit form. (c) The digital equivalent some tasks. of the Toffoli gate or symbol. Applying the Toffoli function to Technology and educational materials for clas- its output results in reversal of the computation. sical computers have a role in quantum computing, often with minimal adaptation. Students and employ- ees familiar with tools like the SPICE circuit simula- ongoing need to recruit young people to quantum as tor, Visual C++, logic design, and so forth can readily a part of science, technology, engineering, and math understand how those tools and skills can be adapted education efforts. This will also be essential to recruit- to quantum. If a qubit’s value is a bit, that is, a 0 or 1, ing new IEEE Computer Society members. then the diagram in Figure 1(b) has the same meaning The first step in attracting people to quantum is irrespective of whether it is a classical reversible or to convince them that the tasks they would perform quantum circuit. If a qubit has a value including phase

44 ComputingEdge September 2020 REBOOTING COMPUTING

and entanglement, like (|0> + |1>)/√2, then the circuit be to develop an understanding of classical computer must be a quantum circuit to function correctly. There architecture covering multiple temperature ranges, are university classes that teach quantum concepts which is an exciting and crucial new area of engineer- to electrical engineering students designing digital ing innovation.10,11 circuits, such as integrated circuit design.7 MOON LANDING POINT: STEPPING INTO THE UNKNOWN: TURING-COMPLETE QUANTUM ANALOG GATES COMPUTERS Analog computers have a role in computer history,8 Today’s computers are very much focused on the Tur- but computers are overwhelmingly digital today. What ing machine concept and its ability to compute any happened? A modern (nonquantum) analysis shows computable function. (I’m using the phrase Turing that analog computers must overcome fundamental machine where the more precise phrase would be noise limitations to get variables to represent just a bounded-storage machine.) There is a tremendous handful of bits, which usually ends up being too much body of knowledge on making fast Turing-complete effort for the small return. microprocessors, such as superscalar and paral- Computing with qubits is more powerful than lel architectures, along with associated compilers computing with bits because qubits have phase and and parallel algorithms. Quantum computers obtain multiple qubits can be entangled. These additional high performance in other ways. While a pure quan- properties permit some algorithms to solve polynomi- tum computer can be Turing complete, the quantum ally or exponentially larger problems as a function of part alone does not need to be. In fact, a highly capa- the number of qubits or quantum gate operations. ble quantum accelerator coupled to a classical control Controlling noise may have been too much trouble system containing an Arduino microprocessor could in the 1950s, but the exponential speedup of a single be Turing complete at the system level. quantum computer chip today allows it to outrun the I predict that the moonshot will land as a natively world’s largest supercomputer,9 apparently making quantum architecture designed to service a class of the effort worthwhile: not by reducing the effort but by applications. It is clear from current quantum com- increasing the return. puter hardware that quantum computers will excel at Now there is a new engineering subdiscipline for problems such as optimizations, physical simulations, understanding how to build scalable quantum analog perhaps machine learning, and probably others. So circuits,9 which is in the emerging discipline of quan- the moonshot may yield a Turing-complete quantum tum engineering. The IEEE Computer Society may be computer that can handle any computable func- able to shepherd the development of analog coun- tion at today’s speed but is faster than any classical terparts to many of the disciplines such as in CAD, computer at calculating functions that are reliant on compilers, and theory that currently exist for today’s optimization or one of the other quantum accelerator implicitly digital computer science discipline. examples. A natively quantum architecture seems inevitable, CONTROL ELECTRONICS but the quest has not been formalized so far. Today’s Today’s digital computing seems destined to become quantum computers are typically built from field pro- the control electronics for qubits, but heat dissipation grammable gate arrays and lab equipment in an appar- is starting to limit scaling just as much as the qubits ently ad hoc process for each quantum computer themselves. Control signals can be created at room system. There are publications12 but not many. temperature and transmitted to cryogenic quantum components through cables, but the cables conduct he computer industry has taken past advances heat and noise from room temperature and increase Tin stride, such as Moore’s law almost seam- cooling load. If the control signals are generated in the lessly shifting from bipolar transistors to CMOS in vicinity of the cryogenic qubits, the heat dissipation of the 1980s. But the past evolution of computers has the electronics increases cooling load. The remedy will been at the lower levels of the design hierarchy. The www.computer.org/computingedge 45 REBOOTING COMPUTING

computer industry has always been committed to Tur- no. 1868 , pp. 73 – 90 , 1989 . doi: 10.1098/rspa.1989.0099 . ing machines and their ability to compute any comput- 7. B. Lacour , presented at the Quantum Computing able function. Quantum concepts are entering at the Session 01 , 2019 IEEE Int. Conf. Rebooting Computing lower levels but will ultimately change computing at (ICRC), San Mateo, CA, Nov. 6–7 , 2019 , start time: the high levels as well. 3:28:08. [Online]. Available: https://ieeetv The new type of computer would use different .ieee.org/conference-highlights/educational techniques to solve problems. These will obviously be -programs-in-qis-at-ut-austin-brian-la-cour within the space of all computable problems, but the -icrc-san-mateo-2019 new quantum computer would solve problems intrac- 8. W. Karplus , Analog Simulation: Solution of Field Prob- table for a nonquantum computer or, in other words, a lems. New York: McGraw-Hill , 1958 . [Online]. Available: classical computer would take longer than the life of https://onlinebooks.library.upenn.edu/webbin/book the universe to solve the same problem. /lookupid?key=olbp64909 Some of the new problems are readily imagined, 9. S. Boxio , “Cross entropy benchmarking & quantum although obviously we do not have a lot of practi- supremacy ,” presented at 2019 IEEE Int. Conf. Reboot- cal experience solving problems that have not been ing Computing (ICRC) , San Mateo, CA , Nov. 6–7, 2019 . solved before. It is easy to find examples, such as [Online]. Available: http://ieeetv.ieee.org/conference learning faster, factoring of thousand-digit numbers, -highlights/cross-entropy-benchmarking-quantum and modeling the behavior of proteins, that may exist -supremacy-sergio-boixo-icrc-san-mateo-2019 in our bodies and but are too large to simulate with 10. S. J. Pauka et al., A cryogenic interface for controlling classical computers. many qubits. 2019 . [Online]. Available: arXiv:1912.01299 The best endpoint in my view would be for the IEEE 11. D. P. Franke , J. S. Clarke , L. M. K. Vandersypen , and M. Computer Society to expand the traditional view of Veldhorsta , “ Rent’s rule and extensibility in quantum computers to include support for digital bits, analog computing,” Microprocess. Microsyst., vol. 67 , pp. 1 – 7 , qubits, and cryogenic operation (where needed for June 2019 . doi: 10.1016/j.micpro.2019.02.006 certain types of qubits). These expanded computers 12. A. Butko et al., Understanding quantum control would be Turing machines, yet able to solve problems processor capabilities and limitations through circuit effectively in a larger space that includes the capa- characterization. 2019 . [Online]. Available: https: bilities of both today’s microprocessors and quantum //arxiv.org/abs/1909.11719 computers.

REFERENCES ERIK DEBENEDICTIS is the principal of Zettaflops, LLC, 1. IEEE Spectrum, June 1964 . [Online]. Available: https: editor-in-chief of IEEE Transactions on Quantum Engineer- //ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber= ing, and a volunteer supporting the IEEE Computer Society, 6500551&punumber=6 Council on Superconductivity, Rebooting Computing, and 2. Hello Quantum. Accessed on: Jan. 31 , 2020 . [Online]. Quantum Initiative. Contact him at erikdebenedictis@gmail Available: https://helloquantum.mybluemix.net .com. 3. C. Ferrie and Whurley , Quantum Computing for Babies. Naperville, IL: Sourcebooks , 2018 . 4. “ EPiQC: Developing and communicating the CS in quantum computing,” Canonlab . Accessed on: Jan. 31 , 2020 . [Online]. Available: https://www.canonlab.org /quantumvideos 5. E. DeBenedictis , “Powerball and quantum supremacy,” Computer, vol. 52 , no. 10 , pp. 110 – 112 , 2019 . doi: 10.1109 /MC.2019.2930166. 6. D. E. Deutsch , “Quantum computational networks,” WWW.COMPUTER.ORG/COMPUTINGEDGE Proc. Roy. Soc. London. A. Math. Phys. Sci., vol. 425 ,

46 ComputingEdge September 2020 Get Published in the New IEEE Open Journal of the Computer Society

Submit a paper today to the premier new open access journal in computing and information technology.

Your research will benefit from the IEEE marketing launch and 5 million unique monthly users of the IEEE Xplore® Digital Library. Plus, this journal is fully open and compliant with funder mandates, including Plan S.

Submit your paper today! Visit www.computer.org/oj to learn more. EDITOR: Erik P. DeBenedictis, Sandia National Laboratories, [email protected]

This article originally appeared in DEPARTMENT: REBOOTING COMPUTING

vol. 51, no. 12, 2018

The National Quantum Initiative Will Also Benefit Classical Computers

Erik P. DeBenedictis and Michael P. Frank, Sandia National Laboratories

The U.S. National Quantum Initiative places quantum computer scaling in the same category as Moore’s law. While the technical basis of semiconductor scale up is well known, the equivalent principle for quantum computers is still being developed. Let’s explore these new ideas.

regular, or classical, computer delivers value layers allow a second extension to novel devices and to the user based on the complexity of the circuits, including sensing devices that require low A calculation it performs. Classical computers temperatures to function and computing devices to are overwhelmingly based on transistors, and Moore’s support them. law showed how to make transistors cheaper, faster, and more energy efficient over time. While quantum LOGIC PERFORMANCE computers are expected to be built mostly from clas- AND TEMPERATURE sical devices as well, the value delivered to the user There was an extensive search for transistor replace- will depend on the qubits, not the supporting devices. ment as a potential remedy to the so-called end of The point of this article is to demonstrate that shifting Moore’s law, ultimately producing Figure 1’s scatter- classical devices from the primary role to a supporting plot of energy and delay for many candidate devices.1 role will transform perspectives about computers. Focusing just on the top group for now, we see that Let’s introduce this new thinking by consider- the three data points with whiskers are Josephson ing the central 3D structure of a quantum computer, junction-based reciprocal quantum logic (RQL) and which contains the qubits and a portion of the control adiabatic quantum flux parametron (AQFP) logic fam- system. While nobody knows which type of qubit will ilies. The dots for these cryogenic devices show the go into large-scale production, most candidate qubits sum of device energy per operation plus the additional require cryogenic operation. Through-silicon vias energy used by the cryogenic refrigerator to move (TSVs) are a current example of 3D technology. They heat from 4 K to room temperature; the whisker shows are microscopic bumps that electrically and mechani- the result over a refrigerator efficiency range. None of cally interconnect up to roughly half a dozen integrated the devices in the top group stands out. circuit die along their planar faces to a 3D module. When low-temperature operation is required, all the The first advance we’re seeing is the extension of devices in Figure 1 need refrigeration, and the method these approaches to multiple operating temperatures, of tallying the energy cost of cryogenic refrigeration such as room-temperature layers plus other layers at changes. In low-temperature operation, devices in one or more cryogenic temperatures. The cryogenic both the upper and lower groups operate at temperature T and require a cryogenic refrigerator to remove heat to room temperature. A 100% Carnot-efficient Digital Object Identifier 10.1109/MC.2018.2880025 refrigerator will multiply the heat by 300 K/T in the Date of publication: 5 February 2019 process of moving it to the room temperature (300

RQL 1,000 1.E+2 CMOS HP

1.E+1 CMOS LV AQFP (20 GHz) 1,000

1.E+0 AQFP (5 GHz) 1,00 0

1.E–1 RQL Energy (aJ)

1.E–2

AQFP (20 GHz) 1.E–3 AQFP (5 GHz)

1.E–4 1.E–1 1.E+0 1.E+1 1.E+2 1.E+3 Delay (ps)

FIGURE 1. An energy-delay plot of a comprehensive set of logic devices at room temperature, including Josephson junctions operating at 4 K in three circuits (RQL and, at two speeds, AQFP). None of the devices stands out at room temperature because only the superconducting ones have refrigeration overhead. However, at 4 K, they all require refrigeration, causing the superconducting devices to stand out. (The outlier purple dot is a BisFET, which is too immature to be considered a serious contender.)

K) environment, and refrigerator inefficiency adds good deal. If the increase in purchase price is greater another multiple of 5–20×. However, these factors are than the energy savings over the bulb’s lifetime, it’s the same for all devices, so the best options will be the a bad deal. Energy-savings technology for comput- points closest to the origin of the graph, which are the ers, such as reversible and adiabatic logic, have been cryogenic Josephson junction-based devices. known for years. However, they trade energy efficiency during operation for more complex circuits DIFFERENT CIRCUITS that cost more to buy in the first place—leading to the BECOME EFFECTIVE AT LOW same tradeoff as for light bulbs. TEMPERATURES All other things being equal, you become more Everybody wants to save energy, which is why wise likely to choose the energy-efficient technology as engineers consider energy-efficient light bulbs the price of energy rises. Energy-efficient lights and for their homes. However, once they see that the reversible computers that are bad deals at room energy-efficient bulb costs ten times as much as an temperature become better deals at 4 K because of incandescent one, they do a calculation to see if it’s a the effective 1,000× increase in energy cost—and are www.computer.org/computingedge 49 REBOOTING COMPUTING

Power Versus Frequency, TSMC 0.18, Std. CMOS Versus 2LAL 1.E–05 slope −1 (linear), while 2LAL has slope −2 (quadratic). This creates 1 nJ 1.E–06 a widening advantage for 2LAL at lower clock frequencies, which 100 pJ 1.E–07 eventually ends because of leak-

10 pJ age current Ioff. However, a 200-kHz 10 aJ Standard 1.E–08 microprocessor is not very useful, CMOS 1 pJ even if it is energy efficient. 1 aJ 1.E–09 What happens to reversible cir-

1 eV 100 fJ cuits at cryogenic temperatures? 1.E–10 Studies of standard CMOS transis- 100 zJ 2 V tors operated at 4 K show a large 1 V 10 fJ 3 1.E–11 10 zJ reduction in leakage current, 0.5 V

1,000 while other factors stay about the

kJ in2 0.25 V Energy Dissipated per nFET Cycle Average Power Dissipation per nFET (W) 1.E–12 1 fJ same. Although cryogenic 2LAL 1 zJ has not been systematically ana- 1.E–13 lyzed for this article, the qualitative 100 yJ result annotated on top of Figure 1.E–14 2’s base graph is an extension of 1.E+09 1.E+08 1.E+07 1.E+06 1.E+05 1.E+04 1 the region with downward slope –2

Frequency (Hz) 100,000 by an additional factor of perhaps 10,000× before leakage current Ioff causes the upward sloping behavior. A review of the literature indicates that the advantage should increase a bit more below 4 K and then level off, but further work is FIGURE 2. A comparison of circuit efficiency for standard CMOS (top line) and a indicated. reversible circuit 2LAL, showing a maximum advantage of 1,000× at 200 KHz. How- The exponential growth of ever, if the 2LAL is operated at 4 K, downward-sloping curves should extend further, Moore’s law made semiconduc- leading to a possible 100,000× energy-efficiency improvement over room-tempera- tors seem unbeatable for decades. ture electronics. This may allow a transistorized 2LAL to compete with Josephson However, recent changes in junctions in applications where speed is not essential. FET: field-effect transistor; Moore’s law have enabled super- freq.: frequency; TSMC: Taiwan Semiconductor Manufacturing Company, Limited. conductors to pull ahead for quantum computers. But wait! Now that we take time to think, semiconduc- no-brainers for systems operating at millikelvins, such tors can play this new game, too, by switching from as quantum computers, where the effective cost rises CMOS circuits to 2LAL or some other adiabatic form. 1 million times. If thinking hard can rejuvenate both semiconduc- Adiabatic and reversible logic have been studied tors and superconductors, what could happen if we at room temperature for many years, and the plot in thought about using both at once? Figure 2 shows an example of the benefits when implemented with standard CMOS. The curves are the result HETEROGENEOUS INTEGRATION of many simulations and show transistor power declin- OF SEMICONDUCTORS AND ing with clock period for both standard CMOS and SUPERCONDUCTORS a reversible logic family called 2LAL.2 On the log-log One of us saw a presentation at a recent conference scale, the power dissipation of CMOS declines with reporting on the invention of a superconducting

50 ComputingEdge September 2020 REBOOTING COMPUTING

Gate field-effect transistor.4 It was kind Source Drain Josephson Weak of an odd device. The application Link or Resistor of 90 V would switch it from a 0-Ω Extra Contacts superconductor to a resistor of 50 for Four-Point Measurement Gate Ω or so. Nobody had any idea what to do with it, but we’ll suggest that Capacitor Superconductor Electric it could interface between 2LAL Layer Field and Josephson junction logic in a Semiconductor semiconductor–superconductor CMOS or hybrid module. DRAM Layer

Although the device was Row experimentally tested with 90-V Drive Programmable Voltages swings, the authors claim that Column scaling could reduce the drive Drive voltage to 2.5 V, a typical CMOS voltage swing. This could lead to FIGURE 3. A possible semiconductor-superconductor hybrid. The semiconductor a structure like the one shown in layer applies voltage-based signals to the gates of superconducting FETs, which Figure 3, where CMOS voltage sig- translate the signals into a form readily used by superconducting circuits. The nals are converted to Josephson hybrid would be fabricated using a CMOS wafer as a base for depositing supercon- junction current signals. ductor circuits, in lieu of today’s method of using a blank silicon wafer as a base. The inset in Figure 3 shows DRAM: dynamic random-access memory. a superconducting field-effect transistor (FET).4 Ignoring the green structures for the moment, the blue structure different directions recently. We’ve seen how several is a superconducting wire that will conduct current of these directions can be pursued at once to make a horizontally with zero resistance. However, a narrow more audacious hybrid than has been seen before. superconducting wire conducts with zero resistance Running both semiconductors and superconduc- only up to a maximum current, called the critical cur- tors at 4 K, with suitable circuits and an efficient, rent, above which the device becomes a resistor. A integrated interface between the two, yields a set of narrow wire of this type is called a weak link and is a building blocks with a unique set of properties: type of a Josephson junction. However, the critical current declines in the pres- ›› Semiconductors: A large number of components ence of an electric field, such as the field applied by are available per unit area, and the technology the green capacitor plate in Figure 3 and the cor- has achieved high maturity. Signals have 1- to responding green structure in the inset. Because 2-V swings. However, achieving high-energy superconductor circuits are built from wires with efficiency requires progressively reducing the zero resistance, voltages in a superconductor circuit clock rate. almost always stay between ground and a few milli- ›› Superconductors: Clock rate remains fast down volts. Theory and experiment for the superconducting to the lowest temperatures. These relatively FET show that the weak link’s critical current changes immature devices have large features that would when the green structure applies few volts or more, result in low device density. Signal voltages positive or negative, which is straightforward for tran- are around a millivolt, but there is a method sistorized electronics. Hence, the semiconductor layer for conversion from 1- to 2-V signals from would communicate with the superconducting layer semiconductors. through voltages that change critical currents. Attempts to restore traditional computer per- Is there any circuit in common use that could ben- formance growth rates, like Moore’s law, have taken efit from such an oddball combination of features? Our www.computer.org/computingedge 51 REBOOTING COMPUTING

first thought is a superconducting field-programmable represent the views of the U.S. Department of Energy gate array (FPGA),5 but with a new design where the or the United States Government. configuration logic is implemented with semiconductors. Because it runs on only power-up, it doesn’t have REFERENCES to be fast. After the FPGA has been configured, the 1. D. E. Nikonov and I. A. Young, “Overview of Josephson junction logic executes the user-defined beyond-CMOS devices and a uniform methodology logic at high speed. for their benchmarking,” Proc. IEEE, vol. 101, no. 12, pp. 2498–2533, 2013. doi:10.1109/JPROC.2013.2252317. he standard process for computer design is 2. V. Anantharam, M. He, K. Natarajan, H. Xie, and M. Tto create a Turing-complete processor out of P. Frank. “Driving fully-adiabatic logic circuits using universal gates and then add I/O as an afterthought. custom high-Q MEMS resonators,” in Proc. Int. Conf. However, a quantum computer will have two types Embedded Systems and Applications and Proc. Int. of universal gates: classical and quantum. Standard Conf VLSI (ESA/VLSI). Las Vegas, NV, 2004, pp. 5–11. computer design would lead to a quantum computer 3. R. M. Incandela, L. Song, H. Homulle, E. Charbon, A. system composed of a quantum processor connected Vladimirescu, and F. Sebastiano. “Characterization and to a classical control system via I/O. But this is not to compact modeling of nanometer CMOS transistors at be. Qubits offer their exotic features only in specific deep-cryogenic temperatures,” IEEE J. Electron Devices environments, such as cryogenic temperatures, Soc., vol. 6, pp. 996–1006, Apr. 2018. whereas classical electronics offer both density and 4. G. De Simoni, F. Paolucci, P. Solinas, E. Strambini, and F. performance only at room temperature. This article Giazotto. “Metallic supercurrent field-effect transis- showed how the requirements of each intrude on the tor,” Nature Nanotechnology, vol. 13, no. 9, pp. 802–805, design space of the other, destroying the standard 2018. doi:10.1038/s41565-018-0190-3. abstraction. 5. N. K. Katam, O. A. Mukhanov, and M. Pedram, “Super- So “universal” Boolean logic gates actually have conducting magnetic field programmable gate array,” additional properties attached to them (i.e., density IEEE Trans. Appl. Supercond., vol. 28, no. 2, pp. 1–12, and performance at room temperature). The impact 2018. doi:10.1109/TASC.2018.2797262. of these additional properties was invisible when we only had one example of them. However, mandates to scale up quantum computers force us to deal ERIK P. DEBENEDICTIS is a technical staff member at Sandia with quantum gates where the additional properties National Laboratories Center for Computing Research. He is are incompatible. We must now learn to design with a Senior Member of the IEEE and a member of ACM and APS. “generalized universal gates,” after which we need Contact him at [email protected]. to revise the standard process for computer design. While quantum computing has the lion’s share of pub- MICHAEL P. FRANK is a senior technical staff member at lic attention right now, the same technology could Sandia National Laboratories Center for Computing Research. apply to cryogenic sensor systems that rely on quan- He is a Member of the IEEE. Contact him at mpfrank@sandia tum information and which may enable scientific .gov. breakthroughs.

ACKNOWLEDGMENTS Sandia National Laboratories is a multimission laboratory managed and operated by National Technol- ogy and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International, Inc., for FOLLOW US the U.S. Department of Energy’s National Nuclear Secu- @securityprivacy rity Administration under contract DE-NA-0003525. The views expressed in the article do not necessarily

52 ComputingEdge September 2020 Evolving Career Opportunities Need Your Skills Explore new options—upload your resume today

Changes in the marketplace shift demands for vital skills and talent. The IEEE Computer Society Jobs Board is a valuable resource tool to keep job seekers up to date on the dynamic career opportunities offered by employers. Take advantage of these special resources for job seekers:

JOB ALERTS TEMPLATES WEBINARS

CAREER RESUMES VIEWED ADVICE BY TOP EMPLOYERS

No matter what your career level, the IEEE Computer Society Jobs Board keeps you connected to workplace trends and exciting career prospects.

www.computer.org/jobs Call for Papers: IEEE Transactions on Computers

Publish your work in the IEEE Computer Society’s ﬂ agship journal, IEEE Transactions on Computers (TC). TC is a monthly publication with a wide distribution to researchers, industry professionals, and educators in the computing ﬁ eld.

TC seeks original research contributions on areas of current computing interest, including the following topics:

• Computer architecture • Security and reliability • Software systems • Machine learning • Mobile and embedded systems • Quantum computing

All accepted manuscripts are automatically considered for the monthly featured paper and annual Best Paper Award.

Learn about calls for papers and submission details at www.computer.org/tc. IEEE Internet Computing delivers novel content from academic and industry experts on the latest developments and key trends in Internet technologies and applications. Written by and for both users and developers, the bimonthly magazine covers a wide range of topics, including:

• Applications • Architectures • Big data analytics • Cloud and edge computing • Information management • Middleware • Security and privacy • Standards • And much more In addition to peer-reviewed articles, IEEE Internet Computing features industry reports, surveys, tutorials, columns, and news. www.computer.org/internet

VOLUME 22, NUMBER 2 MARCH/APRIL 2018 IEEE INTERNET COMPUTING

July/August 2018 VOLUME 22, NUMBER 4 JULY/AUGUST 2018 IEEE INTERNET COMPUTING IEEE INTERNET COMPUTING Evolution of Rack-Scale Systems

January/February 2018 VOLUME 22, NUMBER 1 JANUARY/FEBRUARY 2018

May/June 2018 VOLUME 22, NUMBER 3 MAY/JUNE 2018 IoT-Enhanced Human Experience Connected and Autonomous Vehicles

IoT-Enhanced Human Experience Evolution of Rack-Scale Systems Volume 22 Number 1 Volume 22 Number 4

www.computer.org/internet

Healthcare Informatics and Privacy www.computer.org/internet

Connected and Autonomous Vehicles

Join the IEEE ComputerVolume 22 Number 3 Society

www.computer.org/internet for subscription discounts today!www.computer.org/internet www.computer.org/product/magazines/internet-computing IEEE Computer Architecture Letters is a forum for fast publication of new, high-quality ideas in the form of short, critically refereed technical papers. Submissions are accepted on a continuing basis and letters will be published shortly after acceptance in IEEE Xplore and in the Computer Society Digital Library. Submissions are welcomed on any topic in computer architecture, especially:

• Microprocessor and multiprocessor systems • Microarchitecture and ILP processors • Workload characterization • Performance evaluation and simulation techniques • Interactions with compilers and operating systems • Interconnection network architectures • Memory and cache systems • Power and thermal issues at the architectural level • I/O architectures and techniques • Independent validation of previously published results • Analysis of unsuccessful techniques • Domain-speciﬁ c processor architecture (embedded, graphics, network) • High-availability architectures • Reconﬁ gurable computer architectures www.computer.org/cal

Join the IEEE Computer Society for subscription discounts today! www.computer.org/product/journals/cal IEEE Security & Privacy is a bimonthly magazine communicating advances in security, privacy, and dependability in a way that is useful to a broad section of the professional community. The magazine provides articles with both a practical and research bent by the top thinkers in the ﬁ eld of security and privacy, along with case studies, surveys, tutorials, columns, and in-depth interviews. Topics include:

• Internet, software, hardware, and systems security • Legal and ethical issues and privacy concerns • Privacy-enhancing technologies • Data analytics for security and privacy • Usable security • Integrated security design methods • Security of critical infrastructures • Pedagogical and curricular issues in security education • Security issues in wireless and mobile networks • Real-world cryptography • Emerging technologies, operational resilience, and edge computing • Cybercrime and forensics, and much more www.computer.org/security

E-Currency and Fairness ■ Ransomware Defense ■ A National Cybersecurity Policy & PRIVACY SECURITY IEEE

Software and Cybersecurity ■ Big Data: Privacy Versus Accessibility ■ Resiliency in Cloud Computing ETHICS AI

Blockchain Technologies ■ The Fuzzing Revival ■ Cybersecurity for the Public Interest & PRIVACY SECURITY IEEE & PRIVACY SECURITY IEEE DIGITAL FORENSICS, PART 2 FORENSICS, DIGITAL SCREENING AUTOMATED AIRPORT AND PRIVACY VOLUME 17 VOLUME

VOLUME 16 VOLUME NUMBER 1

JANUARY/FEBRUARY 2019

NUMBER 3

January/February 2019 CYBERSECURITY AND WWW.COMPUTER.ORG/SECURITY Vol. 17, No. 1

VOLUME 17 VOLUME PRIVACY ISSUES IN BRAZIL

November/December 2018 Vol. 16, No. 6 NUMBER 2 MAY/JUNE 2018 MAY/JUNE

MARCH/APRIL 2019MARCH/APRIL

WWW.COMPUTER.ORG/SECURITY May/June 2018 Vol. 16, No. 3 March/April 2019 Join the IEEE ComputerWWW.COMPUTER.ORG/SECURITY SocietyVol. 17, No. 2 for subscription discounts today! www.computer.org/product/magazines/security-and-privacy Only Quantum Can Take On Quantum. Most pproches to quantum Join Zapata at the ready cybersecurity fall down outside a laboratory environment. forefront of quantum They require new dark fibre infrastructure, are too expensive, or unscalable — or otherwise unfit We are looking for creative, curious, and for the real world. revolutionary mindsets. Is that you? Meanwhile, Quantropi’s unique, Then join our world-class team of resourceful patented QEEP™ quantum gate problem solvers who are pioneering commercial technology is not just enterprise QEEP•KD application layer quantum algorithms, product development and grade and application ready. It solution for quantum key generation, management & distribution. research. We are committed to creating an works right here, right now, on ANY existing network. Even the environment that accelerates your growth and WIRELESS Internet. positive impact in the world. Bring it on. Industry leading performance. We are growing rapidly across our various teams. Plug and play deployment. Search open jobs and fill out a profile with your Unbreakable confidence. resume at zapatacomputing.com/careers so we quantropi.com can contact you for future openings. Today, tomorrow, forever. [email protected]

ZAPATACOMPUTING.COM/CAREERS

ComputingEdge September 2020 Quantropi_IEEE-Ad_2020b.indd58 1 2020-08-20 10:07 stay

on the When we launched the ﬁrst quantum Cutting Edge computer accessible on the cloud in 2016, we were astonished when we reached of Artificial Intelligence 8,000 users. Today, we’ve reached over 250,000 users. This is the world’s largest and most IEEE

January/fEbruary 2016 engaged quantum developer community Also in this issue: IEEE Intelligent Systems provides peer- aI’s 10 to Watch January/FEB 56 real-Time Taxi Dispatching 68 running over 1 billion circuits daily on a f rom flu Trends to Cybersecurity 84 ruary IEEE 2016

fleet of the world’s most powerful quantum PUTTING AI INTO PRACTICE reviewed, cutting-edge articles on the computers, all on the IBM Cloud.

Online Beh theory and applications of systems Our next goal is to reach the world’s 25 A vi OrAl AnA lysis OrAl million developers, to empower them to

that perceive, reason, learn, and put quantum to use in solving today’s VOL

uME 31 nu uME intractable challenges. MBE r 1 act intelligently. www.computer.org/intelligent IS-31-01-C1 Cover-1 January 11, 2016 6:06 PM Our strategy is to make quantum computing as seamless and as easy to use as today’s classical programming frameworks.

If you’d like to learn more, please contact The #1 AI Magazine [email protected] www.computer.org/intelligent IEEE IEEE COMPUTER GRAPHICS AND APPLICATIONS APPLICATIONS AND GRAPHICS COMPUTER IEEE IEEE COMPUTER GRAPHICS AND APPLICATIONS APPLICATIONS AND GRAPHICS COMPUTER IEEE IEEE COMPUTER GRAPHICS AND APPLICATIONS APPLICATIONS AND GRAPHICS COMPUTER IEEE IEEE COMPUTER GRAPHICS AND APPLICATIONS APPLICATIONS AND GRAPHICS COMPUTER IEEE

November/December 2016 July/August 2016 September/October 2016 January/February 2017 November/December 2016 September/October 2016 January/February 2017January/February

July/August 2016July/August Qualit Assessment and Defense Quality Assessment and Perception in Computer Graphics Computer in Perception and Assessment Quality Perception Applications Element Human Water, the Sky, and

in Computer Graphics Visualization Data Sports Defense Applications

VOLUME 36 NUMBER 4 NUMBER 36 VOLUME VOLUME 37 NUMBER 1 37 NUMBER VOLUME VOLUME 36 NUMBER 5 NUMBER 36 VOLUME VOLUME 36 NUMBER 6 NUMBER 36 VOLUME

c1.indd 1 12/14/16 12:21 PM

c1.indd 1 6/22/16 1:20 PM c1.indd 1 8/22/16 2:59 PM c1.indd 1 10/24/16 3:44 PM

CG& www.computer.org/cgaA IEEE Computer Graphics and Applications bridges the theory and practice of computer graphics. Subscribe to CG&A and • stay current on the latest tools and applications and gain invaluable practical and research knowledge, • discover cutting-edge applications and learn more about the latest techniques, and • benefit fromCG&A ’s active and connected editorial board.

www.computer.org/computingedge 59 Thank You to Our Sponsors!

CVPR 2020—the premier annual conference on computer vision and pattern recognition—virtually delivered more than 5,000 ﬁ rst-class papers, keynotes, sessions, workshops, and tutorials to an audience of 7,600 from all over the world!

We are so grateful to our wonderful volunteers and our amazing lineup of sponsors for their continuing support.

Champion

Supporter

cvpr20.com IEEE TRANSACTIONS ON SUBMIT TODAY BIG DATA

SCOPE

The IEEE Transactions on Big Data (TBD) publishes peer reviewed articles with big data as the main focus. The articles provide cross disciplinary innovative research ideas and applications results for big data including novel theory, algorithms and applications. Research areas for big data include, but are not restricted to, big data analytics, big data visualization, big data curation and management, big data semantics, big data infrastructure, big data standards, big data performance analyses, intelligence from big data, scientific discovery from big data security, privacy, and legal issues specific to big data. Applications of big data in the fields of endeavor where massive data is generated are of particular interest.

SUBSCRIBE AND SUBMIT For more information on paper submission, featured articles, calls for papers, and subscription links visit: www.computer.org/tbd SHARE AND MANAGE YOUR RESEARCH DATA IEEE DataPort is an accessible online platform that enables researchers to easily share, access, and manage datasets in one trusted location. The platform accepts all types of datasets, up to 2TB, and dataset uploads are currently free of charge.

2TB Cloud Storage Supports Research Open Access Options Per Dataset Reproducibility

Link Dataset to Generates Citations ORCID Manuscript Integration

Host Data Competitions DOI Provided Broad Range of Data Topics

UPLOAD DATASETS AT IEEE-DATAPORT.ORG stay connected.

Keep up with the latest IEEE Computer Society publications and activities wherever you are. Follow us:  | @ComputerSociety

 | facebook.com/IEEEComputerSociety

 | IEEE Computer Society

 | youtube.com/ieeecomputersociety

 | instagram.com/ieee_computer_society Conference Calendar

EEE Computer Society conferences are valuable forums for learning on broad and dynamically shi ing top- Iics from within the computing profession. With over 200 conferences featuring leading experts and thought leaders, we have an event that is right for you. Questions? Contact [email protected].

OCTOBER Computing), virtual 15 November 3 October • SERVICES (IEEE World Con- • SC, virtual • PACT (Int’l Conf. on Parallel gress on Services), virtual 16 November Architectures and Compila- 21 October • FOCS (IEEE Symposium on tion Techniques), virtual • FIE (IEEE Frontiers in Educa- Foundations of Computer Sci- 5 October tion Conf.), virtual ence), Durham, USA • EDOC (IEEE Int’l Enterprise 25 October • LCN (IEEE Conf. on Local Com- Distributed Object Computing • LDAV (IEEE Symposium on puter Networks), virtual Conf.), virtual Large Data Analysis and Visu- 29 November 12 October alization), virtual • ICDCS (IEEE Int’l Conf. on Dis- • ISSRE (IEEE Int’l Symposium • VIS (IEEE Visualization Conf.), tributed Computing Systems), on Software Reliability Eng.), virtual Singapore virtual 30 November • QCE (IEEE Int’l Conf. on Quan- NOVEMBER • ICHI (IEEE Int’l Conf. on Health- tum Computing and Eng.), 6 November care Informatics), Oldenburg, virtual • CCEM (IEEE Int’l Conf. on Germany 17 October Cloud Computing in Emerg- • MICRO (IEEE/ACM Int’l Sym- ing Markets), virtual DECEMBER posium on Microarchitecture), • SmartCloud (IEEE Int’l Conf. on 2 December virtual Smart Cloud), virtual • CSDE (IEEE Asia-Pacific Conf. 18 October 9 November on Computer Science and Data • ICCD (IEEE Int’l Conf. on Com- • ICTAI (IEEE Int’l Conf. on Tools Eng.), Gold Coast, Australia puter Design), virtual with Artificial Intelligence), • ISM (IEEE Int’l Symposium on • MODELS (ACM/IEEE Int’l virtual Multimedia), Naples, Italy Conf. on Model-Driven Eng. • IRC (IEEE Int’l Conf. on Robotic 6 December Languages and Systems), Computing), virtual • HOST (IEEE Int’l Symposium virtual • ISMVL (IEEE Int’l Symposium on Hardware-Oriented Secu- 19 October on Multiple-Valued Logic), rity and Trust), virtual • DFT (IEEE Int’l Symposium on virtual 7 December Defect and Fault Tolerance in 11 November • BDCAT (IEEE/ACM Int’l Conf. VLSI and Nanotechnology Sys- • SEC (IEEE/ACM Symposium on on Big Data Computing, Appli- tems), virtual Edge Computing), San Jose, cations and Technologies), • EDGE (IEEE Int’l Conf. on Edge USA virtual

64 September 2020 Published by the IEEE Computer Society 2469-7087/20 © 2020 IEEE • UCC (IEEE/ACM Int’l Conf. on 2021 21 April Utility and Cloud Computing), • SELSE (IEEE Workshop on Sil- virtual JANUARY icon Errors in Logic - System 9 December 5 January Eﬀ ects), Los Angeles, USA • CC (IEEE Int’l Conf. on Conver- • WACV (IEEE Winter Conf. on sational Computing), virtual Applications of Computer MAY • AIKE (IEEE Int’l Conf. on Arti- Vision), Waikoloa, USA 3 May ficial Intelligence and Knowl- 10 January • NAS (IEEE Int’l Conf. on Net- edge Eng.), virtual • ICPR (Int’l Conf. on Pattern working, Architecture and 10 December Recognition), Milan, Italy Storage), Riverside, USA • BigData (IEEE Int’l Conf. on Big 17 January 10 May Data), virtual • BigComp (IEEE Int’l Conf. on • CCGrid (IEEE/ACM Int’l Sym- 14 December Big Data and Smart Comput- posium on Cluster, Cloud and • CloudCom (IEEE Int’l Conf. on ing), Bangkok, Thailand Internet Computing), Mel- Cloud Computing Technol- 27 January bourne, Australia ogy and Science), Bangkok, • ICSC (IEEE Int’l Conf. on 17 May Thailand Semantic Computing), Laguna • IPDPS (IEEE Int’l Parallel and • HPCC (IEEE Int’l Conf. on High Hills, USA Distributed Processing Sympo- Performance Computing and sium), Portland, Oregon, USA Communications), Cuvu, Fĳ i MARCH 23 May 16 December 22 March • SP (IEEE Symposium on Secu- • BIBM (IEEE Int’l Conf. on Bio- • PerCom (IEEE Int’l Conf. on rity and Privacy), San Fran- informatics and Biomedicine), Pervasive Computing and cisco, USA virtual Communications), Kassel, • HiPC (IEEE Int’l Conf. on High- Germany Performance Computing, • MIPR (IEEE Int’l Conf. on Mul- Data, and Analytics), virtual timedia Information Process- 29 December ing and Retrieval), Tokyo, • BigDataSE (IEEE Int’l Conf. on Japan Big Data Science and Eng.), Guangzhou, China APRIL • EUC (IEEE Int’l Conf. on Embed- 12 April Learn more ded and Ubiquitous Comput- • ICST (IEEE Conf. on Software about IEEE ing), Guangzhou, China Testing, Verification and Val- • TrustCom (IEEE Int’l Conf. on idation), Porto de Galinhas, Computer Trust, Security and Privacy in Brazil Society Computing and Communica- 19 April conferences tions), Guangzhou, China • ICDE (IEEE Int’l Conf. on Data computer.org/conferences Engineering), Chania, Greece

ce9con(all).indd 73 8/18/20 3:41 PM NEW EVENT IEEE QUANTUM WEEK

1216 OCTOBER 2020

REGISTRATION IS OPEN! qce.quantum.ieee.org

The Future Directions Quantum Initiative invites you to IEEE Quantum Week 2020—the inaugural IEEE International Conference on Quantum Computing and Engineering (QCE).

Council on Superconductivity